How to build enterprise-ready AI solutions with Azure AI Agents
Learn about architectural options for enterprise-ready LLM AI solutions using Azure AI Agents
Integrating large language models (LLMs) into apps has become popular over the past few years. But there’s challenges hiding beneath the surface.
The way some apps send general queries to LLM endpoints (like Open AI’s API) is basically a black box. If the responses start to vary or the quality decreases, figuring out the “why” becomes a challenge — unless you build governing logic yourself. Not ideal.
Introduction
This issue has become even more imminent, as hastily pushed out AI features are in production for quite some time now and the need for standardization and governance is rising.
This is where agents come in. Agents are wrappers around LLMs. They’re fine-tuned and specifically configured to excel at certain tasks, making them way better at recurring jobs than regular, all-purpose LLM endpoints. They’re easier to monitor, debug, and tweak — giving you much more control.
You can either develop AI Agents yourself, or use one of the options available on the market. In this post, we’re diving into how Azure AI Agents can be used to create a enterprise-ready AI integration.
AI Assistants vs. AI Agents
When integrating AI into your applications, understanding the core differences between AI Assistants and AI Agents is crucial. Here’s a quick summary to help you differenciate them:
TL;DR:
- AI Assistants: Perfect for quick and simple projects (e.g., MVP) with minimal governance. Handles the “Question — Answer”-pattern nicely.
- AI Agents: Built for enterprise-scale AI solutions with standardization, security, observability and advanced capabilities. Handles complex logical chains.
Choose based on the complexity and scale of your use case. In the next section, I’ll introduce you to the options available on Microsoft Azure.
Options on Microsoft Azure
Now, as we know when to use AI Assistants, vs. when to use AI Agents, I would like to introduce you to the relevant offerings of the Azure Platform.
We have both paradigmns available to use:
My personal recommendation: Choose OpenAI Assistants for small use cases and MVPs, due to it’s increased development speed. For all other use cases, I would recommend Azure AI Agents due to their suitability for the enterprise.
Question: Can I access Azure AI Agents with an API Key?
This question has been proven to be difficult to answer, primarily due contradicting information by Microsoft. It is not possible to use Azure AI Agents with an API Key. Only Entra ID will work.
Building your own Agent vs. Azure AI Agents
You can build your own agent if you need a highly custom solution, but it’s time-consuming and requires.. resources. You’ll handle a lot — model connectivity, data integration, authentication, pipelines, and observability.
It’s recommendable to either use a library (like e.g., LangChain), or a one-stop-shop solution. In this post, we’ll look at Azure AI Agents as a simple out-of-the-box AI Agent solution.
Azure AI Agents
Azure AI Agents are ready-to-use, cost-effective solutions within the Azure ecosystem. They come with seamless integrations (AzureML, OpenAI, Azure Search, Bing Web Search Grounding, ..) and built-in capabilities for vision, speech, observability, and enterprise-grade security via Azure AD. Plus, no additional costs — for now. (besides the LLM Tokens)
TL;DR:
- Give Azure AI Agents a try, if time is limited and you don’t need a highly costumized solution.
How to work with AI Agents
Before we get started on how we work with AI Agents, I would like to introduce you to some key concepts for creating scalable, governable and observable AI solutions.
The first step is to understand the downsides of very large, monolitic prompts in comparision to smaller, specialized ones. More in the section below.
Prompt Monoliths vs. Prompt Sequences
When working with AI, breaking big prompts into smaller, focused ones generally yields better results. The reasons are simple:
- Accuracy: Small prompts produce clearer, more precise outputs.
- Governance: Each step’s quality is easier to assess and refine.
- Obserability: Only specialized prompts can be observed for their quality.
- Flexibility: Modular workflows can adapt and evolve quickly.
Instead of creating one gigantic prompt, divide tasks into logical, small steps. To understand that paradigm better, I provided an example of a trip planning prompt and divided it up into smaller tasks:
Smaller prompts ensure control, clarity, and better outputs. Start with individual tasks anditerate — building a modular architecture right away. It will pay off later once you’ll need to maintain your solution in production.
AI Agent Archictures on MS Azure
Previously, we discovered that smaller prompts are almost always better than big ones. Let’s take a moment to explore different architecture styles that we could choose from.
There are multiple options available, and how we design these systems will impact flexibility, functionality, and ease of monitoring.
Abstract vs. Specialized Agents
Think of it this way: abstract agents handle broader responsibilities, while specialized agents focus on narrower, well-defined tasks. An abstract agent is a generalist, while a specialized agent is — well — a specialist.
While abstract agents might feel easier to design upfront, they can be much harder to govern and maintain later.
On the other hand, specialized agents are more complicated to design and setup in the first place, but allow for better monitoring and increased control later on.
For example:
- Abstract/multi-task agents: If an agent is tasked with juggling multiple objectives, it doesn’t have a clear purpose — it will work, but monitoring will become a nightmare.
- Specialized agents: When an agent is assigned a single, clear task, it’s much simpler to continuously evaluate its quality based on that specific responsibility. You could even deifne test cases for that specific task.
Why Smaller Prompts Matter
To implement advanced features — like customization, broader knowledge integration, tools like grounding, orchestration, observability, and trust — you’ll find that splitting big prompts into smaller, manageable components becomes essential.
This ties directly into agent-building styles. You could construct a single-purpose agent, but you will find out that designing a layered ecosystem of specialized agents with a modular approach will make a difference in performance and quality.
Let’s evaluate some approaches and see what works for building agents. Each choice has trade-offs, and it’s about balancing development simplicity (time + money) with the accountability, observability and quality requirements.
Use Case: Personalized Travel Itinirary Recommender
Have you ever found yourself wondering where to go for vacation? Or maybe you have been stuck debating how to and where to go for a long weekend? Sounds familiar?
Decisions like these can be tricky, especially when you’re trying to balance travel styles, location options, and personal preferences. That’s where AI Agents come into play and could help us with ideation.
Mission: We use AI Agents, and pair them with personal preferences to generate personalized recommendations for travel itineraries.
Let’s take a look at some architectures to achieve that goal.
Abstract agent
The Abstract Agent pattern uses a single agent to handle many tasks within one domain, like travel planning. It’s easy to set up and integrate since all tasks are managed in one place, making it ideal for quick implementations or MVPs.
In this pattern, all the logic is handled within an external application (“orchestrator”) — allowing flexibility and control over data flows. You’ll handle all the prompting within your app and may create different threads for different queries. That’s a lot of manual development effort if you want to ensure high quality.
Monitoring and evaluating response quality is difficult, and issues in specific tasks can be overlooked. The lack of specialization may lead to shallow answers, making users unhappy who expect detailed answers.
This pattern is suited for simple systems where (development) speed matters more than precision. For more complex or high-quality applications, specialized approaches are recommended.
- Pros: Easy to set up, easy to integrate
- Cons: Hard go govern, hard to monitor
Specialized Agents — External Orchestration
The Specialized Agents with External Orchestration pattern involves splitting agents by specific tasks but managing the connection and utilization through an external application (“orchestrator”). The orchestrator could be implemented in e.g., .NET or Python — using the corresponding libraries. It requires a similiar setup as the “Abstract agent”-approach on the orchestrator side.
This approach keeps agents focused on specialized tasks while allowing full control of data flow and interactions through the orchestrating application. It’s ideal for use cases that require control.
This design ensures better governance and monitoring compared to abstract agents, as tasks are split up into individual AI Agents and are easier to evaluate individually.
This pattern is well-suited for systems needing precision, accountability, and strong control over data flow. While it demands more effort upfront, it offers scalability and transparency for complex applications.
- Pros: Better governance, monitoring of agents possible, full control of data flow, accountability of responses increases
- Cons: Higher complexity, more setup effort, higher development costs
Specialized Agents — Orchestration within AI Agent Toolset
The Specialized Agents with Orchestration within AI Toolset pattern enables agents to directly communicate and assign specialized tasks on their own, while the external application acts as a lightweight wrapper for the main Agent Endpoint, not as an orchestrator. Accountability and data flows are handled within the agents themselves, minimizing external dependency and creating a flexible, autonomous system.
This approach prioritizes modularity and scalability, making integration seamless. It saved a low of time, as the framework offers all fucntions you would implement in an orchestrator. It’s closely aligned with Microsoft’s vision for AI Agents and the Azure AI Agent ecosystem — that’s why it could be seen as the “best practice”.
However, the reliance on proprietary agent implementations increases the risk of vendor lock-in dramatically, limiting future migration options and tying the use case to the vendor platform.
- Pros: Modular, scalable, reduced external orchestration
- Cons: Vendor lock-in possibility, harder to switch out agents with different ones if better options arise.
Conclusion
Specialized prompts are required to split up AI tasks into specialized prompts to allow for monitoring and observability. Abstract prompts just can’t offer that level of accountability and should be avoided in enterprise solutions.
Hand in hand with specialized prompts, we should utilize AI Agents. They dramatically help in making AI accountable, observable and provide reliable monitoring through their established frameworks.
Vendors are also seeing this potential and are increasingly offering their own AI Agents on their platforms — besides known libraries like langchain. That’s good and bad — be cautious of vendor lock-in.
That’s why I would personally still recommend to do the orchestration externally like shown in Specialized Agents — External Orchestration. That approach balances accountability and observability requirements while trying to avoid a possible future vendor lock-in.
tl;dr
- Abstract Agents are best suited for MVPs or simple, low-stakes implementations where ease of setup and rapid deployment are key priorities.
- Externally Orchestrated Agents offer stronger accountability and monitoring capabilities, making it the general rule of thumb for scalable and maintainable applications.
- AI Agent Toolset orchestrated Agents promote autonomy, flexibility and allow to save a lot of time. But they introduce a significant risk of vendor lock-in, potentially limiting long-term adaptability and should be approached cautiously.
Comments