Martin Jurran Blog

How to Choose an Agentic AI Orchestrator

Thu, 02 Apr 2026 00:00:00 GMT

Beyond traditional automation: AI Workflows empower teams to move away from administrative heavy-lifting and into a high-level oversight role. Lay back and let the AI do the job. Probably no one has been able to escape the hype around Agentic AI over recent months. I think we have all seen incredible demos: AI Agents acting as subject matter experts, or autonomously writing reliable code to draw insights from data and make business decisions. In the past, those tasks were incredibly time-consuming for humans, involving multiple steps and navigating complex data ambiguity. Now, that ambiguity and complexity can be automated. In my personal opinion, AI Agents do not flourish in isolation. They truly shine when they are integrated directly into your core business processes. Naturally, both business leaders and tech experts are excited by these developments. But as reality sets in, a critical question remains: How do we bridge the gap between isolated demos and reliable, real-world business value? The speed at which the entire ecosystem around Agentic AI is developing is like nothing I have personally ever seen before. However, specific solution types are emerging and maturing, and can now be evaluated for strategic decisions. It may be time to look for a generalized Agentic AI Workflow platform that covers the vast majority (95%) of your business demands. In this blog post, I want to provide you with actionable insights you can use today. Who this guide is for This guide is meant for teams that are beyond the first AI demo and now need to make platform decisions: - Enterprise and platform architects who must decide how Agentic AI fits into existing enterprise landscapes. - Engineering leads and senior developers who will actually build and operate multi‑agent workflows and need to understand concepts and trade-offs. - Product owners and business stakeholders who need enough structure to ask the right questions when evaluating AI orchestration platforms and budgeting for them. If you are still experimenting with one‑off prototypes, this article can serve as a roadmap for what will matter more when you want to scale to production. Concept Agentic AI in the context of workflows adds a new level of complexity. Agents, models, and tools need to reliably communicate, execute tasks, and share their results with one another without breaking down. An AI Orchestrator is designed to solve these exact challenges by offering a centralized platform where Agentic Workflows can be executed and managed. It provides standardized interfaces, handles complex dependencies, and ensures repeatability. In a nutshell, it introduces the necessary enterprise guardrails - such as robust error handling, state management, and seamless human-in-the-loop oversight. Finally, that's why these platforms are becoming the essential foundation for running complex, production-ready Agentic Workflow systems. Short Definition: "What is an AI Orchestrator?" Think of it like a musical orchestra: the instruments provide the sound, but the conductor provides the direction. An AI Orchestrator performs the same vital role for your Agentic Workflows. In literature, an AI orchestrator is defined as a central management layer that turns independent components into a coherent, goal-oriented system. [1]. It is the "business glue", that coordinates interactions between separate AI models, tools and data sources. [2] Nowadays, two main types of orchestrators can be distinguished: - Workflow Orchestrators (combining deterministic steps with AI decisions, e.g. n8n/Azure Logics Apps/Camunda) and - Agent Orchestrators (Agents, Tools, Memory, Governance, e.g. Azure AI Foundry, Databricks) Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Categorization of AI Orchestration Solutions Choosing an AI orchestration platform is like choosing clothes: the right option depends on the weather, your mood, and even your size. The market for AI orchestrators in the broader sense is currently fragmented and defined by different design philosophies. Without a clear definition for different categories, we might see a "Conceptual Retrofitting" [7], which will lead to non-deterministic approaches/workflows being put into perspective from a deterministic model of understanding. To better get a picture of what is available in the Agentic AI workflow market, we should classify solutions into individual dimensions: 1) Orchestration Approach - Central Orchestration: A central orchestrator AI agent acts as the coordination of the system, delegates tasks to different agents but performs all final decisions. [3] - Decentralized Orchestration: Specialized AI agents interact directly with each other and reach a shared statement via peer coordination. [3] - Hierarchical Orchestration: There are levels of AI agents, whereas parent agents steer the execution of their children. [4] - Federated Orchestration: Collaboration between independent AI agents, without sharing all data or handing off tasks completely. [4] - Deterministic Orchestration: Combining the probabilistic nature of LLMs with the reliability of statically typed business logic. Utilizes pre-defined paths (workflows) to ensure consistency. [8] 2) Development Approach - Code-First SDKs and Frameworks: Uses code-native functionalities for developing/orchestrating AI agents and provides granular control for software developers over architecture and processes. [5] - Visual Tools, Low-Code/No-Code-Platforms: Uses drag-and-drop or other mechanisms to lower entry barriers for non-technical users. [6] - Enterprise Platforms: Consolidates all above-mentioned types on a single enterprise-native platform to use economies of scale and primarily offer governance, safety and compliance features. Full List of Requirements for AI Orchestrators The current (as of 04/2026) literature defines which requirements are the most common for AI orchestration solutions. Below you find a table based on literature research: Connectivity & Architecture PriorityCapabilityWhy it mattersMust‑haveProvider agnostic designThe platform can talk to multiple LLM providers, including internal or self‑hosted models, so you are not locked into a single vendor. [16]Must‑haveAPI and data source integrationBuilt‑in mechanisms to connect external systems and data sources (REST, databases, event streams, etc.) as first‑class inputs into your workflows.Must‑haveSupport for open protocolsUses standardized protocols such as MCP (Model Context Protocol) or Agent‑to‑Agent patterns so agents, tools and models can interoperate across ecosystems. [13]Must‑haveSandboxed tool executionAgent tools and actions run in isolated environments to protect corporate infrastructure and reduce blast radius. [10]DesignModular agent componentsClear separation between workflow logic, model layer and tools, so each layer can evolve independently without breaking the others. [16]Must‑haveStructured outputs as an interfaceAgent outputs are exposed in a structured, machine‑readable way (JSON, events, schemas) so downstream systems can reliably consume them. [14], [16]QualityExternal prompt managementPrompts are managed outside the agent definition (e.g., in a prompt store), so versions can be updated and rolled back without redeploying the whole platform. [12]QualityDeveloper experienceAgents can be authored flexibly – via code, visually, or a mix of both – to support different skills and teams. Performance & Scalability PriorityCapabilityWhy it mattersPerformanceLow end‑to‑end latencyTasks are processed quickly enough that interactive and near‑real‑time scenarios remain usable for end users. [9]PerformanceHigh throughputThe orchestrator can handle large request volumes (e.g., thousands of calls per second for simple API orchestration) without degradation. [9]PerformanceHorizontal scalabilityWorkloads can scale out across nodes or instances to absorb traffic spikes and growth. [9]PerformanceResource efficiencyTechniques like caching, batching and reuse of context reduce compute and token costs per workflow run. Governance, Security & Compliance PriorityCapabilityWhy it mattersMust‑haveRBAC and audit trailsFine‑grained, role‑based access control plus complete logging of actions and decisions to satisfy compliance requirements. [16]Must‑haveOutput evaluationSystematic evaluation of agent outputs (e.g., tests, automatic scoring, LLM‑based judges) to track quality over time. [16]Must‑haveLayered security controlsMultiple defenses such as sandboxing, prompt shields and prompt‑injection protection to reduce security risk. [10]QualityObservability and tracingDeep traces on node and workflow level to shorten mean time to detect and repair issues. [16]GovernanceExplainabilityAbility to reconstruct and explain why a given decision path was taken ("explainable AI" for workflows and agents). [16]Must‑haveUser impersonation for tool useTools and downstream systems are called under the active user’s permissions so authorization rules stay intact. [10]Must‑havePolicy‑driven executionCentral policies (e.g., DLP, data residency, allowed actions) define what agents may or may not do. [16]Must‑haveWorkflow lifecycle managementOrchestrator supports full lifecycle control via CI/CD (promotion, canary, rollback) for workflows and agents. [16]Must‑haveCatalog of agents and actionsCentral catalog of tools, agents and data assets to avoid "agent sprawl" and duplicate capabilities. [10], [16]QualityGuardrails on inputs and outputsInput filters and safety policies prevent harmful prompts and responses from reaching users or systems. [16] Cognitive Control & Reasoning PriorityCapabilityWhy it mattersMust‑haveState and memory managementShort‑ and long‑term memory across interactions so agents can maintain context over multi‑step workflows. [15]QualityDeterministic paths where requiredCritical business logic can be constrained to fixed, deterministic paths to limit LLM unpredictability.Must‑haveContext managementCompression, summarization and retrieval keep context windows focused and reduce hallucinations in long threads. [11]QualityOntologies and knowledge modelsUse of enterprise ontologies or knowledge graphs to steer how context is selected and interpreted. [16]Must‑haveUncertainty awareness and escalationThe system recognizes when it is unsure or the risk is high and routes to a human instead of guessing. [9], [16]PerformanceMeta‑reasoning and trust calibrationQuantitative metrics (e.g., calibration measures) help track when agents are over‑ or under‑confident and tune them accordingly. Economics PriorityCapabilityWhy it mattersPerformanceCost visibility and routingReal‑time tracking of token and compute spend, plus routing of simple tasks to cheaper options. [15]PerformanceModel routing strategiesDynamic assignment of steps to cost‑effective models, reserving premium models only for complex reasoning. [10] Operations, Reliability & Interaction PriorityCapabilityWhy it mattersMust‑haveResilience and fault tolerancePatterns such as retries, circuit breakers, fallbacks and self‑healing agents keep workflows running despite partial failures. [15], [16]Must‑haveVersioning and safe rollbackPrompts, workflows and agents are versioned and can be rolled back instantly if regressions appear. [12], [16]InteractionHuman‑in‑the‑loop checkpointsExplicit approval steps for risky or expensive actions (for example, "review before apply") ensure humans stay in control. [16] Summary of Key Requirements for AI Orchestrators When you look across the dimensions in the tables, a few requirements stand out as central for evaluating Agentic AI platforms: Integration and openness The platform should connect to existing APIs, data sources and tools, work with multiple model providers, and expose structured outputs (for example JSON or events) that downstream systems can process reliably. Scalability and performance It needs to handle interactive use cases with low latency, scale horizontally for higher load, and use resources efficiently (caching, batching, model routing) to keep costs predictable. Economics and cost control The platform should provide real-time visibility into token and compute spend and apply cost-aware routing strategies, assigning simple tasks to cheaper models or runtimes while reserving premium options for complex reasoning. Security, governance and compliance Capabilities such as RBAC, audit trails, sandboxed tool execution, policy‑driven behavior, and a central catalog of agents and tools help align AI workflows with existing security and compliance standards. This also includes systematic evaluation of outputs, explainability of decision paths, lifecycle controls (promotion, canary, rollback), and guardrails on inputs and outputs to enforce safety policies. Control over reasoning and behavior State and memory management, context management, deterministic paths for critical steps, and mechanisms for uncertainty handling and escalation support both flexibility and predictable behavior. Enterprise ontologies or knowledge models, plus meta-reasoning and confidence calibration, further help orchestrators pick the right context and tune agent behavior over time. Operational robustness Observability, tracing, resilience patterns (retries, fallbacks), versioning and rollback, plus human‑in‑the‑loop checkpoints are needed to operate Agentic workflows as reliable production systems rather than one‑off experiments. Market Situation for AI Orchestrator Solutions Before looking at individual solutions, it helps to remember that platforms differ along two main dimensions: how they orchestrate work and what kind of product they fundamentally are. In regards to Abstraction Level, some solutions focus only on Agentic AI orchestration, while others are more generic platforms that have added AI capabilities as just one of their many offerings. When it comes to Orchestration Styles, not every platform supports all patterns equally. Some are optimized for central "orchestrator" agents, others shine in hierarchical supervisor–worker setups, and only a subset can truly support decentralized or federated collaboration scenarios in a robust and trustworthy way. Understanding which orchestration styles are solid and production-grade vs. awkward and instable to use in each solution is important long‑term. Thought: Those two aspects combined matter. Generic enterprise platforms tend to perform nicely in governance, integration and operations, while specialized tools often move faster in Agent features and developer productivity. Additionally, the correct Orchestration Style may depend on your internal processes and tool landscape. You should look at some of these solutions yourself and see how they fit into your companies' processes. This following table is intentionally high‑level; it helps you shortlist tools rather than rank them: NameOrchestration StyleDevelopment Modality (Abstraction Level)Agent Bricks- Central orchestration ("multi‑agent supervisor") - Hierarchical orchestration (supervisor pattern coordinating specialized agents) - Federated orchestration (via MCP catalog)Enterprise platform (integrated Mosaic AI governance)AutoGen- Central orchestration - Decentralized orchestration (peer conversations) - Hierarchical orchestration (orchestrator model) - Federated orchestrationCode‑first SDK and frameworksAzure AI Foundry- Central orchestration - Hierarchical orchestration (agents interacting via tool calls) - Deterministic orchestration (workflow‑based)Enterprise platform (governance and security focus)Azure Logic Apps- Central orchestration (agent loop) - Hierarchical orchestration (workflows can call other workflows) - Deterministic orchestration (originally deterministic concept, extended with agent capabilities)Enterprise platform (managed iPaaS)Camunda 8- Deterministic orchestration (process‑driven based on BPMN standard) - Hierarchical orchestration (via BPMN sub‑processes) - Central orchestration (non‑deterministic components within deterministic flows)Enterprise platform (enterprise workflow engine)CrewAI- Hierarchical orchestration (supervisor–worker) - Central orchestration (sequential process model) - Decentralized orchestration - Federated orchestration - Deterministic orchestrationVisual low‑code/no‑code platform (via CrewAI Studio)LangGraph- Central orchestration (supervisor) - Decentralized orchestration - Hierarchical orchestration - Federated orchestrationCode‑first SDK and frameworksn8n- Hierarchical orchestration (orchestrator–worker graphs) - Deterministic orchestration (workflows) - Central orchestrationVisual low‑code/no‑code platformVellum AI- Central orchestration - Hierarchical orchestration (manager–worker) - Federated orchestration - Deterministic orchestration (workflow builder)Enterprise platform (focus on compliance and evaluations)Google Vertex AI Agent Builder- Central orchestration - Hierarchical orchestration (agents with roles and responsibilities) - Federated orchestration (MCP protocol) - Decentralized orchestration (agent‑to‑agent) - Deterministic orchestrationEnterprise platform (managed cloud) As this field is evolving fast like I never saw before, I decided against including a massive comparison matrix that scores every platform against every requirement. At this point, you know the theoretical categories of Orchestration Solutions. Start by looking at your own organization and processes: What kind of orchestration patterns do you actually need, and which development modality fits your teams? Then shortlist one or more options from the table above and evaluate them hands‑on. If you end up with a small set of finalists, take your most critical requirements from the requirements tables and check how well each candidate satisfies them in practice. If you already use a major cloud platform like Azure or Google Cloud, it is usually smartest to start with their native offerings rather than introducing a completely new stack. Likewise, if you already rely on a process automation engine such as Camunda, first explore how its emerging Agentic capabilities can be extended before bringing in yet another orchestration layer. Conclusion (tl;dr) - Stick with what you have (Enterprise): If you are already utilizing a major cloud platform like Azure, check their native tools before looking elsewhere. Azure Logic Apps (Standard) with Agent Loop for example is now a top-tier AI workflow solution with all major features required according to literature and has a perfect integration into your enterprise governance. - But, watch the scale (if you approach hundreds of workflows, for example 500+): If you have a high volume of workflows, Azure Logic Apps can get expensive. Assess standalone platforms like n8n or Camunda. Keep in mind that while self-hosting is "free," the maintenance costs could exceed those of a managed cloud service. It depends on your specific situation. - Starting from scratch (SME): If you are an SME and don't have a dedicated cloud provider, either just try Azure Logic Apps or n8n (if you like self-hosted / on-prem friendly). Those are the most flexible "Swiss Army knives" for rapid prototyping, and you pay close to nothing at small scale. Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Sources [1] The Orchestration of Multi-Agent Systems: Architectures, Protocols, and Enterprise Adoption. [2] Foundations of GenAI Orchestration: RAG, MLOps, and LLMOps – Zengzheng Jiang. [3] A Taxonomy of Hierarchical Multi-Agent Systems: Design Patterns, Coordination Mechanisms, and Industrial Applications. [4] What is AI Agent Orchestration? | IBM. [5] A Comparative Architectural Analysis of LLM Agent Frameworks: LangChain, LlamaIndex, and AutoGPT in 2025 | Uplatz Blog. [6] Multi-Agent Supervisor Architecture: Orchestrating Enterprise AI at Scale | Databricks Blog. [7] Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directions. [8] AI-Driven Orchestration Systems in Cloud-Native Financial Applications: A Framework for Next-generation Investment Platforms. [9] How to Evaluate and Benchmark AI Orchestrators? [10] Process to build agents process across your organization with Microsoft Foundry and Copilot Studio (Cloud Adoption Framework, Microsoft Learn). [11] AgentX: Towards Orchestrating Robust Agentic Workflow Patterns with FaaS-hosted MCP Services. [12] A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows. [13] The Orchestration of Multi-Agent Systems: Architectures, Protocols, and Enterprise Adoption. [14] Multi-Agent Supervisor Architecture: Orchestrating Enterprise AI at Scale (Databricks Blog). [15] Orchestral AI: A Framework for Agent Orchestration. [16] Trustworthy Orchestration Artificial Intelligence by the Ten Criteria with Control-Plane Governance.

How to build enterprise-ready AI solutions with Azure AI Agents

Wed, 13 Aug 2025 00:00:00 GMT

AI is sometimes too good to be true — it’s so easy to integrate and use compared to rule based approaches. That’s why governance is important. Befriend the AI and be accountable together. (Photo by the author) Integrating large language models (LLMs) into apps has become popular over the past few years. But there’s challenges hiding beneath the surface. The way some apps send general queries to LLM endpoints (like Open AI’s API) is basically a black box. If the responses start to vary or the quality decreases, figuring out the “why” becomes a challenge — unless you build governing logic yourself. Not ideal. Introduction This issue has become even more imminent, as hastily pushed out AI features are in production for quite some time now and the need for standardization and governance is rising. This is where agents come in. Agents are wrappers around LLMs. They’re fine-tuned and specifically configured to excel at certain tasks, making them way better at recurring jobs than regular, all-purpose LLM endpoints. They’re easier to monitor, debug, and tweak — giving you much more control. You can either develop AI Agents yourself, or use one of the options available on the market. In this post, we’re diving into how Azure AI Agents can be used to create a enterprise-ready AI integration. AI Assistants vs. AI Agents When integrating AI into your applications, understanding the core differences between AI Assistants and AI Agents is crucial. Here’s a quick summary to help you differenciate them: AI Assistants vs. AI Agents (Photo by the author) TL;DR: - AI Assistants: Perfect for quick and simple projects (e.g., MVP) with minimal governance. Handles the “Question — Answer”-pattern nicely. - AI Agents: Built for enterprise-scale AI solutions with standardization, security, observability and advanced capabilities. Handles complex logical chains. Choose based on the complexity and scale of your use case. In the next section, I’ll introduce you to the options available on Microsoft Azure. Options on Microsoft Azure Now, as we know when to use AI Assistants, vs. when to use AI Agents, I would like to introduce you to the relevant offerings of the Azure Platform. We have both paradigmns available to use: Overview Azure OpenAPI Assistants vs. Azure AI Agents (Photo by the author) My personal recommendation: Choose OpenAI Assistants for small use cases and MVPs, due to it’s increased development speed. For all other use cases, I would recommend Azure AI Agents due to their suitability for the enterprise. Question: Can I access Azure AI Agents with an API Key? This question has been proven to be difficult to answer, primarily due contradicting information by Microsoft. It is not possible to use Azure AI Agents with an API Key. Only Entra ID will work. Building your own Agent vs. Azure AI Agents Developers love to create code. Somtimes though, it’s more productive to work with whats already there. (Photo by the author) You can build your own agent if you need a highly custom solution, but it’s time-consuming and requires.. resources. You’ll handle a lot — model connectivity, data integration, authentication, pipelines, and observability. It’s recommendable to either use a library (like e.g., LangChain), or a one-stop-shop solution. In this post, we’ll look at Azure AI Agents as a simple out-of-the-box AI Agent solution. Azure AI Agents Azure AI Agents are ready-to-use, cost-effective solutions within the Azure ecosystem. They come with seamless integrations (AzureML, OpenAI, Azure Search, Bing Web Search Grounding, ..) and built-in capabilities for vision, speech, observability, and enterprise-grade security via Azure AD. Plus, no additional costs — for now. (besides the LLM Tokens) TL;DR: - Give Azure AI Agents a try, if time is limited and you don’t need a highly costumized solution. How to work with AI Agents Before we get started on how we work with AI Agents, I would like to introduce you to some key concepts for creating scalable, governable and observable AI solutions. The first step is to understand the downsides of very large, monolitic prompts in comparision to smaller, specialized ones. More in the section below. Prompt Monoliths vs. Prompt Sequences When working with AI, breaking big prompts into smaller, focused ones generally yields better results. The reasons are simple: - Accuracy: Small prompts produce clearer, more precise outputs. - Governance: Each step’s quality is easier to assess and refine. - Obserability: Only specialized prompts can be observed for their quality. - Flexibility: Modular workflows can adapt and evolve quickly. Instead of creating one gigantic prompt, divide tasks into logical, small steps. To understand that paradigm better, I provided an example of a trip planning prompt and divided it up into smaller tasks: An example of splitting up a prompt monolith into a prompt sequence (Photo by the author) Smaller prompts ensure control, clarity, and better outputs. Start with individual tasks anditerate — building a modular architecture right away. It will pay off later once you’ll need to maintain your solution in production. AI Agent Archictures on MS Azure Previously, we discovered that smaller prompts are almost always better than big ones. Let’s take a moment to explore different architecture styles that we could choose from. There are multiple options available, and how we design these systems will impact flexibility, functionality, and ease of monitoring. Abstract vs. Specialized Agents Something abstract or something specialized? Would you hire a generalist or a specialist to take care of your precious machine? (Photo by the author) Think of it this way: abstract agents handle broader responsibilities, while specialized agents focus on narrower, well-defined tasks. An abstract agent is a generalist, while a specialized agent is — well — a specialist. While abstract agents might feel easier to design upfront, they can be much harder to govern and maintain later. On the other hand, specialized agents are more complicated to design and setup in the first place, but allow for better monitoring and increased control later on. For example: - Abstract/multi-task agents: If an agent is tasked with juggling multiple objectives, it doesn’t have a clear purpose — it will work, but monitoring will become a nightmare. - Specialized agents: When an agent is assigned a single, clear task, it’s much simpler to continuously evaluate its quality based on that specific responsibility. You could even deifne test cases for that specific task. Why Smaller Prompts Matter To implement advanced features — like customization, broader knowledge integration, tools like grounding, orchestration, observability, and trust — you’ll find that splitting big prompts into smaller, manageable components becomes essential. This ties directly into agent-building styles. You could construct a single-purpose agent, but you will find out that designing a layered ecosystem of specialized agents with a modular approach will make a difference in performance and quality. Let’s evaluate some approaches and see what works for building agents. Each choice has trade-offs, and it’s about balancing development simplicity (time + money) with the accountability, observability and quality requirements. Use Case: Personalized Travel Itinirary Recommender Vacation is always a nice examples.. (Photo by the author) Have you ever found yourself wondering where to go for vacation? Or maybe you have been stuck debating how to and where to go for a long weekend? Sounds familiar? Decisions like these can be tricky, especially when you’re trying to balance travel styles, location options, and personal preferences. That’s where AI Agents come into play and could help us with ideation. Mission: We use AI Agents, and pair them with personal preferences to generate personalized recommendations for travel itineraries. Let’s take a look at some architectures to achieve that goal. Abstract agent Chart showing an abstract agent setup using a Azure Function and Azure AI Agents (Photo by the author) The Abstract Agent pattern uses a single agent to handle many tasks within one domain, like travel planning. It’s easy to set up and integrate since all tasks are managed in one place, making it ideal for quick implementations or MVPs. In this pattern, all the logic is handled within an external application (“orchestrator”) — allowing flexibility and control over data flows. You’ll handle all the prompting within your app and may create different threads for different queries. That’s a lot of manual development effort if you want to ensure high quality. Monitoring and evaluating response quality is difficult, and issues in specific tasks can be overlooked. The lack of specialization may lead to shallow answers, making users unhappy who expect detailed answers. This pattern is suited for simple systems where (development) speed matters more than precision. For more complex or high-quality applications, specialized approaches are recommended. - Pros: Easy to set up, easy to integrate - Cons: Hard go govern, hard to monitor Specialized Agents — External Orchestration Chart showing an specialized agent setup using a Azure Function and multiple Azure AI Agents (Photo by the author) The Specialized Agents with External Orchestration pattern involves splitting agents by specific tasks but managing the connection and utilization through an external application (“orchestrator”). The orchestrator could be implemented in e.g., .NET or Python — using the corresponding libraries. It requires a similiar setup as the “Abstract agent”-approach on the orchestrator side. This approach keeps agents focused on specialized tasks while allowing full control of data flow and interactions through the orchestrating application. It’s ideal for use cases that require control. This design ensures better governance and monitoring compared to abstract agents, as tasks are split up into individual AI Agents and are easier to evaluate individually. This pattern is well-suited for systems needing precision, accountability, and strong control over data flow. While it demands more effort upfront, it offers scalability and transparency for complex applications. - Pros: Better governance, monitoring of agents possible, full control of data flow, accountability of responses increases - Cons: Higher complexity, more setup effort, higher development costs Specialized Agents — Orchestration within AI Agent Toolset Chart showing “best practice” set up solution where Azure AI Agents orchestrate each other while an Azure Function acts as a wrapper for the main Azure Agent endpoint. (Photo by the author) The Specialized Agents with Orchestration within AI Toolset pattern enables agents to directly communicate and assign specialized tasks on their own, while the external application acts as a lightweight wrapper for the main Agent Endpoint, not as an orchestrator. Accountability and data flows are handled within the agents themselves, minimizing external dependency and creating a flexible, autonomous system. This approach prioritizes modularity and scalability, making integration seamless. It saved a low of time, as the framework offers all fucntions you would implement in an orchestrator. It’s closely aligned with Microsoft’s vision for AI Agents and the Azure AI Agent ecosystem — that’s why it could be seen as the “best practice”. However, the reliance on proprietary agent implementations increases the risk of vendor lock-in dramatically, limiting future migration options and tying the use case to the vendor platform. - Pros: Modular, scalable, reduced external orchestration - Cons: Vendor lock-in possibility, harder to switch out agents with different ones if better options arise. Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Conclusion Specialized prompts are required to split up AI tasks into specialized prompts to allow for monitoring and observability. Abstract prompts just can’t offer that level of accountability and should be avoided in enterprise solutions. Hand in hand with specialized prompts, we should utilize AI Agents. They dramatically help in making AI accountable, observable and provide reliable monitoring through their established frameworks. Vendors are also seeing this potential and are increasingly offering their own AI Agents on their platforms — besides known libraries like langchain. That’s good and bad — be cautious of vendor lock-in. That’s why I would personally still recommend to do the orchestration externally like shown in Specialized Agents — External Orchestration. That approach balances accountability and observability requirements while trying to avoid a possible future vendor lock-in. tl;dr - Abstract Agents are best suited for MVPs or simple, low-stakes implementations where ease of setup and rapid deployment are key priorities. - Externally Orchestrated Agents offer stronger accountability and monitoring capabilities, making it the general rule of thumb for scalable and maintainable applications. - AI Agent Toolset orchestrated Agents promote autonomy, flexibility and allow to save a lot of time. But they introduce a significant risk of vendor lock-in, potentially limiting long-term adaptability and should be approached cautiously.

Build a Full Stack Application with SAP CAP & Vue.js 3

Tue, 29 Apr 2025 00:00:00 GMT

Name	Origin Country	Coat	Pattern
{{ catBreed.name }}	{{ catBreed.originCountry }}	{{ catBreed.coat }}	{{ catBreed.pattern }}

Dreaming about easy back-ends.. (Photo by the author) I have to admit - SAP has never been my favorite tech company. For a long time, I associated them with bulky, overly complex, and somewhat outdated solutions that often felt unintuitive to developers like me. Obviously, that's a pretty simple opinion fueled by prejudice and misguided feelings of supremacy. But hey, we've all been there. SAP seems a bit strange at first - but it's actually friendly. (Photo by the author) That's why I was pleasantly surprised when I stumbled upon SAP CAP (short for SAP Cloud Application Programming Model). It's an open-source development framework for building cloud services - and it's pretty cool. SAP CAP is very nice because it solves a lot of the usual questions in back-end development pretty much out-of-the-box: - Conceptual Data Modeling: Support for features like temporal data types, localization, and user-managed data. - Data Abstraction: A unified ORM with support for Postgres and HANA. - Built-In Authentication & Authorization: No need to reinvent the wheel! - Advanced Security: Native support for JWT, CORS, X-CSRF, and more. - Local Development: Easily spin up locally cloned service instances. - API Options: Choose between REST, OData, or GraphQL for exposing endpoints. - OpenAPI Documentation: Publish OpenAPI docs and even convert them to TypeScript for your front-end (a big win!). Here's the key takeaway: while SAP CAP integrates seamlessly within the SAP ecosystem, you're not locked in. The framework's flexibility is very nice, and having it released as an open-source piece that's backed and maintained by an enterprise-grade company is a huge plus. So, let's dive right into the framework. And if you've made it this far, thanks for sticking around - I promise it gets better from here! The first impression "Getting offered the CAP" (Photo by the author) As developers, we're no strangers to frameworks like LoopBack, Strapi, and PocketBase - tools designed to make back-end development faster and more efficient. SAP CAP takes a similar approach to these frameworks but, in my opinion, does it better. One standout feature is its surprisingly active and robust community, which has contributed tons of cool add-ons and extensions. Think Websockets, GraphQL integration, Audit Logging, Change Tracking, Telemetry, and more. Here's a taste of just some of the nice things the SAP CAP community brings to the table: CAP Plugins & Enhancements | capire Quick Comparision Where would journalistic objectivity be if we simply let my previous statements stand unchallenged? Right. To provide a more balanced perspective, I've put together a quick comparison of several back-end frameworks. This should give you a brief idea of where SAP CAP stands in the landscape: Comparision of various alternatives (SAP CAP, PocketBase, SupaBase, LoopBack) to create back-ends / api-based services. (Photo by the author) SAP CAP can be roughly compared to LoopBack (former IBM) both in scope and underlying business need. In general SAP CAP has a more modern, developer-friendly approach but comes with some drawbacks: - It's not fully hosted on open platforms like GitHub or GitLab, which makes it harder to compare with other open-source projects. - It personally feels more like a hybrid solution than true open source. - There aren't a ton of external contributors, which kinda gives off the vibe that it's not drawing as much attention from the external community. In my personal opinion, it would be awesome to see SAP open things up more in the future and really get the external community involved! Understanding CAP - the basics SAP CAP comes with a wide range of capabilities, covering aspects such as domain modeling, providing and consuming services, events & messaging, and much more. At the time of writing, there are about 16 distinct aspects of SAP CAP, all thoroughly documented in The CAP Cookbook. But don't worry - you don't need to master all of them. As your project evolves and new requirements emerge, you'll naturally explore these aspects one by one. The iterative approach is one of the core values of SAP CAP. For building an application that's split into a front-end (avoiding Fiori/UI5 for the front-end) and back-end, you only need to focus on three key aspects to get started: A very generic conceptual structure of a SAP CAP project (Photo by the author) - 1 Domain Modeling - 2 Providing Services - 3 Consuming Services (Clients) Before we start implementing (I know you're excited for the code), I will introduce you to the three aspects we are going to use today. The aspects are already tailored towards our implemented use case: A simple catalog of cat breeds. 1. Domain Modeling SAP CAP is designed in a way that makes you start out with domain modeling - which makes sense. To do this effectively, it's recommended to collaborate with business stakeholders or domain experts to design the model together. This is exactly what CDS (Core Data Services) is designed for - modeling data close to natural language, so stakeholders can be loosely involved with implementation. SAP CAP encourages keeping models as flat as possible with minimal relationships, which brings us to a data structure like this, written in CDS: namespace catcat.domain; @description: 'Cat breeds with their properties' entity CatBreeds { key name : String @mandatory; originCountry : String; coat : String; pattern : String; } As you see, modeling in CDS can be pretty straightforward. This section was just for the general concept. I will show the implementation later. 2. Providing Services Once we've created a basic data model, the next step is to build services on top of it. These services are also modeled using CDS. Let's say we create a service CatCatalog that returns an entity CatBreeds. We create a very simple service definition followed by a projection onto the database entity. To add functionality, we added annotations for the route (@title), permissions (@requires), and some more. See the code below: using catcat.domain as domainSchema from '../db/domainSchema'; @title: 'cat-catalog' @Core.LongDescription: 'Provides all entities/services related to handling cat breeds.' service CatCatalog{ @readonly @description : 'Returns a list of all cat breeds' entity CatBreeds as projection on domainSchema.CatBreeds { * } } As you see, providing services in CDS can also be straightforward. This section is still just for the general concept. I will show the implementation later. 3. Consuming Services With the previous two aspects we can create a very basic service - potentially up and running. To now consume this service, we have multiple approaches. The most straightforward method is to use the Swagger integration that SAP CAP provides. With just a single npm install, we can generate an interactive API definition that looks like this: SAP CAP service viewed through native Swagger UI (Photo by the author) Regarding integrating front-ends The native approach for front-ends in SAP CAP is by leveraging the Fiori/UI5 framework. While this method is nice when staying within SAP's architecture, it can be limiting when development requires flexibility. I personally prefer using MVVM front-end frameworks like Angular, React or Vue.js for faster, and more flexible development. With SAP CAP, generating an OpenAPI 3 definition is simple and requires just one command: cds compile srv --service all -o docs --to openapi --openapi:diagram false The definition can then be translated into a typescript definition to be used within any typescript-based front-end. Perfect! That's how you can use Angular, React or Vue.js. Typescript definition generated with SAP CAP - ready to use in any TypeScript-based codebase (Photo by the author) That's all you need to know about the concepts involved. Let's do an actual implementation together. Building our Application Constructing our application in SAP CAP - carefully. (Photo by the author) Thank you for taking the time to go through the earlier section where we explored the fundamentals of SAP CAP. By now, you should have a solid understanding of the key components of this framework. Now, it's time to dive into the fun part: implementation. Objective: We want to build a simple Cat Catalog service that serves Cat Breeds to a Vue.js front-end. In this section, we'll build that application step by step using SAP CAP. Here's what's on the agenda: - 1. Setting up - Project setup in VS Code - 2. Building - back-end - 3. Building - front-end - 4. Completion - Seeing it in action! - 5. Conclusion & Retrospective 1. Setting up - Project setup in VS Code Make sure you have the prerequisites (NodeJS, VS Code) set up. More info: Getting Started | capire Install "@sap/cds-dk2" with command "npm add -g @sap/cds-dk" in your terminal. Create an empty folder for your project Open VS Code & open the folder Setup completed! 🥳 2. Building - back-end The approach in SAP CAP is domain centric: That's why our implementation starts with data modeling in the back-end. SAP CAP (Cloud Application Programming) and the CDS (Core Data Services) language are designed as modeling frameworks for less tech-savvy users. It allows domain experts to actively participate in the creation of the data model. By collaborating directly in CDS, you can simplify the process and eliminate redundant data modeling phases in external tools. This collaborative approach supports alignment but also saves time and resources. 1. Create a CAPService folder, like shown below: Creating a new folder within VS Code (Photo by the author) 2. Navigate to CAPService folder and init the project with cds init: cd .\CAPService\ cds init Initial files generated by cds init (Photo by the author) 4. Create your first schema file as domainSchema.cds in ./db/ folder: Creating a schema file domainSchema.cds under ./db/ folder (Photo by the author) 5. The contents of the DB model will be roughly translated to DB tables. For our service, create a model entity CatBreeds like this: namespace catcat.domain; @description: 'Cat breeds with their properties' entity CatBreeds { key name : String @mandatory; originCountry : String; coat : String; pattern : String; } 6. The DB model entities are private by default and are not being exposed. Exposing model entities is done through service definitions. For that, we need to create a new service cat-catalog.cds in ./srv/ folder: Creating a service definition cat-catalog.cds under ./srv/ (Photo by the author) 7. The definition for your service CatCatalog will look like this: using catcat.domain as domainSchema from '../db/domainSchema'; @title: 'cat-catalog' @Core.LongDescription: 'Provides all entities/services related to handling cat breeds.' service CatCatalog{ @readonly @description : 'Returns a list of all cat breeds' entity CatBreeds as projection on domainSchema.CatBreeds { * } } 8. Already done! We can take a first glimpse at our created service using the command cds watch: npm install cds watch 9. The command cds watch allows us to open the service in a debugging mode. It will look like this: SAP CAP standard service landing page during debugging (Photo by the author) 10. Awesome! Looks very nice. We have a cat-catalog service that exposes CatBreeds as an endpoint. Beautiful. Adding Open API support 11. Now, let's try to interact with our service. For that, I recommend to add native Swagger UI support (cds-swagger-ui-express) with: npm add --save-dev cds-swagger-ui-express 12. That's all. We now run cds watch again: cds watch 13. Now, the previously shown UI loaded. As you see, we have a new button for Open API Preview: Additional button Open API Preview appeared after installing cds-swagger-ui-express (Photo by the author) 14. After clicking, we are immediatelly greeted with a Swagger UI. We didn't need to set up anything. How nice! Swagger UI natively served by SAP CAP in debugging mode (Photo by the author) Adding initialization data 15. When you play around in Swagger UI, you will see that the CatBreeds service is empty. For that, we can add some initialization data with the command cds add data: cds add data 16. The command executed and generated a .csv file under .\db\data\\*.csv: *.csv-file to initialize our service with data (Photo by the author) Thats how you can locate it in your explorer: *.csv-file to initialize our service with data (Photo by the author) 17. Now, we just need to add our data to the CSV. I asked AI to generate us 10 items for our data set. Use with caution - these might not be real CatBreeds: name,originCountry,coat,pattern Persian,Iran,Longhair,Solid Siamese,Thailand,Shorthair,Pointed Maine Coon,United States,Longhair,Tabby Bengal,United States,Shorthair,Spotted Sphynx,Canada,Hypoallergenic,None Ragdoll,United States,Longhair,Pointed Russian Blue,Russia,Shorthair,Solid Abyssinian,Egypt,Shorthair,Ticked Scottish Fold,Scotland,Shorthair,Solid Norwegian Forest Cat,Norway,Longhair,Tabby 18. Save the *.csv-file and run cds watch again. Use Open API Preview and test the API again. You will now see that data is being returned: Output of Swagger UI containing data stored by CAP service (Photo by the author) Our fully featured API is already working! Now it's time to move towards the connection of our front-end. Adding the OpenAPI export 19. Let's create an OpenAPI export. I personally like to add it as a command within the CapService folder root through package.json. Modify yours with "genopenapi" like shown below: { "name": "CAPService", "version": "1.0.0", "description": "A simple CAP project.", "repository": "", "license": "UNLICENSED", "private": true, "dependencies": { "@sap/cds": "^8", "express": "^4" }, "devDependencies": { "@cap-js/cds-types": "^0.10.0", "@cap-js/sqlite": "^1", "cds-swagger-ui-express": "^0.10.0", "swagger-ui-express": "^5.0.1" }, "scripts": { "start": "cds-serve", "genopenapi": "cds compile srv --service all -o docs --to openapi --openapi:diagram false" } } 20. We now run the command to generate the service definition: npm run genopenapi As you can see, a new file of type .openapi3.json has been created in the ./docs/ folder. It contains the service definition: OpenAPI3 spec created under ./docs/ for our CatCatalog service (Photo by the author) 21. You see that a new folder has been added. We should add ./docs/ to .gitignore file to avoid using outdated api specs (personal recommendation) Adding docs/ to .gitignore to not accidentially use outdated service definitions (Photo by the author) Done! We can now relieable generate Open API Specifications for your CAP service. Implementing the translation from OpenAPI to TypeScript (TS) 22. Move up to main project root with command cd .. and init a sekeleton package.json with the following command: npm init -y 23. Enhance the generated package.json with the commands pretransferapidef and transferapidef, like shown below: { "name": "bog__catcat_catgatalog_example", "version": "1.0.0", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "pretransferapidef": "npm --prefix ./CAPService run genopenapi", "transferapidef": "npm run pretransferapidef && npx openapi-typescript" }, "keywords": [], "author": "", "license": "ISC", "description": "" } 24. The package openapi-typescript requires a configuration. It can work with redocly.yaml standard. Thats why we create a redocly.yaml in the main project root folder for handling the OpenAPI => TypeScript conversion process: Adding docs/ to .gitignore to not accidentially use outdated service definitions (Photo by the author) 25. For our example, configure it like the following: apis: CatCatalog@v1: root: ./CAPService/docs/CatCatalog.openapi3.json x-openapi-ts: output: ./HTML5Module/src/generated/CatCatalog.d.ts 26. That's all! We can now run the command to test our workflow: npm run transferapidef Success! The TypeScript definition has been created in our folder HTML5Module\src\generated! It's working. Typescript definition CatCatalog.d.ts created by command npm run transferapidef (Photo by the author) We're now done with implementing the back-end and providing the base for consumption. The next step will solely consist of connecting our front-end to the service. 3. Building - front-end As you might have noticed, I am focusing on Vue.js. That choice has two sides: First, it is my personal preference - you could use any TypeScript-based framework instead. Second, Vue.js seems to be popular amongst the SAP CAP team themself - they offer resources on how to work with it. They also seem to have more resources on Vue.js than Angular or React. Initializing a Vue.js project 27. Delete the whole HTML5Module directory, including the file we just generated. Now, it's time to initialize the Vuejs project. It will be named HTML5Module. Use this command npm create vue@latest 28. Work through the setup wizard. Choose the following settings: Wizard for initializing vue projects - npm create vue@latest (Photo by the author) 29. Initialize the project and start the local development server with the following commands: cd ./HTML5Module npm install npm run dev 30. It's running! You can now see a basic vue boilerplate app in your browser. Default landing page ofa blank vue project (Photo by the author) 31. Let's transfer our API TypeScript definition again. Run the following command in the project root: npm run transferapidef Under /HTML5Module/src/generated, you can now find our created API definition. Locating generated typescript definition within our vue project (Photo by the author) 32. Finally, we can perform the final step and connect our front-end to the back-end. We previously installed Pinia, which is handling our state. State is organized in different stores. For that, we create a new file catCatalog.ts under /HTML5Module/src/stores. Locating catCatalog.ts in our vue project (Photo by the author) 33. Pinia stores follow a certain pattern. To simplify things, paste the provided boilerplate into catCatalog.ts: import { defineStore } from 'pinia'; import { ref } from 'vue'; //placeholder type & client import export const useCatCatalog = defineStore('catCatalog', () => { const catBreeds = ref(null); /** * @description Fetches all cat breeds from the remote sorce */ async function fetchCatBreeds(): Promise { try { //placeholder api call } catch (err: unknown) { } } return { catBreeds, fetchCatBreeds } }); 34. We will now import our type definitions from our generated file: import { defineStore } from 'pinia'; import { ref } from 'vue'; //ype & client import import type { components } from "../generated/CatCatalog"; type CatBreed = components['schemas']['CatCatalog.CatBreeds']; export const useCatCatalog = defineStore('catCatalog', () => { const catBreeds = ref(null); /** * @description Fetches all cat breeds from the remote sorce */ async function fetchCatBreeds(): Promise { try { //placeholder api call } catch (err: unknown) { } } return { catBreeds, fetchCatBreeds } }); 35. Next, we install a client. I decided for openapi-fetch. There is multiple libraries that could be chosen instead. Install library with this command: npm install openapi-fetch --save-dev 36. Now, we initialize the client in our Pinia store. It's done like this: import { defineStore } from 'pinia'; import { ref } from 'vue'; //ype & client import import createClient from "openapi-fetch"; import type { components } from "../generated/CatCatalog"; import type { paths } from "../generated/CatCatalog"; type CatBreeds = components['schemas']['CatCatalog.CatBreeds']; const apiclient = createClient({ baseUrl: "http://localhost:4004/odata/v4/cat-catalog" }); export const useCatCatalog = defineStore('catCatalog', () => { const catBreeds = ref(null); /** * @description Fetches all cat breeds from the remote sorce */ async function fetchCatBreeds(): Promise { try { //placeholder api call } catch (err: unknown) { } } return { catBreeds, fetchCatBreeds } }); 37. The last pending thing it implementing the API call. We can now just use our client, call the endpoint and map the data directly. Beautiful: import { defineStore } from 'pinia'; import { ref } from 'vue'; //ype & client import import createClient from "openapi-fetch"; import type { components } from "../generated/CatCatalog"; import type { paths } from "../generated/CatCatalog"; type CatBreeds = components['schemas']['CatCatalog.CatBreeds']; const apiclient = createClient({ baseUrl: "http://localhost:4004/odata/v4/cat-catalog" }); export const useCatCatalog = defineStore('catCatalog', () => { const catBreeds = ref(null); /** * @description Fetches all cat breeds from the remote sorce */ async function fetchCatBreeds(): Promise { try { // api call const {response, data} = await apiclient.GET('/CatBreeds'); if (!response.ok) throw new Error('Malformed response by mode endpoint'); if (!data) throw new Error('Failed to fetch mode details / malformed data'); data.value != undefined ? catBreeds.value = data.value : null; return true; } catch (err: any) { return new Error(err) } } return { catBreeds, fetchCatBreeds } }); Done! We now created the back-end connection. You already see - it's very straightforward and easy to implement. 38. The last setup is to call our store and display the data in the UI. For keeping things simple, we just exchange the whole App.vue under /HTML5Module/src/App.vue with very simple code: Done! Our app is basically ready. After a reload of our front-end view, you can see that the values are being pulled from the back-end: Wow, it's working! Our cat breeds are being displayed (Photo by the author) 4. Completion - Seeing it in action! We have a connection between our back-end and front-end in place now. You can navigate to http://localhost:5173/ to see it in action. That means, we finished our objective. It's time for a retrospective. 5. Conclusion & Retrospective We are at the finishing line! Done! (Photo by the author) Honestly, building a full-stack app with SAP CAP on the backend and Vue.js 3 on the frontend is easy. It's developer-friendly and packed with features that make backend development a breeze. Advantages of this setup Pairing it with Vue.js made the front-end side more fun, especially with how easy it is to pull in the TypeScript definitions for the API. - We can locally develop our application End-To-End, without an online connection being required. - By using a monorepo, we can align versioning. Removing ./src/generated from versioning allows us to generate it during the build process, ensuring our Vue app's implementation code is type-checked against the current API definition - eliminating dead endpoints. We can also deploy both components at the same time. - Authentication & authorization can be handled by an proxy in front and forward auth tokens to the back-end. SAP AppRouter is a perfect example. No unnecessary code in your back-end! Disadvantages of this setup Of course, there were a few hiccups. Deploying CAP outside of SAP's cloud isn't super intuitive. It'd be nice if SAP worked on making their framework a little more open and easier to use outside the BTP bubble. Final conclusion The whole experience feels good and everything works together smoothly. I'd definitely recommend this combination for anyone interested in building modern apps, especially if you're looking to experiment with enterprise-grade frameworks without losing control of your code. GitHub Repository The whole code used & created within this blog post is available on GitHub: Open repository blog-capjs-catcatalog by martinjurran on github.com Feel free to submit issues in case of questions! :) Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Lessons Learned I have used the technology to build a production app that I will release later. Things I learned while building something big: - CORS Implementation is incomplete; SAP CAP doesn't return allowed Methods by default. Custom Implementation needed. Important for production. - X-CSRF is required by SAP CAP default but not implemented in openapi-fetch; Either "missing out on security" (bad idea) or a custom implementation on client side is required. - Conceptual Data Model influences service definitions; unlike stated by SAP CAP team, you're not free from constraints when modeling the service endpoints. Things like flattening out arrays of objects, etc. are not completely implemented. - Deploying SAP CAP outside of BTP is not documented straightforwardly; Connecting the PostgreSQL database presented challenges due to a incomplete documentation at this time. - Deploying SAP CAP outside of BTP is not documented straightforwardly; How to build it for deployment within custom environments (e.g. bare vps, docker, ..) is not fully described. - Service definitions in SAP BTP (MTA.yaml) are challenging; If you don't have access to SAP's graphical configuration tools and/or insufficient permissions to systems. - Using complicated data types, especially geo references, presents challenges; Especially if database extensions like PostGIS are involved, there is a lot of custom code to be written.

Fill Word Templates without Premium Connectors in Power Automate

Fri, 08 Nov 2024 00:00:00 GMT

Filling out Microsoft Word forms in Power Automate (Photo by the author, Illustrations by Takashi Mifune under free use) Enterprise users, especially those involved with the digitalization of existing processes, might be faced with the requirement to fill out certain standardized forms with Power Automate and make these available to the respective stakeholders. Microsoft offers a Populate a Microsoft Word Template action right out of the box in Power Automate, but for many organizations, using Premium Connectors is not feasible due to company policies. There is a simple solution: We can create a workaround for filling out forms and offering them as a *.pdf with a SharePoint Document Library and a Power Automate Flow. The following step-by-step guide will walk you through how to electronically fill out Microsoft Word-based Forms using Power Automate. Concept Very rough concept of how to Fill out a Microsoft Word Form using Power Automate w/o any Premium Connectors (Photo by the author, Illustrations by Takashi Mifune under free use) Before we start with the implementation, let’s take a look at the rough concept: Microsoft SharePoint is one of the most common and free-to-use data sources within the Microsoft Power Platform. With the Document Library functionality in SharePoint, we have a central place to store documents. Even better: We can assign metadata using Document Library columns, which we can then embed into the document in Microsoft Word. That being said, we can break down what we need to do into these steps: - 1 Create a Document Library - 2 Add the Microsoft Word template fields as Document Library Columns - 3 Upload the Template as a Microsoft Word file - 4 Link the Document Library Columns into the Microsoft Word file using Word Desktop. - 5 Create a Power Automate Flow that copies the template file, updates the metadata and creates a *.pdf out of 6 the Microsoft Word file Now as we know what to do in theory, let’s dive right into the step-by-step guide. Solution Table of Contents - 1 Acquire a SharePoint Site - 2 Create a SharePoint Document Library - 3 Add the fields for the template as Document Library Columns - 4 Link the Document Library Columns into the Word document fields - 5 Create a Power Automate flow 1. Acquire a SharePoint Site This approach requires a SharePoint site, as we use SharePoint’s Document Library feature. Please make sure you have access to a SharePoint site first. The process for acquiring a SharePoint site might differ from organization to organization, that’s why it isn’t covered here. [link1] 2. Create a SharePoint Document Library Creating a new Document library (Photo by the author) For our approach, we need to use a Document Library. A Document Library is a file storage in which we can assign additional properties and metadata to a file. Please create a Blank library, as depicted below: Creating a new blank Document library (Photo by the author) Assign a speaking name to the Document Library. That will allow us to locate the Document Library better later on. Creating a new Document library (Photo by the author) 3. Add the fields for the template as Document Library Columns Once created, access the Document Library settings using the settings cogwheel in the M365 main navigation, as highlighted below: Accessing Document Library Settings (Photo by the author) Open “More library settings” to access the column configuration as highlighted below: Accessing more library settings (Photo by the author) When you scroll down, you see an overview of available fields. Below the list, you have the option to add new columns. Create one column for each field in your template Create a new column in a document library (Photo by the author) You will be prompted with a dialog to provide additional information about your field. Define the Name as needed. The columns highlighted are the ones with proven compatibility: Creating a new column in a document library (Photo by the author) Once you have configured the required fields, either upload your existing Template Word File, or create a new one. As I don’t have a template file available yet, I am proceeding with creating a new one. 4. Link the Document Library Columns into the Word document fields Creating a new Word document in a Document Library (Photo by the author) It is crucial for this approach to open the file in the Microsoft Word Desktop application. Please switch to the Desktop app, as highlighted below: Opening a Word document in the Desktop app (Photo by the author) Once you arrive in the Desktop app, you can find and add the SharePoint Library columns through Insert ➡ Text ➡ Document Property ➡ “Column Name” as depicted below: Adding a new property reference to a Word document (Photo by the author) Link all the relevant SharePoint Document Library Columns directly into the Template File. Overview of how property references are displayed in Microsoft Word (Photo by the author) Now, we’re done with the Microsoft Word Template setup and can begin with creating our Power Automate Flow. 5. Create a Power Automate flow There are multiple ways to create a Power Automate Flow. If you navigate to make.powerautomate.com, you will see this screen on the starting page with the option to create a new flow. Press the + Create button to get started. Creating a new Power Automate Flow (Photo by the author) In this example, I create a new Flow. For that purpose, using a Instant cloud flow is the best for demo purposes. Depending on your project requirements, you can pick any Flow type you want. Creating a new Instant cloud flow (Photo by the author) Assign a good name to the Flow. You don’t need use the name I decided for. Creating a new instant cloud flow, with the Flow button type trigger (Photo by the author) The first required action is a Compose action. We will create a temporary file name. Creating a Compose action in Power Automate (Photo by the author) As we would like the temporary file name to be absolutely unique, I am assigning a new GUID to the Compose action. For the Formula, please see below: Assigning guid() to Compose (Photo by the author) Now, we need to add a Copy file action from the SharePoint Connector’s catalog as shown below: Creating a Copy file action (Photo by the author) We just copy the existing template file into the same Document Library. For configuration, please see below: Configuring the Copy file action (Photo by the author) Next, we need to update the SharePoint Columns with their respective values. To update Columns within a SharePoint Document Library, use the Update file properties action, as shown below: Adding the Update file properties action (Photo by the author) Configure the action as below. Notice that we use the output of the Copy file action, more specifically, the ItemId of the newly created item. Configuring the Update file properties action (Photo by the author) Update the individual SharePoint Document Library Columns to your liking. I am using static values for demo purposes. Of course, the input can also be dynamic. Updating the template properties in Power Automate using the Update file properties action (Photo by the author) As we have now have changed the Document’s Document Library Column Properties, it is now time to create a snapshot of the current state. For that, we utilize the Get file content action in SharePoint: Adding the Get file content action (Photo by the author) We use the Identifier of the action Update file properties as input for the action Get file content. Configuring the Get file content action (Photo by the author) The next step is to copy the file from SharePoint to OneDrive for Business. You might be wondering why — right? That is, because conversion to *.pdf is only available in OneDrive for Business as of now. That’s why we create a new Create file action from the OneDrive for Business Connector’s catalog. Adding the Create file action (Photo by the author) Then, we use our temporary file name as the desired file name in OneDrive for Business . As File Content, we supply the result of the Get file content action. Configuring the Create file action (Photo by the author) Now, as we have the file available in OneDrive for Business for Business, we can convert it. For that, we use the Convert file action, as shown below: Adding the Convert file action (Photo by the author) As the action’s input parameter, we supply the Id of the Create file action. Our target type will be PDF: Adding the Convert file action (Photo by the author) After successful conversion, we should transfer the file back into our SharePoint universe. For that, we create another Create file action at the end of our Flow. As the filename, use our temporary file name with the .pdf extension. As for the content, use the output of the Convert file action. Configuring the Create file action (Photo by the author) We are done now and are finally ready to test. Save the flow first, and then click the Test button within the Power Automate designer. Testing the flow (Photo by the author) Opt for Manually testing the Flow and proceed by clicking the big, blue Test button. Testing the flow (Photo by the author) Confirm the action with a click on Run Flow — as shown below: Testing the flow (Photo by the author) We now see that the Flow has been successfully executed — very nice. If you did everything right, the designer would look like this: Testing the flow — success! (Photo by the author) To check our result, navigate back to our Document Library. There, you see the newly created file with the temporary file name we created using Compose: Seeing the newly created file (Photo by the author) Let’s preview the file to check our result. As you can see, the template placeholders have been replaced with the actual values we put into the Document Libraries Columns. So nice! Reviewing the contents of the template (Photo by the author) That’s basically it. Now we have a very simple, running Flow that helps us to fill out templates. To make this solution production-ready, you might need to: - Delete the temporary files within the Flow - Implement error handling - Link your values to another data source, instead of statically placing them. - and more, depending on your requirements Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Conclusion It’s very much possible to fill out Microsoft Word-based forms using Power Automate without relying on Premium Connectors. That said, I personally hope that Microsoft will implement some changes: - First, I hope that the action Populate a Microsoft Word Template will be free to use for everyone in the future. - Second, I hope the data conversion can be done natively, without needing to sync data into OneDrive for Business . Until then, the approach described in this article is the best available.

SharePoint Lists & PowerApps - Issues with Calculated Columns of Type “Number”

Thu, 24 Oct 2024 00:00:00 GMT

Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate PowerApps is receiving calculated columns of type Number in the wrong format (Photo by the author, illustrations by Takashi Mifune under free use) When creating Canvas PowerApps, developers might face the requirement of including values of SharePoint’s Calculated Columns eventually. However, if you have worked with PowerApps before, you might know that the data types are not consistent between SharePoint and Power Apps when it comes to the type Number. That situation can leave developers with quite the headache: "Help! — I have created a Calculated Column in a SharePoint List, but when I want to display the value inside my Power App, a lot of trailing numbers are added. What can I do?" - Someone working with Calculated columns of type Number in PowerApps There is a simple solution: Convert the property to a data format Power Apps can work with. Depending on your setup, there are two different approaches. - Fix the issue in SharePoint (recommended): If data consistency is not required, change the returned data type to Text. - Fix the issue in PowerApps: If data consistency is required, data types should be manually converted in PowerApps. Please take a look at the following decision Flow Chart: Decision chart on what to do when affected by Calculcated Columns of type Number in Power Apps (Photo by the author) The following step-by-step guide will walk you through how to handle trailing zero issues with Calculated SharePoint Columns in PowerApps: Solutions Solution 1: Format the Number in SharePoint and deliver type “Text” (recommended) The best approach is to avoid the bug altogether. In detail, that means, converting the Number to Text in SharePoint right away and serving the column as type “Text”. This approach is only suitable if data type consistency within your solution is not important. When other systems correct data type handling access SharePoint, other issues will arise. The necessary changes are reflected in this comparison: Comparision of before and after (Photo by the author) Solution 2: Use Value() + Round() We can use Value() to convert a Text value to type Number. We can use Round() to remove any trailing zeros in an accurate way. Microsoft Power Fx: Round( Value(ThisItem.CalculatedColumnName), 2 ) Solution 3: Value() + Text() We can utilize Value() to convert Text value to type Number. With Text(), we can then format the Number how we would like to display it inside our application. Microsoft Power Fx: Text( Value(ThisItem.CalculatedColumnName), "[$-en-US]#,#.00" ) Conclusion It’s very much possible to still use Calculated Columns, even when values of type Number are involved. That said, Power Apps doesn’t do data type conversion out-of-the-box right, and some requirements require some workarounds. I personally hope that data type handling will be improved within Power Apps. Until then, the ways described in this article is the best we have.

How to read *.csv files directly in PowerApps

Thu, 22 Aug 2024 00:00:00 GMT

Reading *.csv in PowerApps through calling a Power Automate flow (Photo by the author, Illustrations by Takashi Mifune under free use)) Enterprise users frequently request mass data import features for their Power Apps. While there are various data exchange formats, *.csv files remain one of the most popular, especially due to their ease of use. However, if you’ve worked with Power Apps, you will know that there is no out-of-the-box solution for loading/importing *.csv files. This can leave Power Platform Architects with quite a headache. There is a simple solution: Power Apps and Power Automate can easily read *.csv-files together, providing a simple way to mass import data. They are in one ecosystem and collaborate nicely. The following step-by-step guide will walk you through how to load *.csv files in Power Apps using Power Automate. Let’s get started. Concept Concept of how to read *.csv in Power Apps (Photo by the author](/articles/legal/license)) Before we start implementing, it’s always good to take a look at the rough concept. Power Automate flows can be triggered by Power Apps — there are various ways, including user actions. The Power Automate trigger also allows for handing over properties — such as files. Finally, the trigger supports reading an HTTP response sent back by the Power Automate flow. That being said, we can break down what we need to do into these steps: - Create a flow with the Power Apps trigger - Hand over the file from Power Apps to Power Automate - Process the file in Power Automate - Send a response back to the Power App - Process the response and load the data to use it directly in the Power App Now as we know what to do in theory, let’s dive right into the step-by-step guide. Solution Table of Contents - Creating the Power Automate Flow - Configuring the Trigger - Initialize Global Variables - Determine CSV Separator - Determine CSV Header - Iterate through the *.csv file in parallel - Building the JSON response - Add Power Automate to Power App - Power Automate as data source in Power Apps - File input in Power Apps - Sending the *.csv-file from Power Apps to Power Automate 1. Creating the Power Automate Flow 1.1 Open the Power Automate dashboard, click on + New flow and select Instant cloud flow Creating a new Power Automate flow (Photo by the author](/articles/legal/license)) 1.2. Select the instant cloud flow trigger called When Power Apps calls a flow (V2), click on the big blue Create button. Selecting a trigger of type When Power Apps calls a flow V2 (Photo by the author](/articles/legal/license)) After successful creation, you will be redirected to the Power Automate Designer. 2. Configuring the Trigger 2.1. You are now in the Power Automate designer. Click on our input action When Power Apps calls a flow (V2) and navigate to the Parameter tab. The Parameters tab of our trigger action (Photo by the author](/articles/legal/license)) 2.2. Click on Add an input and add an input of type File. Choosing a parameter type in Power Automate (Photo by the author](/articles/legal/license)) 2.3. Assign a speaking name to the parameter, such as InputFile. Assign an easy-to-understand description. As we load *.csv-files, I decided for “Please select *.csv file”. The values picked here will reflect in our Power App. Configuring the parameter in Power Automate (Photo by the author](/articles/legal/license)) 2.4. Close the popup and go back to our flow view. Click the + Circle below the trigger, then click Add an action. Adding a new action to a Power Automate Flow (Photo by the author](/articles/legal/license)) The initial setup of our flow is completed now. Let’s continue with creating some global variables. 3. Initialize Global Variables 3.1. To initialize a variable, we need an action of type Initialize Variable. Assign a good name, I chose CSV_CONTENTS. Set the type to String and the value to InputFile contentBytes. Initialializing the variable CSV_CONTENTS (Photo by the author](/articles/legal/license)) The format provided by the When Power Apps calls a flow (V2) trigger is not the best to work with. It needs to be modified to be processable later. 3.2. Create another action of type Initialize Variable to format the output. I named it CSV_CONTENTS_FORMATTED, type String. The initial value is: split(Variables('CSV_CONTENTS'),'')[0] When set up, the action should look like this: Initialializing the variable CSV_CONTENTS_FORMATTED (Photo by the author](/articles/legal/license)) 3.3. Initialize another variable which is used for storing each csv row. I named it EACH_ROW_RAW, type array. Set the value to: split(encodeUriComponent(variables('CSV_CONTENTS_FORMATTED')),'%0D%0A') When set up, the action should look like this: Initialializing the variable EACH_ROW_RAW (Photo by the author](/articles/legal/license)) 3.4. Create more actions of type Initialize Variable. We need these variables available: - HEADER_LINE: Type:String / Value:Empty - HEADERS: Type:Array / Value:Empty - CSV_SEPARATOR: Type:String / Value: Empty - JSON_STRING: Type:String / Value:Empty - JSON_STRING_LOG: Type:String/ Value:Empty Initialializing the variable** CSV_SEPERATOR** and others (Photo by the author](/articles/legal/license)) 3.5. Create the last action of type Initialize Variable and name it LOOP_DETERMINEHEADER_RUNNING, type Boolean with default Value=true Initialializing the variable LOOP_DETERMINEHEADER_RUNNING (Photo by the author](/articles/legal/license)) Now, we finished initializing all the required variables and can continue with the actual implementation. 4. Determine CSV Seperator The scope establishes the boundaries within which actions and variables can be accessed and controlled. Here’s why scope is important in Power Automate: Encapsulation: Scope allows you to group related actions and variables together, making it easier to understand and maintain the workflow. (Process Street, How to Use Scope in Power Automate (process.st)) 4.1. First, we create a scope to group our first complicated implementation and assign a speaking name. Creating a Scope in Power Automate (Photo by the author](/articles/legal/license)) 4.2. Instead of using variables like before, we will now use the Compose Action. We first create a Compose Action for counting characters of type semicolon in our CSV_CONTENTS: length(split(variables('CSV_CONTENTS'), ';')) When set up, the action should look like this: Creating a Compose action in Power Automate (Photo by the author](/articles/legal/license)) 4.3. Now, we set up the same for counting comma: length(split(variables('CSV_CONTENTS'), ',')) When set up, the action should look like this: Creating a Compose action in Power Automate (Photo by the author](/articles/legal/license)) 4.4. Now, we calculated the count of commas and semicolons. We would now like to find out if the file is separated by comma or a semicolon. We just compare the two counts, and the highest value is most likely to be the dominant separator. This approach works in most cases but is not the best solution. Especially if the dataset can contain a lot of comma/semicolon itself. For the matter of keeping things simple, I continue with this approach. Creating a Condition in Power Automate (Photo by the author](/articles/legal/license)) Now we determined the separator used by our *.csv file. We can continue with determining the header. 5. Determine CSV Header For determining the header, we need to loop through our dataset. We assume, that the *.csv-file that is to be loaded is standardized. 5.1. We use a condition action to check if a line contains a specific header field. If it does, it’s our header. This approach works in most cases but is not the best solution. Especially if the dataset can contain a data set that also contains the header name. For the matter of keeping things simple, I continue with this approach. Creating a Condition in Power Automate (Photo by the author](/articles/legal/license)) 6. Iterate through the *.csv file in parallel The probably most important constraint for our solution is time — processing a *.csv-file line by line consecutively can take a lot of it. That’s where we can make use of parallelism in Power Automate, called concurrency. Each line of the *.csv-file will be worked on in parallel — pushed into an array variable as soon as processing finishes. That means, we will not be able to maintain the *.csv-files in original order. But, that is usually not important, as there should be identification fields in the *.csv-file depicting the correct order. 6.1. Create a scope and assign a speaking name Creating a Scope in Power Automate (Photo by the author](/articles/legal/license)) 6.2. Create an Apply to each loop. As the source we use EACH_ROW_RAW. Creating a Apply to each loop in Power Automate (Photo by the author](/articles/legal/license)) 6.3. Within the Apply to each action settings, we enable Concurrency Control with Degree of Parallelism=50. Activating Concurrency Control in Power Automate (Photo by the author](/articles/legal/license)) 6.4. Create a Compose action to decode the line value, with this code: decodeUriComponent(items('Apply_to_each')) When set up, the action should look like this: Creating a Compose action in Power Automate (Photo by the author](/articles/legal/license)) 6.5. Create a new Compose action that splits the csv line with the separator determined earlier. Use this code: split(outputs('Compose_EACH_ROW'), variables('CSV_SEPARATOR')) When set up, the action should look like this: Creating a Compose action in Power Automate (Photo by the author](/articles/legal/license)) 6.6. Validate the data set against the headers we extracted earlier. For the check, we compare if the length of the line element array: length(outputs('Array_with_each_element_in_the_CSV_line')) is equal to: length(variables('HEADERS')) That way we ensure compliance of individual datasets with the headers read before. Creating a Condition in Power Automate (Photo by the author](/articles/legal/license)) 6.7. If the condition is false, we append a string to our error log. We use the Append To String Variable Action Appending an error message to our response variable (Photo by the author](/articles/legal/license)) 6.8. Parse the individual column values. To ease things, we hard-code the headers to their column indexes. We use a compose action to determine a column value based on its index: replace(outputs('Array_with_each_element_in_the_CSV_line')[0],'"','') When set up, the action should look like this: Creating a Compose action in Power Automate (Photo by the author](/articles/legal/license)) 6.9. Build the JSON String manually in a Compose action. concat('"', variables('HEADERS')[0], '":"',outputs('Compose_COL_0'),'"') When set up, the action should look like this: Creating a Compose action in Power Automate (Photo by the author](/articles/legal/license)) 6.10. Append the JSON String to our result string variable JSON_STRING. Be aware that we add curly brackets and comma to separate the data sets manually. Appending a string to a string in Power Automate (Photo by the author](/articles/legal/license)) Now we are done with reading the individual lines and adding them to a JSON array within a string type Variable. We can now start building the response. 7. Building the JSON reponse 7.1. We initialize a variable called RESULT. There, we define our response format: Building a JSON response in Power Automate (Photo by the author](/articles/legal/license)) - json: replace(variables('JSON_STRING'),'""','"') - messages: replace(variables('JSON_STRING_LOG'),'""','"') 7.2. Create a Response action. That one will allow us to access the flow results directly in Power Apps. Status code should be 200, body will be: replace(variables('RESULT'),',]',']') When set up, the action should look like this: Creating a Reponse action in Power Automate (Photo by the author](/articles/legal/license)) We now completed the set up of our Power Automate flow — the next step is to prepare our Power App for calling the flow and working with the results. 8. Add Power Automate to Power App Connecting Power Automate flow to Power Apps (Photo by the author](/articles/legal/license)) 9. File input in Power Apps Acquiring a file input control in Power Apps is not trivial — per default, there is none. There is a workaround to use out-of-the-box file input in Power Apps. 9.1. Create a dummy SharePoint list which allows attachments. If you don’t know how, follow this guide: How to create SharePoint List - Beyond Intranet 9.2. Connect the dummy SharePoint list to your Power App. If you don’t know how, follow this guide: How to Connect SharePoint with Power Apps - TechEfficiencySolution 9.3. Create a Form in Power Apps with the dummy SharePoint list as data source Creating a form in Power Apps (Photo by the author](/articles/legal/license)) 9.4. Take a look at the newly created form and locate the attachment input Form based in SharePoint in Power Apps (Photo by the author](/articles/legal/license)) 9.5. Cut out the Attachment input and delete the Form from the Power App. File input control in Power Apps, extacted from Form (Photo by the author](/articles/legal/license)) Now, we have an input available which we can use to upload a single file. 10. Sending file from Power Apps to Power Automate 10.1. Insert a Button into your Power App. We will use the button to trigger the Power Automate Flow. Adding a button to Power App (Photo by the author](/articles/legal/license)) 10.2. Set(LCL_FILESATTACHED, Last(Attachment_1.Attachments)); Set(LCL_FILELOADRESULT_RAW,TESTREADCSV.Run({ file: { contentBytes: LCL_FILESATTACHED.Value, name: LCL_FILESATTACHED.Name } })); Now, you have connected the Power Automate Flow to your PowerApp. If you did everything right, you can use the output types defined in your Flow right away in your Power App! It works like a standard Colection. CountRows(LCL_FILELOADRESULT_RAW.json) Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Conclusion It’s very much possible to provide your Power App’s users with a *.csv import functionality right inside your application. That said, Power Apps does not offer functionalities out-of-the-box, and some requirements require quite unusual approaches. I personally hope that such a functionality will be added to Power Apps natively. Until then, the way described in this article is the best we have.

Analyze Service Areas of Transit Stations in Tokyo

Fri, 16 Aug 2024 00:00:00 GMT

Data Science and Mass Transit: A dream team. (Image by author, illustrations by Takashi Mifune under free use) With the world becoming increasingly urbanized [1], public transport has become an omnipresent part of urban life. The probably most urban place in the world is Tokyo [2] — a bustling Megapolis of an unmatched size where most people primarily rely on public transport [3] in their everyday lives. This article introduces you to the concept of the Voronoi diagram in an urban planning context and uses it to divide the service area of train stations in Tokyo. We will use the acquired service areas to obtain various, maybe slightly interesting, statistics about the train station’s surroundings. Introduction A Voronoi diagram (Image by author) Voronoi diagrams and Delaunay triangulations find wide application in many branches of science. [4] Voronoi diagrams, also known as Voronoi grids, are used to divide a flat surface into distinct areas that correspond to specific points. This problem is arising frequently in many varieties. [5] Some examples include: - Government of Melbourne (2024-), when they assign students to their nearest school [6] - John Snow (1813–1853), when he related the outbreak of cholera in London to the location of water pumps [4] - René Descartes (1596–1650), when he investigated the distribution of matter relative to fixed stars [4] Today, Voronoi diagrams are being used in many areas, including computer science, geography, and especially urban planning. Urban planning is the field I would like to introduce to you in more detail — we will determine service areas of mass transit stations in world’s largest metropolis: Tokyo. About Voronoi The Components of Voronoi The components of a Voronoi Diagram (Image by author) The Voronoi diagram consists out of multiple different result sets with distinct names and usage: - 🔵 Voronoi Site is the reference location the Voronoi Region is calculated for. - 🟣 Voronoi Region contains all the points on the surface closer to the related Voronoi site than any other. - 🟢 Voronoi Arc is the straight line segment that is a boundary between two Voronoi Regions [4] - 🟠 Voronoi Vertex is a point where Voronoi Arcs intersect. The Distance Functions for Voronoi Comparing various distance functions (Image by author) The Voronoi diagram in a city planning context is usually based on a distance/reference system relationship. The distance is calculated based on certain measures, such as literal distance or travel time. There are multiple methods for calculating said distance: Overview of Euclidean Distance (Image by author) Euclidean Distance. Is the air-line distance between two points in a coordinate system. It is assuming an open space between two points with nothing blocking them in between. It is the most basic way to calculate distance. Overview of Manhattan Distance (Image by author) Manhattan distance. Ignores the air-line distance and introduces a distance function to approximate travel times in a city grid — just like in Manhattan. It is better suited for certain city environments. Overview of Time-based Distance (Image by author) Time-based distance. The most accurate measure, but also the one most complicated to acquire. Additionally, time based distances may cause anomalies, as travel times between areas might not be linear to their distance and therefore could cause the Voronoi Regions to unevenly split up [10] — just like in the example below. Fig. 5. Voronoi diagrams of selected areas for geographical (a), road (b), and travel time distances ©. The wide blue line is the Warta river, while the magenta blocks are locations of bridges [11] Now, we have all the basics for starting with our real life use case — Tokyo’s public transport. Let’s get started! Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Assessing the Tokyo Transit System One of the most important aspects of urban planning are mass transit networks — especially in a place like Tokyo. Tokyo is a place that many people are longing for — especially public transport enthusiasts. Hence, Tokyo provides us with the perfect example to use Voronoi diagrams with a real life use case. Concept Imagine a city map dotted with public transit stations. Each station serves its surrounding area, called a service zone. But how can we determine these zones? To better understand how to achieve our goal of acquiring service areas of public transport stations, I put the Voronoi diagram into our mass transit perspective: A concept on how to use Voronoi diagrams for acquiring service zones (Image by author) - 🔵 Voronoi site is now a train station in Tokyo - 🟢 Voronoi Arc is now the border dividing service areas of train stations - 🟣 Voronoi Region is now the area serviced by one specific train station based on our distance function That’s what Voronoi Diagrams can offer us. By dividing the map into regions based on the distance to the nearest station, we have a very simple approach defining service areas of train stations. Now, as the concept is clear, let’s get started with the actual implementation. Identitifying Tokyo We need to define the area we would like to create our Voronoi Diagram for. That process is usually referred to as defining the Bounding Box. Difference between Tokyo City and Tokyo Prefecture (Image by author) When someone says “Tokyo”, it can mean different things to different people. Some may think of it as a: - Bustling city (Tokyo City) with towering skyscrapers and vibrant street culture, while others may view it as a - Prefecture (Tokyo Prefecture) with scenic landscapes and natural beauty, while someone else might refer to - Greater Tokyo Area, which is the most populous metropolitan area in the world [8], that consists out of multiple standalone cities (e.g. Tokyo, Saitama, Yokohama, ..). However, it’s important to define what we mean by “Tokyo” in order to avoid confusion. Using the prefecture definition allows us to differentiate Tokyo from its neighboring cities like Saitama, Chiba, and Yokohama while including most locations which people generally refer to as “in Tokyo”. Tokyo Prefecture will be our base border box for all subsequent Voronoi calculations. For acquiring geospatial information about Tokyo prefecture, we resort to official government sources, as referenced below: Dataset: National Land Numerical Information | Administrative Boundaries Data (mlit.go.jp), Licensed under the Open Data Policy, allowing commercial use. 利用規約 (mlit.go.jp) Locating Train Stations Visually checking our dataset. I located Takadanobaba in it’s correct position. (Image by author) The Japanese Government provides a reliable source for information about train stations. Their website offers a detailed dataset containing all train stations across the country, along with other related metadata. It is worth noting that the dataset focuses solely on train stations as per Japan government’s definition and may exclude modes of mass transit like metro and monorail while including modes of transit people generally wouldn’t refer to as trains, such as specific cable cars. There might be opportunities to increase the correctness of data retrieved through combining additional data sources. [9] In this article, I will continue with the governmental dataset without further enhancements. Dataset: National Land Numerical Information | Railway Data (mlit.go.jp), Licensed under the Open Data Policy, allowing commercial use. 利用規約 (mlit.go.jp) Obtaining Service Zones Now that we have established the borders of Tokyo and a list of relevant train stations, we can proceed with calculating our service zones. We will be utilizing KNIME, a powerful tool for scientific computations that minimizes the need for in-depth math knowledge or coding, by abstracting much of the complexity away. Click here to learn more about KNIME 1. Extract data We start with connecting our data sources to the tool. Fortunately, KNIME provides a set of tools for Geospatial operations, which we can use out of the box. We proceed by creating two nodes for importing our data. GeoFile Reader node is capable of handling both Shapefile and geojson data types. Data import in KNIME (Image by author) To use the two datasets, some preparation is necessary. - We remove some fields due to an excess number within each dataset with the Column Filter node. - To enhance readability, we rename certain columns within the datasets with the Column Renamer node. - To avoid confusion later on, identical columns within each dataset are given unique names with the Column Renamer node. Data extraction in KNIME (Image by author) Having completed the data extraction and preparation, we can now move forward with our computations. 2. Work with data Our next objective is to obtain the Voronoi polygons for each station, thereby enabling us to derive their respective service zones. - We create a bounding box for our Voronoi diagram, using our Tokyo Prefecture dataset, with the Bounding Box node. - In order to perform our calculations, we need points instead of polygon representations for the stations. The Geometry to Point node is used to convert them. - We perform the Voronoi calculation using the Voronoi (Thiessen) Polygon node, which produces polygons and associated IDs. However, since we also require station metadata, we must join the Voronoi polygons with this information again using the Spatial Join node. The entire workspace in KNIME (Image by author) That’s everything we need to obtain station service zones in KNIME. Let’s take a look on the results. Service Zones (V1) Now we can see that the Voronoi diagram has divided our map, assigning a unique area to each station. One station has multiple service areas — something went wrong. Visualized with QGIS (Image by author) We should keep in mind that certain stations like Takadanobaba look like one station in person, but actually consist of multiple stations in reality. Because of this, we need to do some extra work to make sure our calculations reflect this accurately. Cleaning Data and Service Zones (V2) The station dataset contains an additional ID that groups stations by their public name and/or real-life appearance. By utilizing this ID and the Group By node, we can join the individual stations into a single one. The entire workspace in KNIME (Image by author) After consolidating the individual stations, we were able to create a more accurate dataset of service zones that better reflects how people see stations in the real world. Look at Takadanobaba — it’s located in a single Voronoi region now. Cleaned up dataset: Now we have one service area per station, visualized with QGIS (Image by author) Result We have completed our calculations and obtained service zones that can provide useful insights and statistics. To explore the results yourself, you can find everything in this GitHub repository: Click here to access GitHub repository Next, we will obtain some real-life statistics that we can derive from our service zones. Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Statistics Example — Restaurant Density Finding restaurants isn’t hard in Japan — but where is their highest density? (Illustrations by Takashi Mifune under free use) When it comes to planning a vacation, one of the biggest hassles is figuring out where to stay. I mean, you want to pick a location that is close to all the restaurants, shops and other cool stuff, right? But with so many options out there, it can be confusing to find the perfect spot. That’s where our newly acquired Transit Station Service Areas can help us: Goal: Identifying the station with the most POI in it’s surroundings. To make things easy, we will focus on restaurants. Data valuable for business purposes seems to frequently be protected and hard to retrieve. In the case of restaurants, there is no official source available. The most accurate sources, such as Business Registrations or Google Maps, are either associated with a steep price tag or just aren’t approved for large scale processing usage. The Overpass API offered by the OpenStreetMap foundation, is one of the only sources that offers the data we need. With a simple query in Overpass Turbo, we can acquire all restaurants in Tokyo. nwr[amenity=restaurant](around:60000,{{center}}); out center; The data is being displayed in Overpass Turbo right away and is available for export to a file type of our choice: Overpass Turbo UI (Image by author) We now have a full dataset on all restaurants in Tokyo. It’s coming with it’s limitations, as the data is crowd sourced, not validated and might also be centered around the most popular spots in the city, as that is where people usually contribute to on their platform. As it is the best data available in our case, we continue using it. Dataset: overpass turbo (overpass-turbo.eu), Data licensed under Open Database License (ODbL) Matching Points of Interest (POI) to their service zone To calculate the individual amount of restaurants in a station’s service zone, we need to match the POI to their respective service zones. I have imported the station service zones as a layer in QGIS. That application offers us to calculate node counts within areas out of the box. Counting points inside polygons in QGIS (Image by author) The number of restaurants in each service zone does not meet our requirements, as some large zones have many restaurants — but with long distances to cover in between them. Therefore, we need to develop a new metric to address this scenario. The easiest approach would be to determine the density of restaurants. Formula for POI density For a simple ranking, restaurants/km² per service zone might be a good representation. That way, we can find the service zone with the highest density of restaurants. In some cases, service zones may be small yet have a large number of restaurants, which could inflate their score. However, in our situation, this is not a concern. Small service zones might indicate the presence of another station and more restaurants in close proximity. There is a formula: Formula for calculating POI density where: R = POI density factor in n/km², A = area of polygon in km², n = number of POI We import the dataset with POI Count/service area into KNIME and run our formula for each and every service zone. Calculating POI density in KNIME (Image by author) Finally, we acquired the zones with the highest density of restaurants. Let’s take a look at the results. Result The top 20 station service areas with the highest density of restaurants are: The top 25 stations sorted by restaurant density, visualized with Tableau Public (Image by author) We can also view our results on a map to get more insights: The top 20 stations sorted by restaurant density, visualized with Tableau Public (Image by author) We can see, that the areas with the highest density are appearing in clusters. I did some more research and found out that Tokyo is made out of individual cities (e.g. Taito City, Shibuya City, Chiyoda). These clusters do represent the individual cities Tokyo is made out of in some way — interesting! The high restaurant-density clusters identified (Image by author) It’s important to note that our dataset is crowd-sourced and may not be entirely representative or complete, as it could be biased towards areas that have been particularly well-surveyed. However, based on the data we have, Ueno-Okachimachi Station is the clear winner. The station service zone with the highest restaurant density: Ueno-okachimachi Station If you’re interested in exploring the data further, you can check out the Tableau Public page, where you can interact with the visualization and delve deeper into the results: https://public.tableau.com/app/profile/martin.jurran/viz/Tokyo-RestaurantDensity/Map#1 Conclusion The station service zone with the highest density. Ameya Yokocho is a part of the Ueno-okachimachi station service zone. Voronoi diagrams are more versatile and useful than we often realize. They enable us to uncover insights, such as identifying Ueno-okachimachi station as having the highest restaurant density in Tokyo Prefecture. Even major companies like Uber likely use Voronoi diagrams to efficiently assign drivers to pick-up locations. Their wide range of applications makes them valuable across various industries, especially since they can be computed with minimal resources. I encourage you to explore the capabilities of Voronoi diagrams and see how they can benefit you. By including them into your toolset, you can enhance your data analysis skills and gain access to more insightful statistics. Sources [1] United Nations (2018, September 13), Urbanization, https://www.un.org/development/desa/pd/content/urbanization-0 [2] Demographia (2023, January 24), World Urban Areas 19th Annual, http://www.demographia.com/db-worldua.pdf [3] Demographia (2003, January 1), Where Rail Transit Works, and Why, http://demographia.com/db-htld-rail.htm [4] Vera Galishnikova, Peter Jan Pahl (2018, Mar 15), Constained Construction of Planar Delaunay Triangulations without flipping, https://www.researchgate.net/publication/325582898_Constrained_Construction_of_Planar_Delaunay_Triangulations_without_Flipping [5] Liebling T.M., Pournin L. (2010), Voronoi Diagrams and Delaunay Triangulations: Ubiquitous Siamese Twins. Documenta Mathematica. Mathematics Subject Classification: 01A65, 49- 03, 52C99, 68R99, 70–08, 92–08, http://emis.icm.edu.pl/journals/DMJDMV/vol-ismp/60_liebling-thomas.pdf [6] Government of Melbourne (2024), School Catchment Map, https://www.findmyschool.vic.gov.au/ [7] Wikipedia (2024), Taxicab geometry, https://en.wikipedia.org/wiki/Taxicab_geometry [8] Wikipedia (2024), Greater Tokyo Area, https://en.wikipedia.org/wiki/Greater_Tokyo_Area [9] Public Transportation Open Data Center (2024), Dataset — 公共交通オープンデータセンターデータカタログサイト, https://www.odpt.org/ [10] D.T. Lee, Chung-Shou Liao, Wei-Bung Wang (N/A), TIme-based Voronoi Diagram, http://alumni.cs.ucr.edu/~weiw/paper/VD_highways.pdf [11] Solutions for Planning Smart Hybrid Public Transportation System — Poznan Agglomeration as a Case Study of Satellite Towns’ Connections — Scientific Figure on ResearchGate. https://www.researchgate.net/figure/Voronoi-diagrams-of-selected-areas-for-geographical-a-road-b-and-travel-time_fig5_336071639 Pictograms by かわいいフリー素材集いらすとや (irasutoya.com), © Takashi Mifune

Everything You Need to Know About Graph Databases & Neo4j

Fri, 26 Jul 2024 00:00:00 GMT

(Photo by the author, illustrations by Takashi Mifune under free use) Storing and processing data is an elementary task of software engineering. In the early days of large-scale professional development, relational databases such as Oracle, IBM DB2 and SQL were dominant. Data manipulation systems cannot easily work with structural or relational data, but only with flat data representations. [1] Graph databases try to bridge the gap between relational and flat data representations, while making it easier to access information. [2] The most popular representative of this type of database is Neo4j. [3] Name: Neo4j Software Type: Graph Database (GDB) Initial Release: 2007 Origin: Neo4j, Inc. Target Platform: Cross-Platform, e.g. Windows, Linux, .. Languages: Implemented in Java and Scala, Web-Tools in Typescript, Cloud functionalities in Go Website: https://neo4j.com/ Introduction Transactions of all types in today’s world are becoming increasingly digitally dependent. That is, because there is a significant rise in the usage of cards and e-payment methods in most countries (e.g., Germany). [4] As transactions become more digital, indicators such as IC3 complaint statistics hint that digital criminal activities are increasing as well. [10] TransUnion for example, a vendor for payment processing software, reported a global 149% increase in digital fraud attempts. [5] Without using graph databases, the transactions and relationships of people involved in such activities need to be modeled in a relational way. But, it is substantially easier to model and access relationship data if it is modeled in a more suitable type of database — a graph database. Let’s dive into some example to see where graph databases come in handy. Example 1 — Uncovering the Panama Papers ⚖️ The Panama Papers Neo4j database data model. [13] (Photo by the author) In the year 2016, gigantic quantities of documents were leaked to German journalist Bastian Obermayer from the newspaper Süddeutsche Zeitung. [6] The documents leaked contain clues for tax evasion and money laundry on a scale never seen before. The leak is made up of 2.6 terabytes of data, consisting of 11.5 million individual documents. [7] In order to process and investigate data that is mainly made of relational data between business actors, a graph-based approach is obvious. Mentioned people and companies can serve as 1) nodes while the type and properties of the relationships serve as 2) edges. In the feature image of this section, you see how the Panama Papers were modeled in Neo4j. In order to process and investigate data that is mainly made of relational data between business actors, a graph-based approach is obvious. Mentioned people and companies can serve as 1) nodes while the type and properties of the relationships serve as 2) edges. In the feature image of this section, you see how the Panama Papers were modeled in Neo4j. Example 2 — Property Management ️🛫️ Example on how to model a property management use case in Neo4j (Photo by the author, illustrations by Takashi Mifune under free use) As someone who has spent too much time in an airport can attest, there are countless objects with relationships to each other present at an airport. From planes to terminals to gates and restaurants, each element of the airport is interconnected and relies on a multitude of relationships to function smoothly. That’s what the graph database can support. By mapping out the airport, terminals, planes, restaurants, and more as nodes on a graph, and their relationships as edges, property managers can gain a deeper understanding of the airport’s operations. Example 3 — Train Operations at JR East 🚆 The result of our Cypher query against the example of Tokyo public transport. [14] (Photo by the author) Public Transport involves many different objects, like stations and platforms, that have connections with each other. These connections can have extra details, like the time it takes to travel from one point to another in a public transport network. An example of this kind of network is the train lines and stations in Tokyo. When we load them into Neo4j, we can run queries in its query language, called Cypher to gather information about these objects and their connections. [14] MATCH (source:Station {name: '高田馬場'}), (destination:Station {name: '池袋'}) MATCH path = shortestPath((source)-[r*]-(destination)) RETURN path, reduce(cost = 0, rel in relationships(path) | cost + coalesce(rel.cost, 0)) AS total_cost LIMIT 1 The feature picture in this section shows the outcome of our query. To check if our results are correct, I used Google Maps to calculate the route, and to my surprise, the outcome was the same as the one retrieved with Cypher. It must be easy to implement route optimization algorithms with Neo4j. Viewing Cypher query results in Google Maps: Our output makes sense and Neo4j helped calculating the shortest route. (Photo by the author) If you want to give it a try yourself, I have forked the original repository and included the Cypher file with my request. For the best experience with this example, it is recommended that you have some basic knowledge of Japanese. Enjoy! 🤗 Uri: https://github.com/martinjurran/neo4j-train-route-sample Alternatives to Graph Databases There are lots of scenarios that can be displayed with graphs. These data could also be stored in relational databases — tables per object type with foreign keys modeling their relations to other objects. The data processing could then be implemented with Joins in SQL-Queries. With the trend towards using NoSQL-solutions [8], it became more acceptable to use database variants besides SQL depending on the specific use case. If you’re wondering if a graph database is the right choice for you, take a look at this overview: Important database types with their typical usage and most popular represenatives (Photo by the author)) The Graph Database Neo4j The first and probably the most famous representative of graph databases is Neo4j. The name was intended to be NeoDB, but NeoDB.com was already taken at launch time, so the team decided to name the application Neo4j. - The word Neo is the Latin word for new, which is inspired by the character Neo in the movie Matrix. [19] - 4j means “for Java”. [8, 9] Today, Neo4j is not a Java embedded application anymore, but a piece of history remains in its name. History of Neo4j Self-created timeline of important events in Neo4j’s history (Photo by the author) 2000: Conceptualization. The founders were bothered by a Content Management System (CMS), which was based on a relational database (RDMBS) called Informix. [15]. Implementing their use cases within the CMS resulted in writing a lot of complicated SQL queries, which became increasingly hard to maintain. The founders thought of their data as being connected to each other, one of paths between our related items of content, metadata, tags, and metatags for that content , which finally led to the development of the property graph model. 2000–2002: Graph layer over RDMBS. The first step was to write a graph layer on top of a relational database called Informix. [15] 2002: First Version of Neo4j. The graph layer on top of Informix presented some challenges. The issue was, that Informix wasn’t optimized to handle all the relationships between their graph-alike data. [15] Faced with the challenge of using an RDBMS for connected data, the developers decided to create a new kind of database optimized for connected data. [15] 2007: Invention of the word graph database. The word graph database was invented by Emil Eifrem and is being based upon one of Facebook marketing statements that “we are a utility for the socialgraph”. He combined the words graph and database, and that’s how the word graph database was born. [16] [17] 2007: Founding of Neo4j Technologies. The company was founded, but was primarily involved in consulting, as there was no own product to sell yet. [17] 2007: Graph database became fully native. The prototype became a fully native graph database. [18] It started out as an embedded Java database. [8] 2010: Release of Neo4j 1.0. [18] 2011: Development of Cypher. The first ever declarative query language for property graphs has been created. [18] It is inspired by MS Visio’s placing objects and relationships on an UI and it is therefor a very human querying model. [8] Over the time, Neo4j has evolved from a prototype drafted on a plane ride [21], into a standalone database application, satisfying its customers needs. These needs also reflect in the architecture of the solution, which is covered subsequently. Stakeholders To discover the architecture of Neo4j, it is important to understand the Stakeholders which might influence architectural decisions, as outlined by Rozanski and Woods. [20] The following Power Interest Grid lists the application’s stakeholders and puts them into perspective to visualize their likelihood of influence on architectural decisions. Power/Interest Grid of Neo4j (Photo by the author) Those stakeholders with high levels of both power and interest, such as Neo4j investors [22], Neo4j Inc. themselves, and their partners [23], have significant influence over software architecture decisions. They need to be closely managed. [24] While the needs and demands of enterprise customers are taken into account, their influence is relatively limited as they do not have the same level of investment in the software as other stakeholders. They need to stay satisfied. [24] Competitors may influence Neo4j’s direction indirectly by challenging their market position. Developers usually have the stack set for their product and are not having enough power to influence Neo4j’s architecture individually. They shall be monitored. [24] The open-source community and individual contributors, play an important role in shaping the conversation around the software architecture and discovering bugs. Still, they have limited influence. The right choice is to keep them informed. [24] Dependencies To better understand the architecture that Neo4j is based on, it is important to know its current dependencies and how they are shaping Neo4j into the solution it is now. Dependency context of Neo4j (Photo by the author)) Architectural Goals The table below presents the primary architectural goals of Neo4j, where the order depicts their importance. Now, let’s explore the various techniques used by the software to accomplish these goals. Hitting all the architectural targets (Photo by the author, illustrations by Takashi Mifune under free use) - 🟢 Ease of use. It’s a standalone platform with graph-native data modeling, a user-friendly querying language called Cypher, and thorough developer documentation that’s easy to understand. That’s what’s making it easy to work with Neo4j and all the involved stakeholders are having a good experience. (Developer Perspective) - 🟠 Performance. Neo4j is graph-native database that’s equipped with a variety of optimizations to ensure fast query response times for complex graph data. The platform also boasts high concurrency and consistency features. Additionally, graph algorithms are running very efficiently. - 🔵 Reliability. It’s important for any database platform to be reliable. Neo4j is achieving that through ACID transactions, monitoring, event logging, TLS wire encryption, and permission systems. - 🟣 Security. When data is involved, security is especially important. Neo4j’s achieves security through event logging, TLS wire encryption, fine-grained permission systems, and data-at-rest encryption. - 🟡 Interoperability. Able to operate on all relevant target systems. Easily integrable into existing environments. A huge variety of clients for data access are supported. - 🔴 Availability. Able to operate in clusters, which is increasing availability to meet enterprise demands. - 🟤 Scalability. Scalable both horizontally and vertically to handle increasing amounts of data and query volume. - ⚪ Extensibility. Both Neo4j development team and external community developers can easily extend the solution. That’s valid for both modifying existing components and adding new functionalities. Neo4j is open for new technological and methodical trends. 🟢 Ease of use Most developers have at least a basic understanding of how relational databases work, but graph-based solutions are new to most. For Neo4j’s success, it is crucial that developers can adapt the solution quickly and build working products without any unnecessary barriers. Conceptional and implemented data model for public transport (Photo by the author) Supporting developers in their adaption is, that graph databases usually equal their concept data model. They are schema-free, like most NoSQL solutions. Data is being represented as nodes and edges. Non-developers can explore data through the interactive Neo4j browser. This includes everything from initial ideas on a whiteboard to developing queries in the Cypher language, which is equivalent to SQL. Exploring data in the Neo4j Browser, based on the result of a Cypher query (Photo by the author) Neo4j’s ability to efficiently execute complicated queries with connections over many nodes and edges is mainly due to its dedicated graph data model. Relationships are directly materialized as a data structure in the database, allowing “pointers” to be traversed without many sub-queries and joins. Neo4j is not based on an existing database but is a highly specialized implementation. 🟠 Performance Neo4j is a highly optimized, graph-native database, which is optimized to handle large scale graph operations. Graph queries are executed much faster and more efficiently, since all the related data is stored in one place and reflected as graphs. Also, Neo4j implements several performance optimizations speficially implemented for graphs: - B-trees for fast retrieval of nodes and edges in a graph dataset - Indexing tailor-made for graph data with the option to define multiple indexed on one property ⚪ Extensibility Neo4j offers official Bolt drivers for .NET, Java, JavaScript, Go, and Python. Community implementations are also available for C/C++ and PHP. Plus, it’s possible to access Neo4j’s database via its HTTP API regardless of the programming language you are using. Sometimes developers need to extend the capabilities of the Neo4j database by creating their own procedures or functions. Neo4j has plugins and unmanaged server extensions for HTTP endpoints that allow for this. Documentation is available for implementation in JVM languages. Neo4j provides many ways to extend its functionality and stay up-to-date with trends. Developers can use Bolt and HTTP access protocols through Cypher or create their own plugins to improve the Neo4j software. Recent updates to Neo4j include GraphQL integration and connectors for popular streaming solutions like Kafka and Spark. Additionally, Neo4j offers a Graph Data Science library that provides various algorithms and machine learning modeling options to cover use cases related to data analytics and machine learning. The Big Picture (🟢/🟠/🔵/🟣/🟡/🔴/🟤/⚪) Now that we’ve explored some individual architectural goals and how Neo4j addresses them, let’s take a step back and visualize the big picture. Imagine a graph of interconnected architectural goals, where each node represents a specific goal, such as data consistency or availability, and each relationship represents how those goals are related or dependent on each other: Enterprise architectural target diagram of Neo4j (Photo by the author) Architectural Structure To understand how Neo4j implements its architectural goals in general, we should look at the solution from a more technical perspective. Rough overview about a typical Neo4j setup [22] (Photo by the author)) Advantages of Neo4j for software architects Now you have received all that information on graph databases and Neo4j. But, how can that help you? Utilizing a graph database can have immediate positive effects for you and your work: - High performance for complex queries - Greater flexibility in data modeling - Ability to handle unstructured data - Improved scalability and availability - Easier integration with other technologies - Greater business agility Conclusion Neo4j has gained recognition in modern software architecture due to its capabilities for handling graphs. But today, it is way more than the all-purpose-answer for handling graph data. For today’s software architects, it is important to choose specialised databases for certain use cases, instead of building whole solutions based around one data backend. That is especially valid as microservice architecture still continues to be on the rise. If you haven’t already, I personally encourage you to explore some test use cases to discover the power of graph databases and how they can benefit you. Whether it’s logistics, finance, healthcare, social media, or e-commerce, graph databases can provide insights that traditional databases cannot. Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Sources [1, 2] Silvescu, Adrian & Caragea, Doina & Atramentov, Anna. (2002). Graph Databases. [3] DB-Engines Ranking of Graph DBMS 2024 https://db-engines.com/en/ranking/graph+dbms) [4] Deutsche Bundesbank (2022, Juli 7). Payment behaviour in Germany in 2021. https://www.bundesbank.de/en/press/press-releases/payment-behaviour-in-germany-in-2021-894120 [5] Leonhardt, M. (2021, June 3). Online fraud attempts are up 25% in the US — here’s why. CNBC. https://www.cnbc.com/2021/06/03/why-online-fraud-attempts-are-up-25percent-in-the-us.html [6] Clark, Nicola (April 5, 2016). “How a Cryptic Message, ‘Interested in Data?,’ Led to the Panama Papers”. The New York Times. ISSN 0362–4331. Archived from the original on August 15, 2016. [7] “About the Panama Papers Investigations”. International Consortium of Investigative Journalists. January 31, 2018. Archived from the original on July 24, 2020. [8] Emil Eifrem (2017 Jul 27), Hello, World: Neo4j, Inc. https://neo4j.com/blog/hello-world-neo4j-inc/ [9] Dr. Jim Webber (2022, June 8). A Personal History of Neo4j. Neo4j Inc. https://www.youtube.com/watch?v=YB723cp9jgM [10] Federal Bureu of Investigation (2023). Internet Crime Report. https://www.ic3.gov/Media/PDF/AnnualReport/2023_IC3Report.pdf [11] Gopala Kr (2017). Neo4j Architecture. https://github.com/gopala-kr/10-weeks/blob/master/Projects-Blogs/07-bigdata-databases/neo4j-architecture.md [12] Dr. Jim Webber (2017, May 2), The Engineering Evolution of Neo4j into a Native Graph Database. https://neo4j.com/blog/evolution-neo4j-native-graph-database/ [13] William Lyon (2018, December 3), Graph Visualization of Panama Papers Data in Neo4j, https://medium.com/neo4j/graph-visualization-of-panama-papers-data-in-neo4j-9c08ca17039c [14] ggszk (2020), Neo4j sample database: Tokyo railway routes (in Japanese), https://github.com/ggszk/neo4j-train-route-sample [15] Emil Eifrem (2016, Mar 29), DB-Engines, Informix and Neo4j: An Origins Story, https://neo4j.com/blog/db-engines-informix-neo4j/?ref=martin.jurran.org [16] Emil Eifrem (Unknown), The birth of graph databases: How Neo4j built its product and category, https://neo4j.com/news/birth-graph-databases-neo4j-built-product-category/ [17] Alastair Dryburgh (2007, Mar 22), Growth Stories: The magical power of a name, https://www.forbes.com/sites/alastairdryburgh/2017/03/22/growth-stories-the-magical-power-of-a-name/#49b4ebe56db9 [18] Neo4j Inc, The History of Neo4j — Open Source, Big Community, https://neo4j.com/open-source-project/ [19] Emil Eifrem (2016, Mar 22), Twitter Post, https://twitter.com/emileifrem/status/712327903032188928 [20] Rozanski, Nick, and Eóin Woods. Software systems architecture: working with stakeholders using viewpoints and perspectives. Addison-Wesley, 2012. [21] Emil Eifrem, (2022, August 8), The first code of this database was written in IIT Bombay, https://www.youtube.com/watch?v=Nhi4XwmCh9A [22] Crunchbase, Neo4j Profile, https://www.crunchbase.com/organization/neo-technology [23] Neo4j Inc, Partner Directory, https://neo4j.com/partners/directory/ [24] Latha Thamma Reddi (2023, Apr 14), Stakeholder Analysis using the Power Interest Grid, https://www.projectmanagement.com/wikis/368897/stakeholder-analysis--using-the-power-interest-grid Pictograms by かわいいフリー素材集いらすとや (irasutoya.com), © Takashi Mifune

Strategies for Migrating SharePoint Custom Script Solutions

Thu, 11 Jul 2024 00:00:00 GMT

Throwing out the deprecated Custom Scripting and replace it with something new and shiny (Photo by the author) If you are working with SharePoint Online, you should be aware that Custom Scripting solutions are being deprecated due to security concerns soon. By November 2024, it will no longer be possible to prevent SharePoint from resetting custom script settings to their original value for all sites. The recommended way to deal with Custom Scripting solutions by Microsoft is to migrate them into the SharePoint Framework (SPFx). There's other options. In this article, we will be exploring the migration of custom scripting solutions in SharePoint Online. Immediate Relief Buying Time — Only until November 2024 Custom Script can be enabled or disabled at the site level, and the automatic 24-hour reset can be disabled at the tenant level. While custom scripting solutions are being deprecated, there may be situations where you need to buy some time to get your application up and running again. In these urgent situations, disabling the automatic reset may be a valid option. However, it’s important to understand that custom scripting solutions come with security risks. Read more here: Allow or prevent custom script — SharePoint in Microsoft 365 | Microsoft Learn Decision Flowchart Flowchart supporting to make a decision for migration of Custom Scripting solutions (Photo by the author) For your solution, it might not be the most effective option for migration to continue staying in the SharePoint ecosystem. For instance, there is scenarios where it’s a good any easy decision to change your approach: - (1) Hosting static HTML assets that (a) do not change frequently, or (b) do not change at all. - (2) Single-Page-Applications without data connections to SharePoint or only use SharePoint of authentication. For all other types of custom scripting solutions, migrating to SharePoint Framework (SPFx) should be a top priority. Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate More Resources For those looking to migrate their custom scripting solutions to SharePoint Framework (SPFx), there are a variety of resources available to help guide you through the process. Microsoft has created several hands-on articles that walk you through real-life use cases and provide step-by-step. Migrate existing Script Editor web part customizations to the SharePoint Framework | Microsoft Learn

How to retrieve Office 365 profile pictures with Microsoft Graph in .NET

Wed, 12 Jun 2024 00:00:00 GMT

Inofficial logo of Microsoft Graph (Photo by the author) In some cases, external services need to use employees’ identities - especially profile pictures. There is probably no more central place than Entra ID to acquire profile pictures. Example: We are setting up an external, premade solution that lacks out-of-the-box integration with identity providers. And along with metadata, we may also want to transfer profile pictures into our solution. Luckily, most organizations have an identity store that we can utilize. In our case, we can use Office 365 with its Entra ID. With these tools, we can easily access the necessary data to build our feature. Idea We want to enhance the quality of data in our external software by importing profile pictures directly from Entra ID. We need to figure out how to upload those raw pictures somewhere or post them to the external software’s API, depending on its design. This will require us to think about our data transfer and storage solutions, as well as how to interface with different APIs. Technical Base Retrieving User Photos with MS Graph API The first step would be to take a look at the Photo API of Microsoft Graph, to get a rough understanding of the available functionalities. The API is providing the profile pictures stored in Exchange Online / Entra ID. We have two types of queries: //Request your own photo Metadata: https://graph.microsoft.com/1.0/me/photo Photo: https://graph.microsoft.com/1.0/me/photo/$value //Request someone else's photo Metadata: https://graph.microsoft.com/1.0/users/max.mustermann@contoso.com/photo Photo: https://graph.microsoft.com/1.0/users/max.mustermann@contoso.com/photo/$value Get profilePhoto — Microsoft Graph v1.0 | Microsoft Learn Implementation 1. Retrieve User Photos with PowerShell / native-cmdlets Get-AzureADUserThumbnailPhoto is a cmdlet for pulling data from Entra ID to retrieve a user’s thumbnail photo. It’s the easiest way to utilize above-mentioned API endpoint without implementing anything yourself. Be aware, that the usage of this cmdlet is most likely to be restricted in almost all organizations and that you need to acquire additional rights in order to execute it. I would not recommend using that approach in production scenarios if it were avoidable. Get-AzureADUserThumbnailPhoto -ObjectId "" -FileName "" Get-AzureADUserThumbnailPhoto (AzureAD) | Microsoft Learn 2. Retrieve User Photos with C# In order to use Microsoft Graph, we need to implement an authentication flow, just as covered in this article: Establishing OAuth 2.0 Authentication to SharePoint Online REST Services in .NET | Medium //Authentification public GraphServiceClient GetGraphServiceClient() { //Create a confidential client application with the provided ClientId, TenantId, and ClientSecret IConfidentialClientApplication confidentialClientApplication = ConfidentialClientApplicationBuilder .Create(ClientId) .WithTenantId(TenantId) .WithClientSecret(ClientSecret) .Build(); //Create an authorization code provider with the confidential client application AuthorizationCodeProvider authProvider = new AuthorizationCodeProvider(confidentialClientApplication); //Create a client credential provider with the confidential client application ClientCredentialProvider authenticationProvider = new ClientCredentialProvider(confidentialClientApplication); //Create a GraphServiceClient and configure it with the authentication provider return new GraphServiceClient(authenticationProvider); } //Retrieve profile picture by id public static async Task GetUsersPhotos(string id) { _ = _appClient ?? throw new NullReferenceException("Graph has not been initialized for app-only auth"); try { return await _appClient.Users[id].Photo.Content.GetAsync(); } catch (ODataError odataError) { Console.WriteLine(odataError.Error.Code); Console.WriteLine("Error Message: " + odataError.Error.Message); return null; } } //use above-mentioned functions to save profile picture to file using (var stream = await GetUsersPhotos(user.Id)) { using (var fileStream = new FileStream(sourceFileName, FileMode.Create, FileAccess.Write)) { await stream.CopyToAsync(fileStream); } } Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate Conclusion By using Microsoft Graph, we can efficiently retrieve employee profile pictures in a simple and accessible way. Now, all your applications can display coworkers' profile pictures.

Establishing OAuth 2.0 Authentication to SharePoint Online REST Services in .NET

Wed, 27 Mar 2024 00:00:00 GMT

/// Example class for holding oAuth authentification data ///

Welcome

Fri, 01 Dec 2023 00:00:00 GMT

Support this blog with a donation I am committed to keep this site completely free of paywalls and ads. If you find my content valuable, consider making an donation to help me with the upkeep. Donate This is the first post of my blog. On this blog, you will be provided with frequent updates on personal projects & occasional results of my (hobby) data science research. If you have any questions or concerns, just reach out to me via one of the linked social media platforms. Thank you!