Multi-Agent Systems for Business Intelligence

Getting answers from company data is broken for most businesses. Analysts spend around 60% of their time finding data instead of studying it. Self-serve reporting tools for business data (BI tools) promised to open data up to everyone. Instead they brought more complexity: dashboards that only their makers understand, database questions (SQL queries) that take days to write, and findings that arrive too late to act on.

The obvious fix is to put a chatbot on top of the data, so anyone can ask questions in plain English. Teams try it, and it fails in a predictable way. The chatbot answers with confidence, the answer sounds right, and some of the numbers are made up. With business data, a confident wrong number is worse than no answer, because someone makes a decision with it.

This post explains why a team of AI agents fixes this, using a system I am designing as the example: Athena. Athena is planned, not built yet (October 2026), so this post describes the design, not results.

Why one model is the wrong shape

A single model asked a business question has to do four jobs at once. It has to work out what is being asked, find the right data, think it through, and check its own work. Models are bad at the last one. Asking a model to grade its own answer while it writes it mostly produces agreement.

The fix is to split the jobs and give each one to a part that does only that. That is what “multi-agent” really means in business. It is not about having many models for the sake of it. It is about splitting duties, the same reason a finance team has one person who prepares the numbers and a different person who reviews them.

The four roles

Athena will have four agents working together, each with its own role. They are wired up with a tool for building agent flows as a map of steps (LangGraph).

The Router sorts each question by what it is after, how hard it is, and which data it needs. A simple lookup goes down a fast path. A hard question goes through the full chain of steps (pipeline). This one choice is meant to make routine questions faster, because most questions do not need the heavy machinery.

The Retriever finds the right data, from both tidy tables and loose documents. It runs two kinds of search at once. One is search by meaning, using a database of number-based fingerprints for text (vector search in Qdrant). The other is exact keyword search, using a classic word-scoring method (BM25). Each covers the other’s blind spot. Search by meaning understands loose, everyday wording but can miss exact matches. Imagine a user asking for “Q3 2024 revenue for product SKU-4421”. A search that only looks at meaning (semantic search) might return similar products. Keyword search catches the exact product code (SKU). The two result lists are then merged by rank.

The Analyst takes the data found and turns it into insight, using a large OpenAI model (GPT-4o). This is not just a summary. The agent looks for trends, flags anything unusual and makes comparisons. Every claim is tied to specific data points, with sources.

The Validator checks the Analyst’s work against the source data. It checks the numbers, flags claims that are not backed up, and gives each answer a confidence score. This agent is the reason the answers can be trusted. It catches answers that sound right but are not supported by the data (hallucinations) before users see them.

Because the Validator is a separate step with a separate job, it can be strict in a way that a model checking itself never is.

Why a fixed map beats a free-for-all

A common pitch in this space is a crowd of agents that work out among themselves who does what. I chose LangGraph over other agent tools, such as two popular team-of-agents frameworks (CrewAI, AutoGen), for one main reason: it keeps clear track of where the work is up to.

LangGraph lets you draw the flow as a fixed map of steps. Each step passes along data in a known shape, branches follow clear rules, and progress is saved as it goes. When the Analyst fails, you want a retry rule that does the same thing every time (deterministic), not an agent deciding for itself what to try next. Live systems need failures you can predict. When something breaks at three in the morning, you want to say exactly which step failed and what the system did about it.

The trade-off is that you design the flow up front instead of letting agents make it up. For business questions that is the right trade, because the flow is mostly known: understand, find, analyse, check.

Two kinds of search are not optional

Teams new to finding documents for an AI to answer from (retrieval) often start with search by meaning alone, because it demos well. It handles vague questions beautifully. Then someone asks about a specific product code, date or account number, and it returns something similar instead of the thing they asked for.

Business data is full of exact codes and numbers. Any system that answers questions about it needs both kinds of search. That is why the Retriever uses both Qdrant and BM25 instead of picking one.

You cannot fix what you cannot see

A team of agents has more places to go wrong than a single model call. So a clear view of what the live system is doing (observability) is part of the design, not an extra. Every step in Athena will be recorded from start to finish with a tracing tool (LangSmith), showing, per agent and per question: how many word pieces it used (tokens), how relevant the data found was, how often the Validator rejected an answer and why, and how long each step took (latency).

That is more than logging. It is a structured view that feeds back into improving the system. When search quality drops for one area of data, I know within hours, not weeks. If you want to build that layer on its own, the companion post on monitoring and evaluating production AI covers it.

What to measure

When Athena is built, three numbers will matter: how often the data found is relevant, how many made-up answers the checker catches, and how long a typical answer takes. The most telling is the second. Catching nearly all made-up answers is what makes the rest usable, because analysts will only rely on a system that rarely lies to them.

When you do not need this

A team of agents means more moving parts to build and run. If your questions are narrow, your data is small and a mistake is cheap, one well-instructed model may be enough. The case for several agents grows as three things grow: the variety of questions, the size and mess of the data, and the cost of a wrong answer.

The short version

Do not ask one model to understand, find, think and check itself. Split the work into a router, a finder, an analyst and an independent checker. Connect them with a fixed map of steps. Search by both meaning and keywords. And record everything. You get answers people can act on.

The full design, with the technical choices behind each part, is in the Athena case study. If you have a similar problem, book a call and we can look at your data together.