In 2025, Monte Carlo published an account of an internal investigation showing how they mistook a data failure for a model problem.
Its troubleshooting agent identified the effects of a data warehouse incident but missed the code change responsible. The team spent days reviewing reasoning, prompts, configuration and sub-agent calls. Nothing explained the mistake.
As it finally turned out, the actual cause was a silent failure of an internal data export that left the relevant GitHub pull request out of the information available to the agent. Once the team restored the missing data and ran the agent again, it correctly identified the root cause.
It’s a useful reminder of something easy to miss when troubleshooting AI agents: an agent’s final response tells you that something went wrong, but it doesn’t tell you why or where.
How many apparent AI failures actually begin in the systems supplying the information?
Key Takeaways
- An agent’s final response shows that something went wrong, not why or where.
- Data that works for analytics can fall short for agents, which interpret and act on information without waiting for human review.
- Usable data for AI agents needs five things: identity and completeness, appropriate freshness, business meaning, provenance, and permitted use.
- Diagnose failures by starting with the verified outcome and tracing the workflow backward to the source.
- Fix the failing dependency, establish reference outcomes, and test before expanding agent autonomy.
Why data that works for analytics may fall short for AI agents
The difference between traditional analytics and AI agents isn’t necessarily the data they use, but what happens to that data once it’s retrieved.
In reporting, information is usually prepared for people to review and interpret. An AI agent may have to do the interpreting itself, pulling records from different systems, applying business rules, and deciding which action to take next. That leaves more room for things to go wrong.
Information can change between requests, definitions may differ across systems, and an error can carry over into subsequent actions without anyone noticing.
Of course, reporting and analytics face many of these challenges too; the difference is that an agent may act on the information without waiting for human review.
Existing data infrastructure can still support these workflows, but it needs to meet the requirements of the particular tasks agents are expected to perform.
What usable data for AI agents means
In practice, data usable for AI agents needs to provide five things:
- Clear identity and completeness: The information needs to be available and linked to the right customers, transactions or other entities.
- Appropriate freshness: Data needs to be current enough for the decision at hand, with timestamps that show when it was last updated.
- Business meaning: People can recognize when systems use the same term differently. Agents need those business rules made explicit and available to them.
- Provenance: Teams need to know where information came from, how it was processed and what changed before the agent received it.
- Permitted use: An agent should have enough authorization to complete the task, and no more.
What This Looks Like in Practice
An “active customer” might be defined by account status in a CRM and by payment activity in billing. Neither definition is necessarily wrong, but an agent combining the records needs to know which one applies.
This is where a shared semantic layer can help. In a KMS’s connected-vehicle data platform project, the team moved business logic out of dashboards and into Snowflake. Reporting tools and a natural-language query interface could then use the same prepared data. It wasn’t an autonomous-agent deployment; it illustrates why business meaning should be available beyond the application that originally defined it.
That’s one part of building an AI-ready data foundation. The next question is how to tell which dependency failed when an agent gets something wrong.
How to tell if the data, the agent or the tools caused the failure
Even when the data is complete and accurate, the agent can still get things wrong. It might misunderstand an instruction, choose the wrong tool, or fail to coordinate its actions. From the final answer alone, these mistakes can be hard to distinguish from missing data or a failed tool call.
So, how do you tell?
Anthropic offers a useful framework in its guidance on agent evaluations:
Three Different Things
What an agent says it did, what its execution trace shows, and what actually happened in the system are three different things.
An agent might report that it updated a customer record, for example. Its trace may confirm that it called the appropriate API. But if the request failed, the record remains unchanged. The only way to confirm success is to check the result in the target system.
The same principle applies to diagnosing failures: start with what actually happened, then trace the workflow backward to establish where things went wrong.
| STEP | QUESTION | WHERE TO LOOK |
|---|---|---|
| 1. Verified outcome | Did the intended action happen, or did the agent only report that it had? | Target system state, transaction records |
| 2. Agent behavior and tool execution | Did the agent interpret the evidence correctly and call the right tools with valid arguments? | Execution traces, tool calls, returned responses |
| 3. Retrieval and context | Did the agent receive the correct records, in the right form and at the required time? | Retrieved context, query results, timestamps |
| 4. Business meaning and permissions | Did it have the right definitions and permission to use the information or take the action? | Semantic layer, business rules, access policies |
| 5. Source and delivery | Did the relevant data exist, and did it reach the systems the agent relies on? | Source records, pipeline logs, export and integration status |
Take an agent that uses the wrong definition of an active account. If the right definition never reached it, check retrieval and context. If it received the definition but applied it incorrectly, examine reasoning or instructions. A semantic layer can establish the reference meaning, but can’t guarantee the agent will use it correctly.
In Monte Carlo’s case, comparing source records with what reached the agent could have pointed toward the failed export sooner. Data monitoring captures upstream processing; agent traces capture what followed. Diagnosing the incident means connecting them.
Repair: Fix the failing dependency before redesigning the platform
Monte Carlo resolved its problem by restoring a missing export, without changing the agent. But not every failure has such a simple fix; sometimes, the issue lies in how data moves between systems.
Consider KMS’s intermodal transportation project. Data from aviation, rail, and maritime systems came in different formats and at different intervals. Making that information usable required connecting batch and streaming pipelines through Databricks, with processing layers that allowed teams to trace changes back to their source.
The scope of the work will depend on what’s failing, but the approach to repairing an agent workflow should follow three steps:
The Repair Sequence
- Fix the dependency that failed. That might mean repairing a data pipeline, correcting a business definition, or changing an access rule.
- Establish reference outcomes. Use tasks with known correct results to verify the change and detect regressions.
- Test before expanding autonomy. Check incomplete data, changing conditions and failed tools before moving from recommendations to approved actions and, where appropriate, independent execution.
How to assess if your data platform is ready for the next agent workflow
Rather than asking if your whole platform is “AI-ready,” start with the work you want the agent to perform:
Four Questions per Workflow
- Dependencies: Which sources does it need, and what happens if one returns only part of the data?
- Freshness: How recent must the information be, and can the agent see when it changed?
- Access: What can it read or modify, and on whose authority?
- Ownership: Who is responsible for each source, definition and permission when something fails?
The answers may point to a missing timestamp, weak monitoring or a larger integration problem.
Conclusion: Diagnose the dependency before changing the model
An unreliable agent doesn’t automatically mean a faulty model or an inadequate data platform. The problem could lie anywhere between the original data source and the agent’s final action.
That’s why dependable AI workflows start with understanding the data infrastructure behind them. Some problems call for better pipelines, clearer business definitions, or tighter access controls. Others may require changes to the agent itself.
The important thing is to know the difference before investing in a fix.
If you’re planning to put another agent into production, talk to KMS Technology about the data dependencies and potential failure points behind your priority workflow. Establish what needs attention before deciding how much needs to change.
FAQ
Why can an AI agent give a confident answer from incomplete data?
Incomplete data often looks complete. A database or API can return a valid response with no sign that a record is missing, so the agent has no reason to question it. It builds an answer that seems reasonable but leaves out the fact that would have changed the conclusion.
What makes data ready for AI agents?
Data for AI agents needs to be complete, correctly linked, and current enough for the task. Agents also need access to business definitions, information about where the data came from, and appropriate permissions. Without these, even a capable model may produce unreliable results.
Can AI agents work with existing data platforms?
Yes. There’s no need to rebuild your data infrastructure simply because you’re introducing AI agents. The question is if your existing systems can provide the information, context, and access controls required by a specific workflow. Some gaps may need targeted engineering changes, while others won’t require any.
How can you tell if an AI agent failure is caused by data or the model?
Follow the data from origin to action and stop at the first step that fails: source and delivery, retrieval and context, meaning and permissions, reasoning and tool execution, and verified outcome. A passing data check doesn’t clear the model, so test upstream quality and agent behavior separately.
What should companies check before deploying AI agents into production?
Start with the workflow itself. Identify its data sources, freshness requirements, business rules, and access boundaries. Then test the agent against known outcomes, including cases with missing information and failed tools. The goal is to establish where things could go wrong before giving the agent more autonomy.
Does KMS have experience integrating fragmented data sources for AI?
Yes. In an intermodal transportation project, KMS and Addepto integrated more than ten data sources across aviation, rail, maritime, and external systems. The project wasn’t an autonomous-agent deployment, but it demonstrates the data engineering work that can support such workflows.
TAGS
