There's an easy version of this story, and most data leaders have heard it by now. Connect Claude to the warehouse, ask a question in plain English, and get a chart a minute later. No ticket, no queue, no waiting on an analyst. If that works, why would anyone buy another analytics product?
If you lead data or analytics, someone has probably shown you this already. And the demo is real. Claude writes the SQL, runs it, and explains the result.
But a demo answers one question, once, for one person. Running a business on those answers is a different job, and that's where the real decision lives.
Claude really can do the analysis
Credit where it's due. For a one-off look at the data, a general model is genuinely useful. It can plan which checks to run, ask connected tools for evidence, weigh possible explanations, and draft a clear summary for the person who has to decide.
So the question isn't whether Claude can do the work. It's what has to be true around Claude before anyone should act on the answer.
The risk is the number that looks right
A query that fails is easy to spot, and so is a number that is wildly off. The expensive one is plausibly wrong.
A join that counts some orders twice.
A holiday exclusion applied to every week.
Last year's definition of an active customer.
Ask the same question in different words and you can get two different numbers. The figure goes into the leadership review, nobody in the room can tell, and the data team answers for it.
Anthropic's own data team, which runs most of its business analytics questions through Claude, names the same failure. In their words: "The answer is wrong, but looks plausible and is used without objection." One of their safeguards is "explicit human sign-off on anything leadership-bound."
The model is not the weak point. What it has no way to hold is the company's metric definitions, thresholds, default segments, and known causes, unless a person rebuilds that context by hand on every question.
And demos usually run on clean data. On public benchmarks with tidy schemas, models do very well. Researchers at MIT built a benchmark from real private data warehouses instead, with intricate schemas and their own domain knowledge. There, AI agents using GPT-5.2 "achieve only 10.8% accuracy."
Accuracy comes from the definitions, not the model
Teams that get self-service analytics working on a general model tend to do the same three things. People write and approve the metric definitions. Answers are tested against cases whose result is already known. And the definitions are updated every time the business changes.
The numbers behind that are striking. Anthropic reports that without its curated reference files, Claude's ability to answer analytics questions accurately "didn't exceed 21%" on its own evals. With them, accuracy is "consistently above 95% in aggregate." Same model, different context.
Independent research points the same way. In a peer-reviewed study, researchers at GSK found GPT-4's accuracy on their own database rose "from 8.3% with the database schema alone to 78.3% with the business context document."
Writing a definition is quick. Agreeing on it is slower. Finance, merchandising and ecommerce can each have their own version of "net sales", each defensible from where that team sits. No model can pick the right one for you.
Here's a test worth applying to every argument in this space, including ours: does it get stronger or weaker as the models get better?
Stronger. Models really are getting better at SQL, and quickly. But a better model writes better SQL against the same definitions. It cannot decide what your business means by revenue. As better models are used by more people and more tools, one agreed set of definitions and limits becomes more valuable, not less.
One business question is really four
Business questions rarely stop at a number. Take a fictional example.
Between 10:00 and 10:30 one morning, fewer customers complete checkout for fresh produce with two-hour delivery in Phoenix. Checkout conversion falls from 22.1% to 16.3%, and the decline is concentrated on one Android app version. Customers on iOS and on the previous Android version are not showing the same decline.
Across all checkout traffic, that's a soft morning. Inside that one segment, it's a real problem.
From here, the work is four connected questions:
What changed?
Why did it change?
What should we do?
Did that help?
What you learn in one step should guide the next, rather than being reconstructed by another person or another tool.
In the example, the evidence points to a new address-checking feature, and the fix is tested on 10% of the affected group before anything wider changes. The test group recovers 5.1 percentage points. Even then, recovery alone does not prove the cause.
Every new tool will ask the same question
You will not connect just one tool. Count the AI assistants, internal agents and applications that will want access to your data over the next year. Each one can ask what checkout conversion was in Phoenix, and each one can arrive at its own answer.
The same question in two different weeks producing two different numbers is disqualifying in revenue reporting, and it is not fixed by better reasoning, because the ambiguity lives in the data model.
Many teams will also look at the AI built into their warehouse, and they should. It's useful when the question and the context both live in the warehouse. But like any assistant, most of it starts after someone asks the question.
Answering a question is not the same as owning a KPI
Trusted Q&A still begins with a question. Operational analytics begins with a declared goal: these KPIs, these segments, checked every day, whether or not anyone asks.
To be fair, a Claude agent can run on a schedule or fire when a metric moves. The production work is deciding which KPIs and segment intersections it owns, which tests apply, how often they run, what movement is materially important, what gets suppressed, and who receives the resulting incident. Until someone builds that, the Phoenix decline waits for a person to notice and ask. It was buried in one segment, because no one can check every segment at once.
Break the job into its parts and it becomes clear which parts suit a language model and which need something else:
Semantics defines the number.
Statistics finds the signal.
The model ranks explanations.
Policy bounds the action.
Verification measures the outcome.
Only one of those five is a natural job for a language model. The others are better handled by agreed definitions, statistics and rules, with a model helping at the edges.
Detect, Explain, Act, Learn: who should do what
One way to put those five parts in order is a four-stage loop: Detect, Explain, Act, Learn. It sits on a foundation of agreed definitions. Each stage has its own table stakes, and each is best handled by a different kind of tool.
The foundation: agreed definitions. Semantics defines the number: what counts as a checkout session, which order statuses count as revenue, which calendar the comparison uses. These are business decisions, made by people and versioned. A model can propose a candidate, but as the research above shows, accuracy comes from people approving it.
Detect. Statistics finds the signal. The table stakes are coverage and consistency: check every declared KPI across every segment and dimension, and tell an unusual move from normal variation. That suits SQL and statistical or machine learning methods, which are cheap enough to run on everything and return the same answer every time. A language model asked to scan thousands of segment combinations would be slower and costlier, and could answer differently from one run to the next. Coverage should be cheap. Deep reasoning should be selective.
Explain. The model ranks explanations. This is where Claude earns its place. Once a move is detected and scoped, someone has to decide which evidence could explain it, pull that evidence from the systems where it lives (releases, app errors, payment results, stock), weigh the possible causes and say what was ruled out. That is planning and tool use across systems: agentic work that language models do well. The table stakes are about honesty. The evidence comes from defined queries, not from memory, and the explanation states what could not be seen. Reporting only what you found is a story. Reporting what you could not see is evidence.
Act. Policy bounds the action. A model can draft the proposal, but the limits are rules, not reasoning: who owns the change, who approves it, how far it may reach and what triggers a rollback. In Phoenix, the action is narrower than the finding: 10% of one segment, a named approver, a 30-minute window and a stop rule. An action may narrow the scope of a finding, but never broaden it without new approval.
Learn. Verification measures the outcome, and verification means comparison: the group you changed against a group you didn't. The strongest evidence is a holdout or canary, like the 10% test against the untouched 90%. Weaker options, in order, are a matched peer group, a trend against its expected range and a plain before-and-after with its limits stated. Guardrail metrics such as response time and payment approvals are checked at the same time. What held up is recorded, and a person approves which lessons shape the next investigation.
- Foundation. Table stakes: agreed, versioned definitions. Best handled by: people, in a governed semantic layer. Where the language model fits: proposes candidates; people approve.
- Detect. Table stakes: every declared KPI and segment checked; unusual told apart from normal. Best handled by: SQL, statistics and machine learning. Where the language model fits: not the scanner; can summarize what was found.
- Explain. Table stakes: evidence from the right systems; causes ruled out; gaps stated. Best handled by: a language model using tools. Where the language model fits: plans the checks, gathers evidence, ranks causes.
- Act. Table stakes: owner, approver, scope, rollback. Best handled by: rules and an approval workflow. Where the language model fits: drafts the proposal.
- Learn. Table stakes: comparison with an unchanged group; guardrails; approved lessons. Best handled by: statistics, plus human review. Where the language model fits: writes up the result.
One stage is model-assisted. The rest is system work. None of this replaces Claude: Claude can be the reasoning engine, and for the one-off question, the summary or the first draft, an assistant is the right tool and worth keeping. The question is who builds and runs the system around it.
Building it is a project. Keeping it current is the job.
None of this means you can't build it yourself. Some teams do, and capable ones make it work. So assume the strongest realistic alternative: your semantic layer, your dbt models, a frontier model with warehouse access, and a warehouse-native agent, built by a capable team.
The honest comparison isn't the setup. It's the upkeep. Building that once is a project; keeping it current while the business changes is the actual job, and it is the part that does not finish.
Anthropic learned this firsthand. With a dedicated data team and curated definitions, its system reached about 95% accuracy. When nobody kept those definitions current, accuracy drifted "from ~95% at launch to ~65% over a month." Gartner makes the same point more broadly: AI-ready data "is not 'one and done'," and predicts that "through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data."
Before you decide, ask a narrower question: what can your existing platforms already provide, what is worth buying, and what genuinely needs custom work?
Whichever way you go, your team should own metric meaning, importance, access, and approvals. It should not have to build and operate every detector, evidence contract, policy workflow, evaluation harness, and incident-memory service.
Where to start
Start with one decision, not a big rollout. Agree what one number means and which parts of the business to check. Connect the evidence needed to explain a change. Then approve a next step and check the result.
Then ask four questions:
Did you find important changes sooner?
Did you reach a supported explanation with less repeated work?
Did the chosen action improve the affected part of the business?
Could the next investigation reuse something useful?
Record false alarms, unsuccessful actions and side effects alongside the improvements. A pilot that only reports its wins has not been evaluated.
The question is not whether your team can make Claude useful. It is how much of the complete decision process you want your team to build and maintain.
Where Bicycle fits
Bicycle puts this framework to work on your own data and keeps it current as the business changes, so your data team can spend its time on what the numbers mean rather than on the machinery behind them. Each stage above has a direct counterpart.
The foundation. Bicycle starts with numbers teams can defend: approved definitions, source context, evidence, permissions, and a clear record of what was checked.
Detect. Bicycle proactively watches every KPI across every segment and dimension at once, so a move surfaces before anyone asks.
Explain. Bicycle tests business, technical, partner, and external drivers in parallel and returns the likely cause with evidence, including what was ruled out.
Act. Bicycle recommends the next step to the owner, with every action inside governance.
Learn. Bicycle captures accepted causes, outcomes, and reusable context, so the next explanation is faster and more relevant.
Everyone can ask. Data & Analytics controls what gets trusted, published, and acted on. And because the work begins at Detect rather than with a question, the order is reversed: most warehouse AI starts after someone asks the question; Bicycle starts when the KPI moves.
Sources
Anthropic, "How Anthropic enables self-service data analytics with Claude," 3 June 2026. https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude
Chen et al., "BEAVER: An Enterprise Benchmark for Text-to-SQL," arXiv preprint, revised 13 May 2026. https://arxiv.org/abs/2409.02038
Painter, Chalamalasetti, Kassekert and Bate, "Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL," JAMIA Open, 2024. https://arxiv.org/abs/2406.10690
Gartner, "Lack of AI-Ready Data Puts AI Projects at Risk," 26 February 2025. https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk
The Phoenix checkout example and every figure in it are fictional. The research cited describes other organizations' systems and does not endorse Bicycle.
