How to check an AI analytics tool's numbers against your own reporting

The test is a diff. Pick a report your team already trusts, ask the tool to produce the same numbers from the same period, and compare every cell. You are not looking for a perfect match. You are looking for whether the differences are explained or explained away.

Every vendor in this category will show you a convincing investigation. A citation, a source link and a chain of reasoning are all things a wrong answer can also produce. Trust here is usually asserted, and the buyer is asked to take the reasoning on faith because checking it properly looks like work.

It is less work than it looks. The artifacts you need already exist, you already own them, and the whole test can be run without the vendor's help.

What should I ask an AI analytics vendor to prove?

Ask for three things, in this order.

The same numbers you already have. Not a demo dataset. A report one of your own people built, for a period that is closed, with values you would defend in a meeting.

The working. Which tables, which date boundaries, which filters, which definition of the metric. A tool that cannot tell you how it got a number cannot be checked, and a number that cannot be checked cannot be used in a decision that matters.

What it does when the evidence disagrees with itself. This is the one most buyers skip, and it is the one that separates tools.

How do I actually run the diff?

Pick a scorecard with real breadth. One metric is not a test. A scorecard that covers several surfaces, say mobile app, mobile web and desktop, across a couple of weeks, gives you enough cells that a systematic problem shows up as a pattern rather than a coincidence.

Then compare, cell by cell:

When we ran this with a large retailer, we took 40 new KPI definitions and checked them against a product manager's existing scorecard. Of 48 values across mobile app, mobile web and desktop, 47 matched, revenue matched to the cent, and one cell differed by a single visit. Separately, the retrieval of their engineering change records was checked against a purpose built extraction script, and both queries returned identical result sets with every structured field matching.

The one cell that disagreed is documented rather than rounded away. That is the point of the exercise. A test designed to produce a clean result is not a test.

What should happen when the numbers do not match?

Three answers are acceptable and one is not.

Acceptable: the definition differs, and here is how. Two teams counting a session differently is the most common cause of a mismatch in enterprise reporting, and it is a finding, not a failure. It is also worth more than the diff itself, because it tells you two parts of your business have been making decisions on different arithmetic.

Acceptable: the source data changed, and here is when. Reclassifications, consent changes and tag migrations all move historical numbers. A tool that spots the change date is doing its job.

Acceptable: we do not know yet, and this is what we are testing. Said plainly, with the open question kept open.

Not acceptable: a confident reconciliation that picks the tidier of two facts. In one investigation at a large retailer, clicks held flat across a break while attributed revenue fell by two thirds. The convenient conclusion was that the whole thing was a measurement artifact with no business impact. We reported both facts unreconciled instead, and carried the revenue question forward as its own open item. It did not recover with the clicks.

If a vendor never shows you a result like that, either their product has never met a messy week, or you are not being shown the messy weeks.

What does a vendor's own defects tell you?

More than their uptime page does.

The same validation exercise that produced the 47 of 48 result also surfaced a defect in our own product: a database column that was missing in production was silently blanking chart data on a share of newly created metrics, intermittently, in a way consistent with a partial rolling deploy. Nobody had noticed, because the failure mode was absence rather than an error message.

Two things follow from that, and they apply to any vendor you are evaluating.

First, ask what their failure mode looks like. Software that fails loudly gets fixed. Software that fails by quietly showing you less than it should is the kind that ends up in a board deck.

Second, ask what they do when they find one. In a separate reconciliation we found that clearing a chat hard deletes it from our database while the traces survive, which means any usage figure taken from the database alone is a floor rather than a total. We now state that every time we quote the number. A vendor who tells you where their own numbers are soft is a vendor whose other numbers you can use.

How long should this take?

A first diff is an afternoon if the scorecard already exists, which it does in every enterprise we have worked in.

Compare that with the alternative. At one large retailer, an earlier measurement problem, where a consent change suppressed about 15% of daily web traffic for six weeks, took their own team 31 days to diagnose across five functions. Orders held steady while traffic fell, so conversion and click through improved on every dashboard. Nothing looked broken. That is the shape of the problem a validation habit catches early, and the reason to build the habit before you need it.

Once the same question is being answered continuously rather than assembled by hand, the realistic steady state is days rather than weeks. For that retailer, question to resolution now runs about seven days down to two.

What does this cost you in review time?

Less than reporting costs you now.

At that retailer, eight domains were each taking roughly four owner hours a week to assemble numbers for review. Over a year that is about 1,664 hours of senior time spent rebuilding reports that can reproduce themselves from the definitions. That figure assumes the hours are fully recovered rather than reallocated, which is the optimistic end; the conservative read is half.

Key takeaways

• The test is a diff against a report you already trust, not a demo.
• Compare every cut, not the total. A matching total can hide two errors that cancel.
• Judge the explanation of a mismatch, not the mismatch. Definitions differing is a finding worth having.
• Ask what the tool does when two pieces of evidence disagree. The honest answer is to report both.
• Ask what the vendor's failure mode looks like, and what they did the last time they found a defect in their own product.
• Run it on us. Both tests above were run against artifacts the customer already owned, and both are repeatable without us.

Key terms

• Diff: a cell by cell comparison between two sources that should agree.
• Denominator problem: a rate that moves because what is being counted underneath it changed, not because behavior changed.
• Scorecard: the existing report a team already uses to run a domain, and the right artifact to test against.
• Data in place: reading the systems a company already has, without moving or copying the data into the vendor's own store.
• Failure mode: how a system behaves when it is wrong. Loud failures are safer than quiet ones.

Frequently asked questions

How accurate are AI generated KPI explanations?

Accuracy of the explanation depends first on accuracy of the numbers underneath it, which is why the diff comes before any judgement of the reasoning. Ask for the numbers and the working, test them against a report you already trust, then judge the explanation.

How do I validate an AI analytics tool's numbers against my own reporting?

Pick an existing scorecard covering several surfaces over a closed period, ask the tool to produce the same values, and compare every cell including each device and channel cut. Investigate every difference rather than accepting a total that matches.

Do AI analytics agents invent causes for metric changes?

Any system that is asked for an explanation can produce a plausible one. The protection is a tool that states which hypotheses it tested, which it refuted, and which remain inconclusive, and that reports two conflicting pieces of evidence as conflicting rather than choosing between them.

Can I audit how an agent reached a conclusion?

You should be able to, step by step: the hypotheses, the queries, the data each one was tested against, and what it ruled in or out. If the working is not inspectable, the conclusion cannot be checked.

Does validating a tool require giving it access to our data warehouse?

No. The tests above read what the tool produces and compare it with what you already have. On access more broadly, an Operator reads the systems you already run and does not move or copy your data.

What if my own two reports disagree with each other?

That is a common and useful result. Two teams counting the same metric differently is worth finding on a Tuesday rather than in a quarterly review.