How to test a product manager's theory about why a metric moved

To test a theory about why a metric moved, test the mechanism, not the attribution. Write the theory so it can be proved wrong, work out what would have to be true if it were right, check that against numbers you can query in minutes, and check the premise the theory rests on before you trust it. At one large retailer, two single-query checks ruled out a product manager's theory the same day. The same method then found 88% of the real decline in the one place nobody was looking.

Why do good theories about a metric stall?

A product manager usually has a theory before anyone asks for an analysis, and the theory is usually reasonable. The trouble is the form it arrives in. "My conversion rate is reading low because of how one traffic source is counted" is a question about attribution, and attribution is often the one thing the data cannot settle.

That was the situation at a large retailer. A product manager for the mobile app asked whether push-notification traffic was inflating visits and making her conversion rate read low. It is a good hypothesis. It is also the kind that costs an analyst three days and usually comes back "inconclusive." Push traffic really was mis-attributed in the retailer's analytics, with nearly all of it landing in the direct channel. An analyst starting from her framing would have spent the three days and returned exactly that: the question, as asked, was not separable.

What is the difference between testing attribution and testing the mechanism?

Testing attribution asks which visits came from which source. When the source data is wrong, that question has no answer. Testing the mechanism asks a different question: if the theory were true, what else would have to be true, and can I check that quickly?

How do two checks rule out a theory?

Both checks took one query each, against the retailer's own analytics, and anyone with access can repeat them.

The size check. The app's share of all visits had moved by less than a tenth of a point week over week. At that weight, no mix effect of the size the theory needed was even available. If a source is a tiny part of the traffic, it cannot move the whole rate much, however badly it is counted.

The shape check. If the app were deteriorating faster than the web, site-wide top-of-funnel conversion would fall by more than web-only conversion, and the gap would be visible. Site-wide conversion had fallen by almost exactly the same amount as web-only conversion, a gap of a hundredth of a point. The mechanism was not there.

Neither check needed the attribution problem to be solved. That is the point of testing the mechanism: it sidesteps the part of the data you cannot trust.

What do you do with the premise?

Check the numbers the theory takes for granted. Her premise had been that orders and visits kept pace with each other. They had not. Conversion was down about 3% on visits down about 3.5%, which means app orders had in fact fallen by roughly 6%. A theory can be refuted and its premise still be worth more than the theory, because the premise was telling her something about her own business that she did not know.

Where do you look when the theory is refuted?

At the movement itself, cut by surface. Once the push theory was ruled out, the question became where the decline actually was. Desktop accounted for 88% of it, and it had been hit twice: its conversion rate had slipped, and its share of web visits had slipped by about a point.

The year-over-year picture inverted as well. Desktop conversion was actually up close to a full point on the year, while its share of visits had fallen from roughly 23% to 17%. Traffic reallocating from the best-converting surface to the worst was swamping a genuine rate gain. A blended rate can fall while a segment improves, because the mix underneath it moved. The full case study has the detail, and Why did our ecommerce conversion rate suddenly drop? lists the other causes to rule out, in order.

What should a good answer sound like?

It should say plainly what was ruled out, what was found, and what is still open. A refuted theory is reported as plainly as a confirmed one, and "inconclusive" is an allowed answer when the evidence does not settle it. A release or a traffic change that came before a decline is not proof it caused it, so a good answer says which of the two the evidence supports.

At that retailer, question to resolution went from about a week to two days, counting the human steps on either side of the analysis. The investigation itself runs in minutes. The product manager got her answer the same day, and it redirected her from a dead end to a live one.

What is the method in five steps?

1. Write the theory as a claim that could be wrong.
2. List what else would have to be true if it were right.
3. Check the size: is the thing big enough to move the number?
4. Check the shape: does the pattern match what the theory predicts?
5. Check the premise, then cut the movement by surface to find where it really is.

Augmeta calls the practice of running KPIs this way Agentic KPI Ops: an AI Operator takes ongoing responsibility for a KPI, investigates what changed, and checks whether a theory holds before it explains anything, with people setting the goals and guardrails.

Key takeaways

• Test the mechanism, not the attribution. When the source data is wrong, ask what else would have to be true and check that.
• Check the size before anything else. A source that is a tiny share of the traffic cannot move a blended rate much.
• Check the premise. The numbers a theory takes for granted are often the most useful thing the investigation finds.
• When a theory is refuted, cut the movement by surface. A blended rate can fall while a segment improves.
• Say what was ruled out and what is still open. "Inconclusive" is an allowed answer.

Key terms

• Attribution: assigning a visit or an order to the channel or source it came from.
• Mechanism: the chain of events a theory says produced the movement, and the other numbers that chain would have to move.
• Mix shift: a change in the share of traffic coming from different surfaces or channels, which can move a blended rate without any surface changing its own rate.
• Premise: something a theory takes for granted, such as orders and visits keeping pace with each other.
• AI Operator: an AI agent that takes ongoing responsibility for one KPI.
• Agentic KPI Ops: the practice of running business KPIs with AI Operators that carry the work from a change in the number through to an action and a check on it, with people setting the goals and guardrails.

Frequently asked questions

How do you test a theory about why a metric moved?

Test the mechanism, not the attribution. Write the theory as a claim that could be wrong, list what else would have to be true if it were right, and check that against numbers you can query quickly. Then check the premise the theory rests on.

What if the theory cannot be tested because the data is mis-attributed?

Test the mechanism instead. At one large retailer, push traffic was mis-attributed and nearly all of it landed in the direct channel, so the question as asked was not separable. Two checks that did not depend on attribution, the size of the source and the shape of the pattern, ruled the theory out.

What is a mix shift?

A change in the share of traffic coming from different surfaces or channels. It can move a blended conversion rate without any surface changing its own rate. At that retailer, desktop conversion was up close to a full point on the year while its share of visits fell from roughly 23% to 17%.

Is "inconclusive" an acceptable answer?

Yes. A good investigation says plainly what was ruled out, what was found and what is still open, and it reports a refuted theory as plainly as a confirmed one.

How long should a question like this take to answer?

At one large retailer, question to resolution went from about a week to two days, counting the human steps on either side of the analysis. The investigation itself runs in minutes, and the product manager in this example had her answer the same day.

Does Augmeta train models on our data?

No. Augmeta uses foundation models through their APIs and does not fine-tune or train them on customer data. Each organization's data is strictly isolated.

How do I see an Operator test one of our theories?

Pick a theory your team is arguing about and book a demo. We will show you how an Operator would test it.

Next step: If a theory about one of your numbers keeps going round in circles, bring it. Book a demo and we will show you how an Operator would test it.