Finding a broken checkout that the app's averages were hiding

An online pet pharmacy saw prescription orders on its mobile app fall 30% in under three weeks. Web fell about 10% over the same weeks, so roughly two-thirds of the drop was specific to the app. Shoppers were still reaching checkout. Fewer of those checkouts were turning into orders. The break started the week a mobile app release shipped, and it got worse when a production payment error began rejecting some saved-card orders submitted from the app. The payment bug was hotfixed.

The problem

Prescription orders on the mobile app stepped down at the start of its fourth quarter. The question came in the usual form: is it user behavior, app performance, or something outside the business?

Every obvious place to look gave a reassuring answer. Checkout-to-order conversion across the rest of the app stayed in the mid-60s through most of the period. iOS and Android moved together. Subscription attach held steady. Several pharmacy promotions had just started. Nothing on the standard dashboards looked broken, because the failure only affected one product line inside one channel. On those dashboards, prescription checkout on the app was averaged into the rest of the app.

How Augmeta helped

The Operator took the question as asked, then tested 11 hypotheses against each other. It confirmed 4, ruled out 6 and left 1 open.

The chain:

1. The drop was app-specific. App prescription orders fell about three times as fast as web prescription orders over the same windows.
2. It was a conversion problem, not a traffic problem. Checkouts started on the app held up. What collapsed was the last step. App prescription checkout-to-order fell about 570 basis points, from roughly 49% to 43%, starting the week the app release shipped. In that window the app's non-prescription checkout rate was flat.
3. A second break deepened it. About ten days later, the app prescription checkout-to-order rate fell again, to about 35%. That second fall lined up with an open critical production bug: the payment service was returning errors on order submissions from the app only, with saved-card checkouts failing intermittently. The fix had been held behind a code freeze.

The six hypotheses it ruled out:

• Seasonality and promotions. The same weeks the year before rose about 11%, and pharmacy offers had increased, not ended.
• Fewer visits. Lower traffic explained only about 30% of the drop. The other 70% came from fewer orders per visit.
• Outside events. There was no outage, recall or shortage, and a widespread iOS crash had been fixed the same day it appeared.
• A single operating system. iOS and Android moved together, so it was not a one-platform rollout.
• App checkout latency. A known checkout-speed issue did not explain the break.
• A subscription defect in the release. A known auto-reorder defect from testing did not match: subscription and auto-reorder activity held steady or rose after the release.

The one it left open was whether part of the drop was a measurement effect, since the order system showed a smaller decline than web analytics.

What changed

The triage came back with one priority: treat this as a live checkout defect, not a demand problem, and don't wait for the normal release schedule.

The payment bug was hotfixed.

The first break was handed to engineering with a release and a date attached. The conversion break that started with the app release was isolated to the prescription checkout path. The specific change behind it was not identified within the triage.

An order drop that looked like softer demand turned out to be a broken step in one checkout path, with a release date and a production bug attached. Order-system data showed a smaller decline than web analytics did, so part of the app drop may be a measurement effect. The case study does not present the full figure as lost revenue. What it does show is how fast the diagnosis came and how precisely it located the failure.

See how AI Operators run retail and ecommerce KPIs.

Frequently asked questions

Why didn't existing dashboards catch it?

Checkout conversion across the rest of the app stayed steady, and both operating systems looked normal. The failure was limited to prescription checkout on the app, which the standard views averaged in with everything else.

What caused it?

App prescription checkouts stopped converting to orders starting the week a mobile app release shipped. About ten days later, a production payment error on app order submission made it worse by causing some saved-card checkouts to fail.

Has the revenue effect been measured?

Not yet at the time of writing. Order-system data shows a smaller decline than web analytics does, so the recoverable amount is being confirmed against order-system data rather than taken from the analytics figure alone.