Experiment Analysis Framework: Metrics, Checks and Decisions
Summary: Most experiments fail not because the wrong statistical method was chosen but because metrics were never tiered, sample size was never calculated, and results were read by looking only at significance. This article gives an executable framework in three stages: metrics, checks, decisions.
Key Takeaways
- Tier metrics into three layers: guardrail, primary, explanatory. Watching only the primary metric lets short-term gains borrow against long-term value.
- Derive sample size from the minimum detectable effect, not from "run it two weeks". Without an MDE you cannot judge whether the experiment is worth running.
- Three checks precede reading the result: clean assignment, balanced split, uncontaminated metrics. Fail any and the statistics are meaningless.
- Significant does not mean worth shipping. Read effect size, affected population and cost before deciding.
- Write decision rules in advance. If you do not, you will find reasons for whatever result appears.
1. Building the metric ladder
Metrics are not a pile of numbers; they form a hierarchy.
Top layer: guardrails. They prove the experiment caused no structural harm — crash rate, complaint rate, refund rate, page load time. Guardrails seek no improvement; they set red lines, and a breach stops the test.
Middle layer: the primary metric. One experiment should have exactly one, tied directly to the hypothesis. With several primaries, one will always turn out significant.
Bottom layer: explanatory metrics. They do not drive decisions; they explain why the primary moved — click-through, dwell time, step completion, zero-result rate.
There is a fourth layer that gets skipped: counter-metrics. Some changes lift A while lowering B, and if B is unmonitored the damage appears after rollout. List counter-metrics at design time, not afterwards.
The practical way to build the ladder is to write "what are we validating" on a whiteboard first, then derive the metrics. Doing it in reverse — listing every available metric and choosing among them — almost guarantees a contradictory set.
How do recommendation and growth metrics turn into daily actions?
See search and recommendation2. Sample size and the minimum detectable effect
This step decides whether the experiment means anything. Too small a sample makes every result noise.
Estimation needs four inputs: baseline, smallest expected change, significance level, statistical power. The first two are business inputs; the last two are conventions.
The MDE can be set two ways. From business value: the smallest change that would justify the work becomes the MDE. From historical variance: if similar past changes moved within a range, take the lower bound of that range.
Combining both is steadiest. Business value alone can produce an MDE that is statistically impossible; historical variance alone can produce one that is commercially meaningless.
After computing the sample size, check whether traffic supports it. If not, the right choices are narrowing scope, extending duration with a stratified design, or not running the experiment at all and using observational analysis instead. Forcing an underpowered test yields the illusion of improvement.
One overlooked factor: metric correlation. When several metrics are highly correlated, sizing on one leaves the others underpowered. Identify that at design time.
3. Three checks before reading the result
Before any statistical test, run three engineering checks.
Clean assignment. Were users randomised correctly, is any user in both groups, did caching or login state cause drift? Contaminated assignment means differences may come from the users rather than the change.
Balanced split. A 50/50 design often lands at 48/52. Small deviations are tolerable; large ones usually indicate broken assignment logic that must be fixed before analysis.
Uncontaminated metrics. Duplicate event reporting, cross-device attribution, timezone misalignment and event delay all distort metrics. Contamination usually appears as a sudden large swing in one metric rather than a smooth small change.
The output should be a reviewable record: assignment method, sample sizes, ratios, anomalies. Without that record, no later conclusion can be rechecked.
A practical suggestion: turn the three checks into a fixed script that runs automatically per experiment. Manual checks miss things; scripts do not.
4. Reading the effect: from significance to practical meaning
Statistical significance says the difference is unlikely to be random. It does not say the difference is worth doing.
Read in this order.
Direction and magnitude. What is the effect size, and is it in the expected order of magnitude? A significant but tiny effect rarely justifies rollout.
Confidence interval. A wide interval means high uncertainty; even a decent point estimate can vanish after rollout. When the interval crosses zero, write "no clear difference observed", not "slightly improved".
Segment differences. An overall lift may come from one segment while others decline. Segments must be pre-specified; slicing afterwards invites false positives.
Time trend. Does the effect decay during the test? Many changes look strong in week one and return to baseline in week two, meaning the effect came from novelty rather than value.
Only after all four does "ship" or "drop" become a real decision. Reading only the first is the most common error in experiment analysis.
5. Turning results into reusable rules
The decision table should exist before the test starts.
| Outcome | Rule |
|---|---|
| Primary significant, guardrails clean, effect above MDE | Ship to a wider cohort with a staged ramp |
| Primary significant, guardrails breached | Stop; investigate the guardrail before any relaunch |
| Primary flat, explanatory metrics improved | Do not ship; diagnose why the improvement did not convert |
| Primary improved, effect decays over time | Ship only if the decayed level still beats baseline |
| Inconclusive, interval wide | Extend or redesign; do not declare a win |
Two disciplines make this durable. First, write the conclusion in the form "in context X, at scale Y, change Z moved metric M by N", so the finding is reusable and bounded. Second, log every experiment with its MDE, duration and outcome; after a year that archive is worth more than any single result.
Three frequent misjudgments: treating a p-value as the decision (it is a filter), analysing segments after seeing the aggregate (guaranteed false positive), and running the same test repeatedly until it turns significant.
FAQ
Q: How long should an experiment run?
At least one full business cycle, and long enough to reach the computed sample size. Stopping early because "it looks significant" invalidates the result.
Q: Can we run several experiments at once?
Only if they do not overlap in traffic or in metrics. Overlapping tests make attribution impossible.
Q: What if traffic cannot support the required sample?
Reduce scope, raise the MDE to what traffic can detect, or switch to observational analysis with a stated causal caveat.
Q: Should we trust a significant result with a tiny effect?
No. Read the interval and the cost of implementation; a 0.2% lift rarely pays for the engineering.
Q: How do we handle a metric that moved but was not pre-registered?
Treat it as a hypothesis, not a finding. Re-test it in a separate experiment.
Related Reading
- Automated analytics reporting
- How to monitor AI brand mentions
- Content freshness optimization for AI search
Data Sources
- Creating helpful, reliable, people-first content — Google Search Central — the verifiability standard that applies when experiment results are published.
- 2025 AI Index Report — Stanford HAI — public statistics on how enterprises apply data and AI.
- AI features and your website — Google Search Central — how measurement conventions shift when AI surfaces intervene.