Prepare experiments from audit findings
Convert an observed problem into a hypothesis, a proposed change, a metric and a review, before anybody builds anything.
- Evidence
- Context-dependent. The reasoning is sound and the outcome genuinely depends on your business, your customers and your production. Worth testing rather than adopting wholesale.
- Difficulty
- Moderate. Difficulty is an initial editorial assessment of how much work an idea usually takes, not a guaranteed estimate for your systems. A change that is small in one storefront can be significant in another because of where the data lives.
The commercial problem
An analysis produces a list of findings. The list becomes a backlog. The backlog gets built. Nobody records what each change was supposed to achieve, so nothing can be evaluated afterwards and the next analysis produces a similar list.
Why it matters
A finding is an observation about the current state. It says nothing about whether changing it helps. Writing the expected outcome down before building is the only cheap way to find out you were wrong, and it costs one paragraph.
Where it applies
- After any storefront review, internal or external
- Before any change justified by user behaviour rather than a defect
- Any change somebody described as obvious
Suggested implementation
- Write the customer problem first, separately from the proposed solution.
- State what you expect to change and roughly by how much, before building.
- Choose the metric in advance, and choose one.
- Set in advance what result would mean the change should be reverted.
- Record the outcome including when it is nothing, which is the most common and least recorded result.
Risks and constraints
- Insufficient traffic makes a formal test meaningless. At low volume this is a documented judgement, not an experiment, and calling it one is dishonest.
- Testing everything is slower than shipping obvious corrections. Defects should be fixed, not tested.
- A metric chosen after the result is available will always confirm the change.
What to measure
- The proportion of changes with a written expected outcome
- How often the result contradicts the expectation, which should not be zero
- How many findings are still open after a review cycle
How this relates to Specify
This is the loop described on the excellence strategy page, at the scale of a single change.
When an engagement is worth it
Useful when the finding is contested internally, or when the metric is genuinely unclear, which is common for high-consideration purchases with long cycles.
Related ideas
Start with evidence
Or start with one of your own enquiries
The complimentary Enquiry Audit takes one real customer enquiry and shows what a structured reading of it produces. It runs on this site and costs nothing.