Rollout & Reviews
Part of Retail technology performance reviews
Comparing customer outcomes before and after deployment
Define a shopper outcome, compare eligible cases across periods and report what the change can and cannot show.
Compare a customer result the technology was meant to improve, using the same definition, eligible population and measurement point before and after deployment. Report it as an observed difference unless the comparison can credibly separate the technology's effect from other changes in the store.
Choose an outcome a shopper can experience
Start with the original service promise. A stock tool might aim to give shoppers a dependable answer about another size; a collection tool might aim to cut the time from a customer's arrival to the correct parcel being handed over. Screen views, button presses and sign-ins show activity, not either result.
Specify the unit of analysis: an eligible request, a collection visit or a shopper. Define the start, finish and unsuccessful outcome. For a collection visit, include someone who leaves without the parcel, not just completed handoffs. If one person makes two visits, decide in advance whether both count.
Use a small set of measures that can be checked together:
- Outcome:the proportion of eligible cases ending in a useful, correct result.
- Timeliness:time to a result, with incomplete cases and time until exit shown separately.
- Experience:a consistently asked question or observed need for assistance.
- Possible harm:wrong answers, repeat visits or customers unable to use the new route.
These are choices for a review, not a required scorecard for every technology.
Make both periods comparable
Record the stores, dates, trading hours, entry route, staffing, stock position, promotions and service rules in each period. Use the same question wording and collection method for feedback. A survey offered only to successful users after launch cannot be compared fairly with one offered to all arrivals before launch.
Keep the denominator visible beside every rate. If a new system records requests that staff previously handled informally, the count may rise without a rise in underlying demand.
Check a sample of records against what happened at the counter or on the shop floor, including whether the recorded answer was correct. Treat missing timestamps, outages and changed status definitions as limitations, not as zero waits or successful cases.
If rollout happens in stages, a store or period without the system may provide a useful comparison. Choose it for similar trading conditions and examine whether its earlier trend resembles the deployment stores. A comparison store can still differ in ways that affect service. Where a causal estimate matters, obtain suitable evaluation expertise before presenting one.
Read the change alongside its explanation
Show the before and after counts, rates and time distributions rather than a percentage change alone. Break out relevant groups, such as busy and quiet shifts, only when the samples support it. Inspect failed and unusually slow cases to learn whether the cause was the technology, source data, stock, staffing or a customer handoff.
Ask staff and willing customers what became easier or harder, using the same prompt across periods where practical. Their accounts can explain a pattern; one comment is not a measured outcome for all shoppers. Include customers who used an assisted route or did not use the new interface.
Finish with a bounded statement: what changed, where and when it was observed, how many cases were measured, and what else could explain the difference. State a service finding without claiming more certainty than the design supports.
