Turning a Pilot into a Team Workflow
Working with AI · Part 6
A support tool can look good in a demo and give outdated advice as soon as a product update moves a menu. Assign product ownership for versioned documentation, support ownership for representative answers, and an operator and backup who can disable suggestions.
Test what changed
Check guidance for the new release, the previous release, and inquiries with an unknown version. Retain the question, version, and retrieved document when a test fails. Fix the document or retrieval rule and rerun the three groups. Measure review, correction, and customer wait times alongside drafting time.
Apply Google’s staged release approach
Google’s SRE workbook describes comparing a new release on limited traffic with the existing version. Compare similar version inquiries in support. When guidance is wrong, restore manual support and identify recipients who need a correction.
Google SRE Workbook on canary releases
Rollout and recovery
Replay questions about the changed menu before rollout.
- Measure the existing version
- Apply update to a subset
- Misses criteria → roll back
- Meets criteria → expand
Illustrative samples. Each rate uses its own sample size.
Investigate the four errors by type and check that the backup owner can disable the update and find affected customers.