What Should Teams Learn from Using AI?
Working with AI · Part 10
AI drafts a refusal for a refund. An experienced agent checks the original sales channel and terms, which can change the answer.
Record why the decision changed
Keep the relevant facts in anonymized training cases. Have staff identify what they would check before showing the AI draft. Discuss assumptions behind disagreements and document the agreed approach. Review recurring corrections weekly and check whether people applied the new guidance in subsequent cases.
How a support study examined learning
Brynjolfsson, Li, and Raymond reported a 15% rise in resolutions per hour on average, with greater gains for less experienced agents. They used temporary outages to explore learning. A support team can use unaided training cases for a similar check without interrupting live service.
Brynjolfsson, Li, and Raymond in the Quarterly Journal of Economics
Turning corrections into team learning
Save the channel and policy in effect at purchase so colleagues can apply the reasoning to new cases.
- Save draft and evidence
- Record why it changed
- Review cases and revise guidance
- Test without tool assistance
Illustrative assessments with different cases of comparable difficulty and no AI assistance.
Before attributing a change to training, check case difficulty, scoring, and the participant mix. Review evidence people routinely missed.