The Weekly AI Conversation Review: Thirty Minutes That Keep an Assistant Honest
An AI assistant drifts out of date quietly. A weekly review of thirty conversations, with a five-column sheet and a fixed agenda, finds wrong answers before customers complain.A clinic's AI assistant answers appointment questions well for three months. Then the clinic changes its Saturday hours. Nobody updates the assistant. For five weeks it tells patients the clinic is open until 17:00 on Saturdays, and several arrive to a locked door. The assistant never malfunctioned. The information behind it went stale, and nobody was looking. A weekly AI conversation review is the habit of looking.
Why dashboards are not enough
A dashboard reports counts: messages answered, average response time, share handled without a person. All can be excellent while answers are wrong. The assistant that sent patients to a closed clinic had a perfect response time.
Only reading actual conversations shows whether customers got correct, useful answers. Thirty minutes a week is the price of knowing.
Choosing the thirty
Twenty at random. From everything the assistant handled in the week. Random matters. If you only read the ones that look interesting, you miss the quiet failures.
Ten on purpose:
- Conversations handed over to a person.
- Conversations where the customer stopped replying right after the assistant's answer.
- Conversations containing words such as "wrong", "not what I asked", "complain", "human", "person".
- Conversations about anything that changed recently: a new price, a new product, new hours.
- The longest conversations of the week.
The random twenty measure normal quality. The chosen ten find problems.
The sheet
Five columns are enough.
| Conversation | Correct? | Helpful? | Issue type | Action |
|---|---|---|---|---|
| Link or ID | Yes, partly, no | Yes, partly, no | From the list below | What to change, by whom |
Correct means the facts were right. Helpful means the customer's actual need was met. An answer can be correct and unhelpful: accurate opening hours given to someone who asked whether a doctor is available today.
Six things to look for
1. Wrong facts. Prices, hours, stock, policy. The most serious and the easiest to fix once found.
2. Invented details. Anything specific that is not in your reference material: a delivery time, a discount, a feature. This shows the assistant filling a gap, and the fix is to tell it what to do when it does not know.
3. Missed intent. The customer asked one thing and got an answer to another. Often caused by short or ambiguous messages. Look for patterns in how customers phrase things.
4. Late or missing handover. The customer was upset, asked for a person, or asked the same thing twice, and the assistant carried on. Check your handover rules.
5. Tone. Too formal, too casual, too long, or cheerful at the wrong moment, such as an exclamation mark in reply to a complaint.
6. Dead ends. The assistant answered and the conversation stopped without the customer getting to an order, a booking or a person. Ask what the next step should have been.
Tag each problem with one of these six. After a few weeks the tags show where the work is.
The thirty-minute agenda
- Minutes 0 to 20: read and mark. Two people, fifteen conversations each, or together on one screen. One line per conversation on the sheet.
- Minutes 20 to 25: count. How many correct, how many helpful, how many of each issue type.
- Minutes 25 to 30: decide. Choose the one or two fixes that address the most frequent or most serious issue. Assign each to a person with a date.
Do not try to fix everything. One real fix each week compounds. Ten good intentions do not.
Turning findings into fixes
Each issue type has a usual remedy.
| Finding | Usual cause | Fix |
|---|---|---|
| Wrong fact | Outdated reference | Update the source, date it, name an owner |
| Invented detail | Gap in reference | Add the information, or instruct to say "I will check" |
| Conflicting answers | Two documents disagree | Remove or label the old one |
| Missed intent | Ambiguous phrasing | Add example questions and the right answer |
| Late handover | Rule too narrow | Widen the triggers: repeated question, negative words |
| Wrong tone | Thin style guidance | Add three example replies in your voice |
| Dead end | No next step given | Instruct to end with a specific next step |
After a fix, test it. Send the same question again and confirm the answer changed. Keep the question in a list of test questions to re-run after future changes.
Track three numbers over time
From the random twenty each week:
- Correct rate. Share marked correct.
- Helpful rate. Share marked helpful.
- Handover quality. Of the handed-over chats reviewed, the share handed over at the right moment with enough context.
Plot them weekly. A drop after a price change or a new product tells you the reference material was not updated. A steady rise tells you the review is working.
Twenty is a small sample, so do not read much into one week's movement. The trend over a month is what counts.
A change log
Keep a simple list: date, what was changed in the assistant's information or instructions, why. When quality shifts, the log shows what changed just before. Without it, every drop is a mystery.
When to review outside the schedule
Do an extra, shorter review:
- The day after any change to prices, hours, policy or products.
- After any change to the assistant's instructions.
- When the AI provider changes the underlying model, if you are told.
- After a complaint that mentions the assistant.
- During a promotion or a busy season.
Common mistakes
- Reviewing only when something goes wrong.
- Reading only the bad ones. You lose the measure of normal quality.
- No sheet. Impressions fade by next week.
- Finding problems and fixing none.
- Fixing ten things at once, so nobody knows which change helped or hurt.
- Letting it lapse after a good month. The clinic's assistant was good for three months.
- Blaming the tool. Most findings trace back to information the business did not provide or did not update.
What it costs
Half an hour for two people each week is about four hours a month. Compare that with one customer who received a wrong price, or five patients at a locked door. The review is cheap insurance, and it is also the fastest way a team learns what the assistant can be trusted with.
In Polynusa
Conversations handled by Polynusa's AI agents sit in the same inbox as the team's, so they can be opened, read and searched for a review like this, and the business information the agents read from can be updated in the workspace. See the pricing page.
Summary
Each week, read twenty random conversations and ten chosen ones. Mark each as correct and helpful or not, tag the issue type, and count. Choose one or two fixes, assign them, and test them. Track correct rate, helpful rate and handover quality over time, keep a change log, and review again after anything changes. Half an hour a week keeps a good assistant from quietly becoming a wrong one.
FAQ
How many AI conversations should we review each week?
Thirty is a practical number for a small business: twenty chosen at random and ten chosen on purpose, such as chats that were handed to a person, ended without a reply from the customer, or contained a complaint. It is enough to see patterns and small enough to finish in half an hour.
Who should do the weekly review?
The person who owns the assistant, together with someone who talks to customers every day. The owner knows what can be changed in the instructions, and the frontline person knows what a good answer looks like. Rotating the second person spreads understanding through the team.
What do we do when the review finds a wrong answer?
First check whether the customer was harmed and contact them if so. Then find the cause: missing information, outdated information, conflicting documents or an unclear instruction. Fix that source, test the same question again, and note the change with the date.