How to Test an AI Customer Service Agent Before It Talks to Real Customers

To test an AI customer service agent properly, use your own customers' real questions, not invented ones. Collect 50 to 100 past messages grouped by type, write down what a correct answer looks like and where that answer comes from, decide in advance what counts as a pass, and include the awkward cases: angry customers, requests outside your policy, and questions the AI should hand to a person. Then test on a real phone, run the agent in draft-only mode for one or two weeks, and go live gradually with limits and a way to switch it off.

Most AI failures in customer service are not exotic. The agent quotes an old price, invents a delivery time, or keeps answering when it should have passed the chat to a person. A structured test catches these before a customer does.

Step 1: collect real questions

Export recent conversations from WhatsApp or your inbox and pick 50 to 100 customer messages. Group them into categories, for example opening hours, prices, availability, delivery, complaints, refunds and booking. Keep the customers' original wording, including typos and mixed languages, because that is what the agent will face. Remove names, phone numbers and other personal data before using the messages as test material.

Step 2: write the expected answer and its source

For each test message, write two things: what a good answer says, and where that information lives. If the answer depends on your price list, the price list must be in the knowledge base the agent reads. If you cannot point to a source, the agent cannot answer reliably either, and the right behaviour is to hand over.

Mark the messages that should always go to a person, whatever the AI knows. Refund disputes, complaints about staff, legal questions and anything involving payment details are typical examples.

Step 3: decide what a pass looks like

Agree on the criteria before running a single test, so results are not judged by mood. A practical scorecard:

• Correct: the facts match your source.
• Grounded: nothing is invented, especially prices, dates, times and stock.
• Handles handover: when the answer is not in the source, the agent says it will check with the team instead of guessing.
• Tone: sounds like your business, at a sensible length.
• One clear next step: the customer knows what happens next.

Set a bar, for example every answer grounded and at least nine in ten fully correct, and do not launch until it is met.

How to test an AI customer service agent on edge cases

The everyday questions usually pass. The edge cases are where agents embarrass businesses. Include at least these:

1. An angry customer who uses strong language.
2. A request for a discount or exception your policy does not allow.
3. A question about something you do not sell.
4. A message in a different language from the one you expect.
5. A customer who asks for a human.
6. A message that is only a photo or a document.
7. A follow-up that refers to something said earlier in the thread.
8. A question whose answer changed recently, such as a new price.

Pay particular attention to number 5. A customer who asks for a person should reach one, and you should check exactly what happens in your setup when they ask.

Step 4: test as a real customer, on a real phone

Testing in an admin screen does not show what the customer sees. Message your business from a personal phone and go through the main scenarios end to end. Check how long replies take, whether long answers are split sensibly, whether greetings repeat, and whether the conversation reads like one person talking. Do it at different times, including outside working hours.

Step 5: run draft-only for one or two weeks

Before the agent sends anything on its own, let it draft replies that a person reviews and sends. Track how often drafts are sent unchanged, lightly edited or rewritten. When most drafts for a type of question go out unchanged for a couple of weeks, and the edits are about style rather than facts, that type of question is ready for automatic replies. Types that still need rewriting stay in draft mode.

Step 6: go live gradually, with a way back

Start with the simplest, most common categories and leave the rest in draft mode. Set limits on how many automatic replies one customer can receive per day and in total, so a mistake cannot repeat a hundred times overnight. Make sure you can switch automatic replies off for one customer or for everyone in seconds. Review a sample of sent replies every week.

Re-test after every change

An agent that passed last month can fail today. Re-run your test set whenever prices or policies change, when you add a product, when you change the agent's instructions, or when the underlying AI model changes. Keeping the test set in a spreadsheet makes this a 30-minute job instead of a project.

Testing in Polynusa

In Polynusa, the AI answers from the knowledge base you maintain, and you can try a customer question in the reply settings and see the answer without anything being sent. To see the real experience, you can test as a customer: the CRM routes its replies to your own number for a set period, so you message your business like a customer would. The Copilot level gives you the draft-only phase, where Polybot drafts and a person sends, and you move to Autopilot only when you are ready. When the knowledge base has no answer, the agent tells the customer it will check with the team, stops replying to that lead, and notifies your admins, and the unanswered question is collected so you can add it to the knowledge base. Daily reply limits and a per-lead switch are in the settings.

For what to put in the knowledge base before testing, see our Indonesian guide on knowledge bases for AI customer service. You can see the product at polynusa.com or start a free trial.

FAQ

How many test questions do I need before launching an AI agent?

For a small business, 50 to 100 real customer messages grouped by category is a practical start. The variety matters more than the number, so include complaints, policy exceptions and handover cases, not only easy questions.

What is the most common failure of AI customer service agents?

Inventing facts, especially prices, delivery times and availability, and continuing to answer when it should hand over to a person. Both are caught by checking every answer against a written source.

Should I let the AI reply automatically from day one?

No. Run it in draft-only mode first, where a person reviews and sends each reply, and switch a category to automatic only when its drafts have gone out unchanged for a couple of weeks.

How do I test what customers actually see?

Message your business from a real phone and go through the main scenarios, at different times of day. Check reply speed, message length and splitting, repeated greetings, and how a request for a human is handled.

How often should I re-test the AI agent?

Whenever prices, policies or products change, when the agent's instructions change, and when the AI model behind it changes. Re-running a saved test set keeps this quick.