Skip to content
Voice Agent FleetBook a free review of your phone agents

Changed an AI phone agent? Test it before customers hear it.

This free guide is for owners running several AI phone agents. It shows how to use each platform's own test calls, so a change is checked before any customer hears it.

Book a free review of your phone agentsOr read how a safe rollout works

Page updated .

In short

  • All three platforms can run test conversations with a simulated caller.
  • Save the real calls that went wrong and rerun them before every change.
  • Decide in advance what result stops a change from going live.

Can I test without calling the agent myself?

Yes. Vapi, Retell and ElevenLabs can each run a simulated caller against your agent and report whether the call went the way you wanted.

  • Vapi has Simulations, Evals and Scorecards.
  • Retell has simulation tests and batch tests.
  • ElevenLabs has simulation, next-reply and tool-call tests.

Which calls should I test?

The calls your agent must handle, and every call that once went wrong. Turn each confirmed failure into a saved test, and write down what result is good enough before you see the results.

A good test set has five parts.

  1. Build the set from real calls. The Vapi monitoring docs say to turn confirmed production failures into regression tests, so every call that went wrong becomes a test.
  2. Write the pass rule, for example: every test passes, or a named share passes on repeated runs. ElevenLabs can repeat a test and report a pass rate. Decide the number before you see the results.
  3. Run it before the change reaches callers. Use a draft, a branch or a separate organization where the platform allows it.
  4. Make the change live only when the rule is met.
  5. Keep the result with the version. Store the run next to the version it tested, so when a call goes wrong later you can see what was and was not covered.

When is a separate testing tool worth it?

When the built-in tests stop being enough. Dedicated voice-agent testing tools exist, such as Cekura, Hamming and Coval, and are worth a look in the cases below.

  • You run agents on more than one platform and want one set of tests that is not tied to a single platform's test format.
  • The built-in tests do not cover a call you care about, and you have a list of the gaps.
  • You have outgrown running the tests as a script in your own pipeline.

We name these tools for orientation only. This guide does not compare or rank them.

For your developer: the details on each platform

Technical notes for whoever manages your voice agents on Vapi, Retell and ElevenLabs. Every statement comes from the official docs or forum pages in the Sources list.

On Vapi

  • Vapi Simulations test the latest published version, not the draft. That is the catch: on a live number, a version you publish to test also takes live calls.
  • Simulations run an AI tester through a real voice or chat conversation with an assistant or squad and score the outcome. Scenarios and suites are available through the API too.
  • The path to production page says a published version takes live calls right away, before you rerun your simulations. It also says that to test changes before they reach live calls, use config as code.
  • With GitOps, simulation suites can run on every pull request against the branch's own files. Nothing is deployed, tool calls get mocked results, and each run uses simulation minutes.
  • Evals run on mock conversations at the text and model layer, with exact match, regex or AI judging, and tool-call checks. The docs say they do not test speech recognition, audio quality or turn-taking.
  • Scorecards grade completed calls after the fact. The docs say they are available through the API only for now.

On Retell

  • Retell saves test cases and reruns them as a batch, with pass or fail and a written explanation for each. Simulation tests run as text conversations.
  • A test case holds a prompt for an AI-simulated caller, success criteria, dynamic variables and mocks for custom functions. The docs say saved cases become a regression suite to rerun after every prompt or flow change.
  • Simulation testing works with single-prompt, multi-prompt and Conversation Flow agents. Agents on a custom LLM are not supported.
  • A batch runs many cases together. The docs say the list shows batches against one agent version at a time, and that batches can be created and read through the API to fit CI.
  • A tool call that matches no mock falls through to the real tool, so mock every tool that writes data.
  • The docs we read do not say whether a batch can run against a draft, so create a draft and confirm that on one agent before you build the gate around it.

On ElevenLabs

  • Tests run through the API or the command line against a branch, and the default is Main. That makes a branch the natural place to run the gate.
  • Simulation tests run a multi-turn conversation with a simulated user. Next-reply tests have a model judge the agent's next message. Tool-call tests check the tool and its parameters.
  • A test can be repeated 2 to 20 times, and the results are grouped as a pass rate.
  • The run-tests API takes an optional branch ID, and tests run on the main branch if you give none. The command line tool has a test command that, according to its README, exits non-zero on failure.

On every platform

  • The docs of the three platforms describe running tests in CI. We found no built-in gate that blocks a publish, so the gate is a step in your own pipeline.

Sources

Official documentation and forum pages this guide relies on, as read on 18 November 2026. Platforms change fast, so check the page before you act.

Checked against the Vapi, Retell and ElevenLabs docs on 18 November 2026.

Want someone to look at how your AI phone agents are changed?

The free 30-minute call covers how a change reaches your agents today, what broke last time and what we would set up. We ask for no logins before the call.

Book a free review of your phone agentsOr read how a safe rollout works