If you run an AI product, the safest way to find out whether a second provider is cheaper is not to switch. It is to run both at once. Shadow testing sends the same requests to a second endpoint without returning its answers to users. You keep production on the provider you trust, compare the two side by side on real traffic, and switch only when the evidence is clear.

Here is how to do it without breaking production.

1. Pick the traffic that matters

Do not shadow everything on day one. Choose one high-volume, low-risk path: a summarization step, a classification call, or a support reply draft. You want enough traffic to be statistically useful and little enough that a bad shadow response cannot hurt a user. Keep the shadow path off the critical request path entirely.

2. Send a copy, not the original

In your request handler, fire the shadow call after the production call returns. Use the same prompt and the same input, but discard the shadow response. Never block the user response on the shadow provider. If the shadow call fails or times out, log it and continue. A shadow provider that is down should be a non-event for users.

3. Compare the right things

Three dimensions matter:

  • Cost per successful request. Compute this from the token counts and the published rates, not from a monthly invoice. Cache, input and output are priced separately. A provider that looks cheap on output can lose on cache.
  • Output quality on your own examples. Use a fixed set of 50 to 100 real inputs and score both providers against your acceptance criteria. Do not rely on generic public scores; your prompts and your edge cases are the test.
  • Error and refusal behavior. Count timeouts, malformed outputs, rate-limit responses and refusals. A cheaper provider that refuses more often is not cheaper.

4. Keep the switch reversible

Run the shadow test for at least one full traffic cycle, usually a week. Keep the production path unchanged. When you are ready to switch, route a small percentage of real traffic to the new provider first, watch the same three dimensions, and increase the share only when the numbers hold. If anything regresses, send the traffic back. The switch should be a config change, not a migration.

5. Write down the decision

Record the cost per request, the quality scores and the error counts for both providers. A decision you cannot explain six months later is a decision you will repeat. Keep the comparison table with the prompt version you tested, because prompts change and the comparison ages.

Shadow testing has a cost, because you pay for both providers during the test. Treat that cost as the price of a reversible decision. A week of double spend is usually cheaper than a blind migration that breaks a feature your users depend on.

If you want a second OpenAI-compatible endpoint to shadow against, the base URL is https://webwright.ai/v1, and GET /v1/models lists the model ids without an API key.

When you are ready to design the test, book a 30 minute consult and bring your highest-volume prompt.

The WebWright Team