A candidate performs well offline and shows no obvious errors on live requests. Is that enough to replace the production system? The answer depends on what the shadow test copied, what it isolated and which requests disappeared from the analysis. This is an evergreen release-method discussion, not a claim of recent industry news. All numerical examples are constructed and are not IDENIFE project measurements.
Define the decision the shadow test will support
Business regression testing and staged releases for enterprise LLM upgrades require complementary evidence. Offline replay helps fix materials and repeat diagnostics. Shadow testing exposes a candidate to authorized live inputs while production remains responsible for the actual service. A limited rollout lets some users experience candidate outputs. These methods answer different questions; passing a shadow test does not establish user benefit.
AWS describes copying inference requests to a candidate variant within an endpoint while returning only production responses to the caller.[1] This product capability does not imply that every gateway automatically retains comparable candidate outputs. Istio's mirroring example instead uses fire-and-forget requests and discards mirror responses.[3] Connecting traffic and building an evaluation dataset are separate tasks.
State the question first: are you checking field quality after a model substitution, or the operation of an entire agent workflow? The former can fix retrieval and tool results; the latter needs the candidate's own dependency environment. Mixing the two makes differences difficult to attribute to the model, changing evidence or tool state.
Identical requests may carry different evidence
Assign a correlation identifier at the entry point for each request authorized for copying. Connect production and candidate outputs and record the candidate configuration, prompt revision, retrieved-material identifiers, input time and timeout budget. The identifier should not directly contain personal data. Evaluation content also needs access boundaries and retention limits. Withholding outputs from users does not justify unrestricted production-data copying.
Consider a constructed inventory question. Production reads eight available units at ten o'clock, while the candidate reads six slightly later. Different answers may reflect changed business state rather than worse reasoning. To test understanding of identical evidence, freeze authorized tool responses for both configurations. To test the end-to-end workflow, record the versions each read and classify state differences separately.
A candidate agent may select a tool that production never called, leaving no recorded result to replay. Filling that gap with an empty string cannot support a claim of realistic validation. Narrow the test, build a controlled simulation for the new branch or run it in an independent environment, and identify simulated components in the report. Fixed evidence improves comparability while reducing coverage of live dependency changes; make that tradeoff explicit.
Withholding responses does not prevent production side effects
A response is discarded after the call; database writes, notifications or inventory reservations may already have occurred inside the service. We therefore propose side-effect isolation as an entry condition. Give the candidate a separate identity, deny production writes at the tool gateway and route writes to isolated data and messaging facilities. A prompt telling the model not to act is no substitute for permission and routing controls.
For branches that require a write to complete, record the proposed action and assess its object, parameters and authorization evidence without committing it. This tests the action plan, not the success rate of remote execution. If simulated tools are used, document their inability to reproduce every real lock conflict, rate limit and failure. Simulated success must not be counted as successful production execution.
Isolation also concerns resources. Read-only candidates can still consume shared database connections, retrieval capacity or model quotas. Set independent concurrency and spending limits. Stop copying or drop requests according to a predefined policy when the mirror queue reaches its limit, and include drops in coverage statistics. Provide a switch that stops the experiment without disrupting production, and define what happens to work already in flight.
Keep unpaired requests in the report
Consider a constructed accounting example. Of 1,000 eligible requests, 200 are sampled. The candidate receives 180; 150 produce pairable results before the observation deadline and 30 time out. The other 20 are lost during copying. If 135 of the 150 pairs agree, agreement among paired results is 90 percent.
That is neither accuracy across the 200 sampled requests nor quality across all production traffic. Pairing coverage is 150 divided by 200, or 75 percent; receipt coverage is 90 percent. Preserve the 50 unpaired requests as categories such as delivery loss and timeout. Do not assume they are correct, or indiscriminately label all of them semantic errors. A timeout that violates a service deadline can separately count as a deadline failure.
Agreement is not correctness either: both configurations may share incorrect evidence or rules. Blind-review disagreements and sample agreements against business evidence. If disagreements are oversampled to reduce review cost, report sampling strata and weights separately. The reviewed set's error proportion cannot directly represent production traffic.
Check whether missing results cluster around long materials, particular languages, peak periods or individual tenants. If difficult requests time out more often, the remaining pairs will look better. Explain every loss between eligibility, selection, delivery, completion and pairing before discussing improvement. Track late results within a predefined window rather than waiting selectively for favorable examples.
A small traffic fraction does not establish full-load capacity
AWS recommends progressively increasing mirrored traffic and testing full traffic before promoting a variant, while considering instance configuration, scaling and operational metrics.[2] The practical implication is that success at a low fraction generally supports a conclusion only at that tested load. It is not a full-capacity commitment. Other platforms need validation based on their own quotas, scaling behavior and dependency constraints.
A candidate may meet quality requirements at low concurrency but develop longer queues as demand rises. Timeout policies then change the proportion of usable answers. Record arrival rate, completed work, queueing and end-to-end latency, failure categories and cost. Cover business-relevant peaks and input combinations. A lower average latency does not offset widespread timeouts on important requests.
Full mirroring costs money and can stress shared dependencies. If it cannot be performed within a safe boundary, combine isolated load tests with a controlled expansion of scope and identify untested scale. Do not sacrifice production stability to obtain a full-mirroring label. Copying every request during one interval also does not establish coverage of all seasonal business patterns.
Use an evidence checklist to choose the next step
A release review can retain six items: the question and approved scope; inputs and dependencies actually copied; side effects blocked; coverage and missingness at each stage; quality by stratum and capacity results; and stopping conditions for the next phase. This is our proposed working checklist, not a vendor certification standard. Attach configurations or records to each item instead of simply marking it passed.
If isolation fails, repair isolation first. If many results cannot be paired, repair observation and determine whether missingness is selective. If quality passes but peak capacity remains untested, continue only within validated load. Proceed to a stoppable, limited rollout when evidence supports that next stage, then observe real adoption, rework and business outcomes.
Because the candidate has not changed what users see, shadow testing cannot directly measure follow-up questions, actions or conversion caused by candidate answers. Those conclusions require a separate business experiment with clear allocation and evaluation rules. The most useful output of a shadow test is a smaller area of uncertainty and earlier discovery of release problems, not a claim that unobserved business gains have already occurred.