Shadow-Testing a Vision Model on Live Site Data Before It Goes Live
Deplero Team
Deplero
Your new vision model beat the old one on every held-out test set. It goes live at a customer site, and within a day the support queue fills up: the model is confused by a camera mounted at a new angle in one room. The lab evaluation was right about the lab. It just wasn't the site. Shadow testing closes that gap by checking a new model on each site's own live data before it serves a single result there.
Why lab evaluation isn't enough
Models that run in the physical world see data that shifts from site to site: lighting, camera position and resolution, the procedures or products in front of the camera, even the time of day. A held-out test set captures your training distribution, not the long tail of every customer's room. The more sites you have, the more likely at least one of them sits outside what you tested.
What a shadow test is
In a shadow test, the new model runs on the device next to the current one and sees the same live frames. Only the current model's answers are used. The new model's answers are recorded and compared. After enough frames, you know how the new version behaves on that site's real data, and nobody downstream ever saw its output.
What to compare
- Agreement with the current model, overall and per class. Overall agreement can hide a class the new model now gets badly wrong.
- Disagreements worth a human look. Save a sample of the frames where the two models disagree most; they are the fastest way to see whether the new model is better or broken.
- Confidence distribution. A sharp drop in confidence on one site is an early sign the data there looks unfamiliar.
- Latency and GPU memory, measured on the real device while both models run, so you know the new one fits once it runs alone.
Setting the pass bar
The pass bar belongs to whoever owns model quality, usually the ML lead, not to the deployment system. A typical bar reads like "at least 95% agreement with the current model on at least 1,000 frames, and no class below 90%". Agreement isn't accuracy: when the new model is supposed to be better, some disagreement is the point. That's why the saved disagreements matter, and why the bar is set per product and sometimes per class. What matters most is that the bar is written down before the test, applied the same way at every site, and checked automatically.
Where it runs, and what leaves the site
Run the shadow test on the edge device itself, so frames never need to leave the site. Only the comparison metrics and, if the site's data rules allow it, a small sample of de-identified disagreement frames go back to your team. Two models on one device need GPU headroom, so check capacity first; a device without room can shadow-test on a sampled subset of frames or be scheduled at a quieter time.
When it fails
A failed shadow test is good news delivered early. The new version simply doesn't go live at that site. The current model keeps serving, the rollout continues at sites that passed, and the disagreement frames become training data for the next version. Compare that with finding out from a customer.
What it isn't
For medical and other regulated products, a shadow test is an engineering safety check on each release at each site. It doesn't replace the clinical or product validation your quality and regulatory team owns. It makes sure the model you validated is the model that behaves as expected in every room it runs in.
INFRASTRUCTURE FOR PHYSICAL AI
Building this is the hard part. That's our part.
Deplero is the platform for running your AI at every site: vision and custom models at the edge, on-prem GPU clusters, and your cloud accounts, in one system. Installed in your environment, run by your team.
See the platform