Back to blog
DeploymentNovember 2026

How to Roll Out a New AI Model to Every Customer Site Without Breaking One

Deplero Team

Deplero

The first customer site works because your best engineers sat with it. The model ran on the edge box next to the cameras, someone tuned the network rules, and the demo became production. Then site two arrives with different hardware, a stricter firewall and a slightly older version of your model. By site ten, nobody can say with certainty which model version runs where, and every new release is a small act of courage. This post is a practical guide to shipping a new model version to every customer site without breaking one.

1. Know exactly what runs where

You can't roll out safely to a fleet you can't see. Before anything else, keep a version map: for every site, every device, and every model on that device, the version that is actually running, read from the device rather than from a spreadsheet someone updates by hand. Robotics and medical-imaging products often run two or three models per device (a detector, a classifier, a policy), so track each one separately.

The version map is also where drift shows up. One operating room still on last quarter's model, one robot that missed an update while it was offline: these are the devices that produce the strange support tickets. Finding them before a customer does is half the value.

2. Sign every release, verify it on the device

A model release is more than a weights file. It is the weights, the runtime engine (for example a TensorRT build for a specific GPU), the pre- and post-processing code, and the configuration that ties them together. Package them as one bundle, sign it in your registry, and have each device verify the signature before it installs anything. That single check removes a whole category of incidents: half-copied files, the wrong build for the wrong GPU, and anything that didn't come from your pipeline.

3. Check capacity before you ship

A new model version often needs more GPU memory than the one it replaces. The edge boxes at your largest customers have room to spare; the smallest box at a small clinic may not. Check, per device, that the new release fits next to everything else that runs there, and hold the devices that don't fit on the current version instead of letting the install fail at three in the morning. A held device is a planning conversation. A failed install is an outage.

One signed bundle fanning out to 6 identical sites, with one site heldA signed package on the left, labelled one bundle, with lines fanning out to 6 identical site outlines on the right, labelled every site. Each site shows the release. On the second release one site, Site 04, is marked held in amber and keeps its current version; the others show a check and continue.one bundlev2.4.1every siteSite 01Site 02Site 03Site 04heldSite 05Site 06one held, the rest continue
Fig. 1. One signed bundle goes to every site the same way, and each site verifies it before installing. When the next release reaches a site that cannot take it, that site is held on its current version and the rest continue. Sample data.

4. Roll out in waves, behind a health gate

Never push to every site at once. Start with one device, ideally in your own lab or at your most forgiving customer, then the rest of that site plus one device elsewhere, then wider waves. Between waves, a health gate decides whether to continue: latency within target, no errors, and the model's answers looking the way they should (see the next section). Waves turn a bad release into a contained problem on a handful of devices instead of a fleet-wide one.

5. Test the model on each site's own data first

A model that passed every test in the lab can still behave differently at one site: different lighting, a new camera angle, a different procedure mix. Before a new version goes live at a site, run it in shadow beside the current one on that site's live data and compare their answers against a pass bar your ML team sets. If it falls short, it never goes live there. We cover how to set this up in a separate post on shadow testing.

6. Make rollback one step, and automatic

Rollback should be boring. Keep the previous release on the device so going back is a switch, not a download. Roll back the whole bundle (model, engine and configuration together), because rolling back only the weights against new pre-processing code is its own outage. And let a failed health gate trigger it without waiting for someone to wake up.

7. Plan for sites that go offline

Hospitals, plants and labs lose connectivity, and some never had it. Devices should keep serving the current model when the network drops, queue the update, and install it on reconnect after the same checks. For air-gapped sites, deliver signed bundles through the customer's own transfer process into a registry mirror inside the site, with signature checks at every hop.

A short checklist

  • Can you list, right now, which model version runs on every device at every site?
  • Is every release one signed bundle, verified on the device before install?
  • Do you know which devices have GPU room for the next release?
  • Does the first wave reach one device, not one customer?
  • Is a new model tested on each site's live data before it goes live there?
  • Is rollback automatic, and does it restore the model, engine and configuration together?
  • Do offline and air-gapped sites get updates through the same checks?

If most answers are "not yet", that's normal for a team with a handful of sites and a product that grew faster than its release process. It's also the point where every additional site starts costing more than the last, and the right time to fix it.

These seven practices are the release path the Deplero platform runs for every site: one signed bundle, verified on the device, checked for GPU room, shadow-tested on that site's live data, rolled out in waves and rolled back on its own when a site fails its check. See the platform.

INFRASTRUCTURE FOR PHYSICAL AI

Building this is the hard part. That's our part.

Deplero is the platform for running your AI at every site: vision and custom models at the edge, on-prem GPU clusters, and your cloud accounts, in one system. Installed in your environment, run by your team.

See the platform