Buyers evaluating on-prem AI vendors keep hearing the same answer to the same question. How do you deploy? A Helm chart. Sometimes an Operator. Sometimes a tarball with a README. The implication is that on-prem AI is a packaging problem: that if a cluster already exists, the rest is a few thousand lines of YAML. That implication is wrong by about an order of magnitude. A Helm chart that lands a model server on top of an existing Kubernetes cluster is roughly fifteen percent of the work of running AI on-prem. The other eighty-five percent is the substrate underneath: bare-metal provisioning, GPU drivers, kernel pinning, NCCL tuning, MIG partitioning, secure boot, signed and pinned artifacts, an audit trail with evidentiary value, multi-tenant fairness, and a lifecycle for internal models that survives review. This post walks through what an honest on-prem AI deployment actually has to provide, and who ends up doing it.
What a Helm chart actually delivers
A vendor Helm chart for AI inference is, in almost every case, a Deployment or StatefulSet, a Service, an Ingress, a ConfigMap with model paths, a Secret for a registry pull, and a values.yaml that exposes a handful of knobs. The container image usually wraps vLLM, TGI, or Triton. The chart assumes a working Kubernetes cluster, a functional CNI, a default StorageClass that can satisfy a PVC large enough for model weights, and at least one node where nvidia.com/gpu is a schedulable resource. Given those preconditions, helm install brings up a pod that loads weights and serves an OpenAI-compatible endpoint. That is the demo.
Every one of those preconditions is a project. nvidia.com/gpu does not appear on a node by accident. It is the visible end of a chain that includes a signed kernel, a compatible NVIDIA driver pinned to a specific kernel ABI, the NVIDIA container toolkit, the device plugin, the GPU Operator's node feature discovery, and a kubelet config that exposes the device plugin socket. A default StorageClass capable of holding a 70B-parameter model in safetensors implies either a CSI driver against an enterprise array or a local-path provisioner with a real plan for backups, snapshots, and reclaim policy. The Ingress assumes a load balancer, which on-prem means MetalLB or a hardware F5, which means BGP peering or a static VIP allocation that the network team owns. The chart does not deliver any of this. It assumes it. The vendor that ships only the chart has handed the buyer a roof and called it a building.
The substrate problem: kernel, drivers, and the GPU operator
The hardest layer to get right on-prem is the one between the BIOS and the kubelet. Bare-metal provisioning has to deliver a known-good OS image (typically a minimal immutable base such as Flatcar, Bottlerocket-equivalent, or a hardened RHEL) onto servers whose firmware, BMC version, and PCIe topology vary across vendors and even across batches from the same vendor. Secure boot has to be enabled with a signing chain the operator controls, not the OEM's default keys, otherwise the chain of custody for the kernel is broken at the bootloader. The kernel itself has to be pinned: the NVIDIA driver compiles against a specific kernel ABI, and an unattended kernel upgrade on a Tuesday will silently break every GPU node on Wednesday morning when the driver fails to load.
Above the kernel sits the NVIDIA GPU Operator, which is itself a stack: the driver DaemonSet, the container toolkit, the device plugin, DCGM, the DCGM exporter for Prometheus, MIG manager, and node feature discovery. Each component has a version matrix against the driver, the kernel, the CUDA runtime in user containers, and the firmware on the GPU. MIG partitioning, which splits an H100 or H200 into isolated slices for multi-tenant inference, is a node-level configuration that has to be applied before the device plugin advertises the slice profile, which means it has to be reconciled by the operator, which means the operator has to survive a node reboot in the right order. NCCL, the collective communication library that makes multi-GPU training and tensor-parallel inference work, depends on the right NIC drivers, the right PCIe ACS settings, GPUDirect RDMA being enabled in the driver, and an MTU and flow-control configuration on the switch. None of this is in the Helm chart. All of it has to work before the chart's first pod schedules.
Supply chain: every binary, signed and version-pinned
Regulated buyers cannot accept an opaque image pulled from an arbitrary registry. The minimum bar is SLSA Level 3 provenance for every artifact in the deployment: the OS image, the kernel modules, the driver bundle, the container runtime, every container image, and, critically, the model weights themselves. Each artifact needs a cosign signature backed by a key the operator controls, with the public side pinned in a policy. The policy is enforced at admission, typically through Sigstore policy-controller or a Kyverno or OPA Gatekeeper rule, so a pod whose image lacks a valid signature against the pinned key never schedules. This is not a checkbox. It is the only way to defend against the registry-substitution and dependency-confusion attacks that have moved from theoretical to common over the last two years.
Model weights deserve special handling. A 70B-parameter safetensors bundle is a binary blob that produces clinical recommendations, generates legal text, or prices loans. It needs the same supply-chain discipline as a kernel module. That means a content-addressable identifier, a cosign attestation that records the training run or the upstream source, an SBOM that lists the tokenizer and config revisions it was bundled with, and a promotion gate that prevents an engineer from pushing an untracked checkpoint into production. BYO-model workflows, where a regulated customer fine-tunes a model on internal data and promotes it through staging to production, require the same gates the rest of the supply chain uses. A platform that signs its own binaries but lets a notebook drop arbitrary weights into the inference path has not actually solved the supply-chain problem. It has only solved half of it.
The audit story regulators actually accept
Auditors and internal QA functions across regulated industries are no longer satisfied with screenshots of a Grafana dashboard. The artifact a regulator wants is a tamper-evident record that, given any inference output, can reconstruct which model bundle produced it, on which node, against which prompt, with which retrieved context, under which policy, and at which time, and prove that the record itself has not been altered after the fact. That requires hash-chained audit logs, where each event commits to the hash of the previous event, and the chain is anchored periodically to an external trust root such as a transparency log or an HSM-signed checkpoint. Standard syslog does not provide this. Neither does an unmodified Loki or Elasticsearch deployment.
OpenTelemetry is the right transport for the underlying signal: traces for the request path, metrics for utilization and latency, structured logs for events. But the audit layer is what sits on top, capturing the pieces of those signals that have evidentiary value and binding them into a chain. The audit record has to include the model bundle digest, the prompt hash, the retrieval source identifiers, the user and project, the policy version that was in effect, and the node and GPU identifier. It has to be exportable in a format the auditor can verify independently, typically a signed JSONL stream with a Merkle root that the platform publishes daily. None of this is provided by a Helm chart that installs an inference server. All of it is required for a deployment in a regulated environment to survive its first audit.
Multi-tenancy, fairness, and chargeback
On-prem AI is rarely single-tenant for long. As soon as the platform works, the second team wants on, then the third, and within a quarter the GPU fleet is oversubscribed. Kubernetes namespaces and ResourceQuotas are necessary but not sufficient. A namespace can cap how many GPUs a project requests, but it cannot prevent one project's batch fine-tuning job from saturating NVLink and starving another project's interactive inference. Fair-share scheduling at the GPU layer requires either MIG slicing with hard isolation, time-slicing with preemption policies that respect tenant priority, or a scheduler-level plugin such as Kueue, Volcano, or Run:ai-equivalent gang scheduling. The choice depends on the workload mix, but the choice has to be made; the default Kubernetes scheduler will not make it correctly.
Per-project chargeback closes the loop. The platform has to attribute GPU-seconds, memory-seconds, network egress, and storage to the project that consumed them, priced against a depreciation schedule for the hardware and an allocation of power, cooling, and operator time. DCGM exporter metrics, joined to Kubernetes labels and a project taxonomy, produce the raw signal; a chargeback report that a CFO will sign off on is a non-trivial pipeline on top of that signal. Without chargeback, the platform's economics are invisible and the second team's oversubscription becomes the platform team's problem instead of the second team's. With chargeback, the conversation about whether to buy more H200s becomes a conversation between finance and the line of business, which is where it belongs. The Helm chart does not produce a chargeback report. The platform underneath has to.
Who owns the substrate
The buyer's question, translated out of marketing language, is whether anyone owns the substrate or only the application. Owning it means committing to the full stack from firmware to inference: bare-metal provisioning with a signed image and a controlled kernel, the GPU Operator with pinned driver versions and tested MIG and NCCL configurations, an admission policy that requires provenance and signatures on every artifact including model weights, an event log with evidentiary value, fair-share scheduling and per-project attribution that finance will accept, and a promotion path for internal models with the same gates as everything else. The deployment is not done when helm install returns zero. It is done when an auditor can be given read-only credentials and walk away satisfied.
So ask the direct question, of a vendor or of your own plan: who owns the kernel, the driver, the bootloader signing key, the utilization report, and the upgrade path? If the honest answer is "the customer", or "we'll figure it out", someone's team is about to absorb a year of unbudgeted work, and it will be the year they were supposed to be shipping the application that justified the hardware.
There are three real answers. Hire the platform team, which is correct at scale and takes six to twelve months to assemble. Buy an appliance and accept its opinions about how your infrastructure works. Or bring in people who have built this before to stand it up and hand it to your team to run, which is the option most mid-size teams do not realize is on the menu. Any of the three can work. The failure mode is the fourth option nobody chooses out loud: assuming the eighty-five percent will be absorbed by whoever happens to be nearest the cluster.
The eighty-five percent is what the Deplero platform is built to own: bare metal to production Kubernetes with pinned drivers, signed releases verified before they install, a record of what ran where that an auditor can check without us in the room, and a path for your own models through the same gates as everything else. Installed in your environment and handed to your team to run. See the platform.
INFRASTRUCTURE FOR PHYSICAL AI
Building this is the hard part. That's our part.
Deplero is the platform for running your AI at every site: vision and custom models at the edge, on-prem GPU clusters, and your cloud accounts, in one system. Installed in your environment, run by your team.
See the platform