Back to blog
InfrastructureApril 2026

How to Size GPU Hardware for Enterprise LLM Inference

Deplero Team

Deplero

Every enterprise AI deployment starts with the same question: how many GPUs do we actually need? Overprovisioning burns capital. Underprovisioning means latency spikes, queued requests, and frustrated users. The good news is that GPU sizing for LLM inference follows a deterministic formula. If you know your model, your quantization strategy, and your concurrency target, you can calculate the hardware requirement on paper before you cut a purchase order.

The VRAM equation: model size times quantization equals GPU memory

The core constraint in LLM inference is GPU VRAM. A model's weights must fit entirely in GPU memory during inference, along with the key-value (KV) cache that stores attention state for active requests. The formula is straightforward:

VRAM_weights = parameters x bytes_per_parameter

VRAM_kv_cache = ~15-25% of VRAM_weights (depends on batch size and sequence length)

VRAM_total = VRAM_weights + VRAM_kv_cache + overhead

Let's run concrete numbers. Llama 3.1 70B has 70 billion parameters. In FP16 (2 bytes per parameter), that's 140 GB just for weights. Add 20% for KV cache and runtime overhead, and you need approximately 168 GB of VRAM. That's two NVIDIA H100 80GB GPUs connected via NVLink, or three A100 80GB GPUs. Qwen 3 14B at FP16 requires 28 GB for weights, roughly 34 GB total -- fits on a single A100 80GB or L40S 48GB with headroom. Gemma 4 27B at FP16 needs 54 GB for weights, about 65 GB total -- a single A100 80GB handles it, but an L40S 48GB is too tight.

VRAM SIZING PIPELINE

PARAMETERS70BLlama 3.1 70B70 x 10^9 paramsxBYTES / PARAM2 BFP16 = 2 bytesFP8=1B | INT4=0.5B=VRAM NEEDED140 GB+ 20% KV cache~168 GB totalGPU MATCH2x H10080 GB each160 GB total

Quantization explained: FP16 vs FP8 vs INT4

Quantization reduces the number of bytes per parameter, which directly reduces VRAM requirements and increases inference throughput. The tradeoff is quality -- lower precision means slightly less accurate outputs. In practice, the quality impact is smaller than most teams expect.

FormatBytes/Param70B VRAMQualitySpeed
FP16 / BF162.0140 GBBaseline1x
FP81.070 GB<1% degradation~1.5-1.8x
INT4 (AWQ/GPTQ)0.535 GB1-3% degradation~2-2.5x

FP8 is the sweet spot for most enterprise deployments. On H100 GPUs, FP8 is a native operation -- no approximation tricks needed. vLLM supports FP8 quantization natively and benchmarks consistently show less than 1% quality degradation on standard evaluation suites. Llama 3.1 70B in FP8 drops to 70 GB of weight memory, fitting on a single H100 80GB with room for KV cache. INT4 (via AWQ or GPTQ) cuts VRAM in half again, but quality degradation becomes noticeable on complex reasoning tasks. Use INT4 for draft models or high-throughput summarization pipelines, not for your primary inference endpoint.

How many concurrent users can one GPU serve?

Raw VRAM capacity tells you whether the model fits. Throughput tells you how many users it can serve. The key metric is tokens per second per user at a given concurrency level. With vLLM's continuous batching engine, incoming requests are dynamically batched in real time rather than processed sequentially -- this is what makes GPU-based inference economically viable for production workloads.

For Llama 3.1 70B in FP8 on a single H100 80GB, vLLM benchmarks show approximately 2,000-3,000 output tokens per second at batch sizes of 16-32 concurrent requests. If your average response is 512 tokens, that's roughly 4-6 complete responses per second, or about 25-35 concurrent users with sub-2-second latency per token. Scale to 2x H100 with tensor parallelism and you roughly double throughput while cutting latency for individual requests. For smaller models like Qwen 3 14B, a single H100 can serve 60-80 concurrent users comfortably.

Tensor parallelism (splitting a model across multiple GPUs) is handled automatically by vLLM. Two H100s connected via NVLink behave as a single 160 GB memory pool from the model's perspective. Continuous batching means vLLM doesn't wait for an entire batch to finish before starting new requests -- it interleaves token generation across all active sequences, maximizing GPU utilization at 85-95% during sustained load.

CONCURRENT USERS VS GPU COUNT (LLAMA 3.1 70B, FP8, VLLM)

Concurrent users (avg 512-token output)~151x A100~352x A100~301x H100~702x H100~1504x H100~3208x H100

Sizing table: popular models across GPU options

The following table maps popular open-weight models to GPU requirements at different quantization levels. All figures assume vLLM with continuous batching, a 4096-token context window, and 20% VRAM reserved for KV cache and overhead.

ModelQuantVRAMA10G 24GBL40S 48GBA100 80GBH100 80GB
Qwen 3 14BFP16~34 GB2x1x1x1x
INT4~10 GB1x1x1x1x
Gemma 4 27BFP16~65 GB3x2x1x1x
FP8~33 GB2x1x1x1x
Llama 3.1 70BFP16~168 GBN/A4x3x2x (TP)
FP8~84 GBN/A2x2x1x
INT4~42 GB2x1x1x1x

TP = tensor parallelism across GPUs via NVLink. "N/A" means the model cannot practically fit on that GPU class even with multiple cards due to interconnect bandwidth constraints. Bold entries indicate the recommended configuration for production workloads.

The Dell, HPE, and Supermicro decision: which server platform?

Once you know the GPU count, you need a server chassis. Three vendors dominate enterprise GPU server sales, and each has a different sweet spot.

Dell PowerEdge XE9680 is the default choice for enterprises with existing Dell infrastructure. It supports 8x H100 SXM5 GPUs with NVLink 4.0, ships with iDRAC and OpenManage integration, and plugs directly into Dell's support and warranty ecosystem. Delivery timelines are typically 6-10 weeks. If your data center already runs Dell and your ops team knows iDRAC, this is the path of least resistance.

HPE ProLiant DL380a Gen11 is strong for organizations already running HPE iLO and OneView. The DL380a supports up to 4x L40S or 4x H100 NVL GPUs in a 2U form factor, making it suitable for environments where rack density matters. HPE's GreenLake consumption model also lets you finance GPU servers as OpEx rather than CapEx -- a meaningful consideration when a single 8-GPU node costs $300K+.

Supermicro GPU servers (such as the SYS-421GE-TNRT) offer the best price-per-GPU when purchased through distribution. Supermicro systems tend to ship faster than Dell or HPE (4-6 weeks), support the same NVIDIA GPUs, and are fully compatible with standard enterprise Linux distributions. The tradeoff is a thinner support ecosystem -- Supermicro's enterprise support is adequate but not at the same depth as Dell ProSupport or HPE Pointnext. For teams with strong in-house systems engineering, Supermicro is often the right answer.

Don't overbuy: start with a pilot, scale from data

The most expensive mistake in GPU procurement is buying for a projected workload that doesn't materialize for 18 months. GPU hardware depreciates -- a $40,000 H100 today will be worth significantly less when Blackwell B200 servers are widely available. The correct approach is to start with a minimal viable GPU configuration, deploy your target model with vLLM, and measure actual utilization under real user load.

A practical pilot configuration for most enterprises: one server with 2x H100 80GB GPUs, running Llama 3.1 70B in FP8 with vLLM and tensor parallelism. This configuration serves 30-50 concurrent users at interactive latency, costs roughly $80-100K for the server hardware, and gives you real production data on throughput, latency percentiles, and utilization patterns within weeks. From there, scaling is linear -- add GPU nodes as demand data justifies the spend.

Monitor three metrics during the pilot: P95 time-to-first-token (should be under 500ms), GPU utilization during peak hours (target 70-85%), and request queue depth (should rarely exceed 2x your batch size). If P95 TTFT creeps above 1 second consistently, you need more GPU capacity. If utilization stays below 50%, you overprovisioned.

DEPLERO

Skip the guesswork -- let us size it for you

Our assessment covers exactly this: sizing and architecture for your models, your concurrency, and your budget, before the purchase order. Then we build it on your hardware or in your cloud accounts and hand it to your team with runbooks: tensor parallelism, quantization, and continuous batching sorted, so your team can focus on the application.

See what we deploy

INFRASTRUCTURE FOR PHYSICAL AI

Building this is the hard part. That's our part.

Deplero is the platform for running your AI at every site: vision and custom models at the edge, on-prem GPU clusters, and your cloud accounts, in one system. Installed in your environment, run by your team.

See the platform