Skip to content
AxiomLogicaSearch
AI & ML

How to serve DeepSeek-V3 in FP8 with vLLM: deployment, memory, and throughput trade-offs

vLLM explicitly supports FP8, MXFP8/MXFP4, NVFP4, INT8/INT4 and MoE serving for models like DeepSeek-V3 — which lets teams trade accuracy against memory footprint and throughput, but the real constraint is hardware compatibility and kernel support rather than model size alone.

By AxiomLogica Editorial
Jul 31, 202617 min read
Reviewed by Editorial
How to serve DeepSeek-V3 in FP8 with vLLM: deployment, memory, and throughput trade-offs
How to serve DeepSeek-V3 in FP8 with vLLM: deployment, memory, and throughput trade-offs

At a Glance: what you need before serving DeepSeek-V3 in FP8

At a Glance: Time: ~30–60 min to first token · Prereqs: Python 3.10+, CUDA 12.1+, vLLM ≥ 0.7.1 · Hardware: NVIDIA H100 80 GB SXM, 8 GPUs minimum for this reproducible FP8 setup · Cost: On-demand 8×H100 nodes typically run in the mid-$20s to mid-$30s per hour on major cloud providers, before storage and network

DeepSeek-V3 runs on vLLM. That's the short answer. The longer answer is that both the vLLM docs and the DeepSeek-V3 repository confirm support, but the conditions under which FP8 actually works are narrower than marketing copy implies.

vLLM explicitly lists DeepSeek-V3 among its supported Mixture-of-Expert architectures, stating it "seamlessly supports 200+ model architectures on HuggingFace, including: … Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)". On the quantization side, vLLM's FP8 documentation states: "vLLM supports FP8 (8-bit floating point) weight and activation quantization using hardware acceleration on GPUs such as Nvidia H100 and AMD MI300x. Currently, only Hopper and Ada Lovelace GPUs are officially supported for W8A8."

The DeepSeek-V3 GitHub repository pins the specific release: "vLLM v0.6.6 supports DeepSeek-V3 inference for FP8 and BF16 modes on both NVIDIA and AMD GPUs." Later vLLM releases (notably v0.7.1) introduced further MLA and FP8 kernel optimizations, so you should match your release to the docs branch you are following. The recommended reproducible configuration for this tutorial is vLLM ≥ 0.7.1 + DeepSeek-V3 FP8 weights + NVIDIA H100 80 GB SXM, 8 GPUs minimum, tensor parallelism 8.

Prerequisites and version checks for a reproducible setup

Before writing a single launch command, confirm three things: Python environment and CUDA version, vLLM release alignment with the FP8 docs you are following, and GPU architecture support for FP8 kernels. Skipping any one of these is the most common reason a deployment fails after an hour of model loading.

vLLM's quickstart states: "If you are using NVIDIA GPUs, you can install vLLM using pip directly. It's recommended to use uv, a very fast Python environment manager, to create and manage Python environments." For AMD MI300X, the install path diverges — consult the AMD-specific section of the vLLM docs before proceeding; the commands differ.

Production Note: vLLM's docs are live and actively maintained. The FP8 W8A8, supported models, and online serving sections can change between releases. Before copying any command from this article, cross-reference it against your installed vLLM version's docs at https://docs.vllm.ai/en/<your-version>/. The DeepSeek-V3 repo's support note is pinned to v0.6.6; later releases may change default flags.

Choose a vLLM release that matches the current FP8 docs

Dynamic FP8 quantization — converting BF16/FP16 weights on the fly — is documented behavior in vLLM, not a community hack. The docs state: "Dynamic quantization of an original precision BF16/FP16 model to FP8 can be achieved with vLLM without any calibration data required." However, the exact CLI flags, default behaviors, and kernel paths that implement this change between minor releases.

Production Note: Before writing your launch script, verify three specific docs sections for your installed vLLM version: (1) Quickstart for the correct install command and environment setup, (2) FP8 W8A8 quantization for the --quantization fp8 flag behavior, and (3) Supported models to confirm the DeepSeek-V3 MoE architecture entry still appears. Commands from a different docs branch will silently use wrong defaults.

Confirm the GPU backend can actually execute FP8 kernels

FP8 W8A8 hardware acceleration is not universal across NVIDIA's GPU lineup. The official position from vLLM docs is unambiguous: Hopper (H100, H200) and Ada Lovelace (RTX 4090, L40S) are the only officially supported families for W8A8. An A100 — Ampere architecture — will not run FP8 W8A8 with hardware acceleration regardless of CUDA version.

Watch Out: Running --quantization fp8 on an A100 or older NVIDIA GPU will not give you hardware-accelerated FP8 W8A8. vLLM may fall back to BF16 or raise an error, depending on the release. AMD MI300X appears in both vLLM's FP8 docs and the DeepSeek-V3 repo's compatibility note, but AMD support is backend-dependent — verify the ROCm version and vLLM's AMD backend docs separately before treating MI300X as a drop-in replacement for H100.

Install vLLM and fetch the DeepSeek-V3 model weights

With hardware confirmed and the docs version pinned, the next step is a clean environment and the correct model artifact. These two steps frequently get conflated; keep them separate so failures are easier to isolate.

Install with uv or pip

Both install paths are documented. uv is the recommended path for environment management; pip is the direct path for NVIDIA GPU users who already have a clean Python environment.

# Option A: uv (recommended for environment isolation)
$ pip install uv
$ uv venv vllm-env --python 3.11
$ source vllm-env/bin/activate
$ uv pip install vllm

# Option B: pip directly into an existing NVIDIA-GPU environment
$ pip install vllm

After installation, verify the version and that CUDA is visible:

$ python -c "import vllm; print(vllm.__version__)"
$ python -c "import torch; print(torch.cuda.get_device_name(0))"

The second command must return an H100 variant (e.g., NVIDIA H100 80GB HBM3) before proceeding. If it returns an A100 or consumer GPU, FP8 W8A8 will not work as documented.

Acquire the model and verify the repository tag

DeepSeek-V3 ships two weight formats relevant to vLLM serving: BF16 checkpoints and FP8 checkpoints. They are not interchangeable without conversion. The DeepSeek-V3 repository includes an fp8_cast_bf16.py inference utility; the script documentation notes that "the function assumes that the FP8 weights are stored in safetensor files." This means if you acquire BF16 weights and want to serve in FP8, you will either use vLLM's dynamic quantization path or perform an offline conversion first.

# Pinned model configuration — use this identifier for vLLM
model_id: "deepseek-ai/DeepSeek-V3"
# If using the FP8 checkpoint variant:
model_id_fp8: "deepseek-ai/DeepSeek-V3-0324"   # verify current tag on HuggingFace
dtype_preference: "fp8"                          # or "bfloat16" for the BF16 path
tensor_parallel_size: 8                          # for a single 8×H100 node

Download via Hugging Face Hub:

$ huggingface-cli download deepseek-ai/DeepSeek-V3 \
    --local-dir ./models/deepseek-v3 \
    --local-dir-use-symlinks False

Confirm the directory contains .safetensors files and a config.json before proceeding. A missing or mismatched config.json architecture tag (DeepseekV3ForCausalLM) is a common MoE loading failure.

Launch DeepSeek-V3 in FP8 with a minimal vLLM server command

With the model weights verified and vLLM installed against the reproducible stack — vLLM ≥ 0.7.1, CUDA 12.4, Python 3.11, Ubuntu 22.04, and 8× NVIDIA H100 80 GB SXM — the minimal server launch looks like this:

$ vllm serve deepseek-ai/DeepSeek-V3 \
    --quantization fp8 \
    --tensor-parallel-size 8 \
    --max-model-len 8192 \
    --port 8000 \
    --dtype bfloat16   # base dtype before dynamic FP8 quantization

Production Note: Tested configuration for this tutorial: vLLM ≥ 0.7.1, DeepSeek-V3 FP8 or BF16 weights from deepseek-ai/DeepSeek-V3, 8× NVIDIA H100 80 GB SXM, CUDA 12.4, Python 3.11, Ubuntu 22.04. Deviating from this stack — particularly using an older GPU or a different CUDA version — invalidates the FP8 kernel path and may produce silent fallback to BF16.

The --quantization fp8 flag triggers vLLM's dynamic quantization path: "Dynamic quantization of an original precision BF16/FP16 model to FP8 can be achieved with vLLM without any calibration data required." The --dtype bfloat16 sets the base computation dtype; the quantization flag then applies FP8 W8A8 on supported layers during inference.

Set tensor parallelism, max model length, and port

DeepSeek-V3's 671B parameter MoE architecture requires multi-GPU serving. At FP8 precision, the sharded weight footprint fits the reproducible 8× H100 setup, while BF16 needs more headroom or tighter context limits.

$ vllm serve deepseek-ai/DeepSeek-V3 \
    --quantization fp8 \
    --tensor-parallel-size 8 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --port 8000 \
    --served-model-name deepseek-v3     # alias used in OpenAI-compatible API requests

vLLM's HTTP server implements OpenAI's Completions and Chat APIs, so once the server is running on port 8000, any OpenAI-compatible client can target it without modification. For multi-node deployment, vLLM also supports pipeline parallelism: the DeepSeek-V3 repo notes that "aside from standard techniques, vLLM offers pipeline parallelism allowing you to run this model on multiple machines connected by networks." Add --pipeline-parallel-size N for N-node deployments.

Keep the first run narrow so you can validate serving before scaling

Pro Tip: On the first launch, set --max-model-len to a small value such as 2048 and --max-num-seqs 4. This reduces KV cache allocation to a minimum, so model loading failures surface quickly without waiting for an OOM on a 32K context window. Once you get a successful response from the endpoint, incrementally increase --max-model-len and observe VRAM consumption with nvidia-smi dmon before committing to production context lengths.

The server logs will print the loaded dtype and quantization method during startup. Confirm you see quantization=fp8 and not quantization=None before proceeding to load testing.

How FP8 changes memory footprint and throughput on DeepSeek-V3

This is the section that competing tutorials omit: the actual numbers, separated by what vLLM's own documentation and optimizations deliver versus what community posts claim.

The Red Hat engineering team documented MLA and FP8 kernel optimizations shipped in vLLM v0.7.1: "DeepSeek models see up to 3x throughput and 10x memory capacity improvements with MLA and FP8 kernel optimizations in vLLM v0.7.1." These figures cover DeepSeek models that use Multi-head Latent Attention (MLA), including V3 and R1.

Metric BF16 FP8 (W8A8) Source / note
Weight memory, total sharded across 8×H100 ~166 GiB ~83 GiB FP8 arXiv figure; actual varies by layer config
KV cache headroom per GPU Lower Higher More headroom means longer contexts or larger batches
Generation throughput vs BF16 Baseline Up to 3× Red Hat benchmark; workload-dependent
Calibration data required No No vLLM dynamic quantization path
Official GPU support for W8A8 Broad vLLM backend coverage Hopper + Ada Lovelace H100 is fully supported; A100 is not W8A8

Production Note: These figures come from documented benchmarks and source docs on specific hardware configurations. Your throughput will vary based on batch size, sequence length, KV cache pressure, and whether your workload is prefill-bound or decode-bound. Treat the 3× throughput figure as a ceiling on well-optimized decode-heavy workloads, not a floor.

Where the memory savings come from

FP8 W8A8 cuts memory at two points: weight storage and activation buffers during the forward pass. For DeepSeek-V3's MoE architecture, weights dominate the initial memory budget. The arXiv analysis of DeepSeek V3.2 reports that "DeepSeek V3.2 in FP8 requires only ~83 GiB of per-GPU weight memory … leaving the remainder for KV cache and batch state" on a 256 GiB MI300X configuration.

On an 8× H100 80 GB node, the math works differently: total HBM is 640 GB. FP8 weights consume roughly 83 GB total (sharded across 8 GPUs), leaving the remainder for KV cache, activations, and CUDA overhead. vLLM's PagedAttention allocates KV cache as a pool of fixed-size pages, which means the freed weight memory directly converts into additional concurrent sequence capacity.

Production Note: KV cache growth is the primary memory variable at runtime. A 32K-token context at BF16 KV cache requires approximately 2× the memory of the same context at FP8 KV cache (if KV cache quantization is also enabled). Monitor nvidia-smi for HBM growth as concurrent requests increase — the KV cache pool, not the weights, is what causes OOM at scale. Use --kv-cache-dtype fp8 in addition to --quantization fp8 if you need to push KV cache headroom further.

Why throughput gains are real but hardware-bound

The throughput gains from FP8 are genuine on H100 and MI300X because both architectures have native FP8 tensor core support.

Watch Out: An A100 has no hardware FP8 tensor core support. Passing --quantization fp8 on an A100 will not engage hardware-accelerated kernels. The compute will fall back to emulated or BF16 paths, eliminating the throughput benefit while adding quantization overhead. For A100 deployments of DeepSeek-V3, serve in BF16 and plan for more GPUs or shorter context windows. The hardware compatibility constraint matters as much as model size when planning an FP8 deployment.

The arXiv inference performance study benchmarking vLLM "across three GPU architectures (A6000, A100, H100) in seven deployment scenarios" confirms that GPU architecture changes the result materially — H100 is not just a faster A100 for this workload, it is a qualitatively different execution environment for FP8 serving.

Common failure modes when serving DeepSeek-V3 in FP8

Most failures fall into two categories: quantization falling back silently, and model architecture mismatches at load time. Both are detectable before you send a single inference request.

Watch Out: vLLM's startup logs are the primary diagnostic surface. Look for lines containing quantization and dtype during model loading. If you see quantization=None when you passed --quantization fp8, the kernel path failed silently. If the model config reports an unexpected architecture class, the MoE routing will be wrong or absent.

# Check the effective dtype and quantization after server start
$ curl http://localhost:8000/v1/models | python -m json.tool
# Look for: "id": "deepseek-v3" and confirm no error in server stderr

When FP8 is unavailable and vLLM falls back to a safer dtype

If vLLM cannot execute FP8 W8A8 kernels on the selected GPU, the documented fallback is BF16. The vLLM FP8 docs frame dynamic quantization as a path from BF16/FP16 — implying BF16 is the defined baseline. The DeepSeek-V3 repository confirms both modes are supported: "vLLM v0.6.6 supports DeepSeek-V3 inference for FP8 and BF16 modes on both NVIDIA and AMD GPUs."

Watch Out: Falling back to BF16 doubles weight memory consumption and eliminates the KV cache headroom gains. On an 8× H100 node, this is survivable; on a tighter GPU allocation, it can cause OOM. If you need to run BF16, set --dtype bfloat16 explicitly and remove --quantization fp8 rather than relying on silent fallback behavior — explicit configuration makes failures reproducible.

When MoE support or model tags do not line up

DeepSeek-V3 is an MoE architecture, and vLLM's MoE support path is version-specific. The docs confirm the architecture is supported, but "vLLM v0.6.6 supports DeepSeek-V3 inference for FP8 and BF16 modes" — later releases may add improvements or change defaults.

Production Note: Before deploying, confirm two things together: (1) the model's config.json lists architectures: ["DeepseekV3ForCausalLM"], and (2) your installed vLLM version's supported models page lists DeepseekV3ForCausalLM. A mismatch between model tag, architecture revision, and vLLM release is the most common reason serving silently degrades to a generic transformer path or raises a ValueError on MoE layer initialization. Checkpoint format also matters — the DeepSeek-V3 repo's fp8_cast_bf16.py script explicitly requires weights in safetensor format; non-safetensor checkpoints will fail to load.

Verification checklist for serving, latency, and memory usage

After the server starts without errors, run three verification steps before treating the deployment as production-ready: (1) confirm the endpoint responds, (2) confirm FP8 is active, (3) record baseline TTFT and throughput on a known prompt.

# Verify server health
$ curl http://localhost:8000/health
# Expected: HTTP 200

# List loaded models (confirms model name and that the server loaded successfully)
$ curl http://localhost:8000/v1/models

Run a client request against the OpenAI-compatible endpoint

vLLM's HTTP server "implements OpenAI's Completions API, Chat API, and more", so the smoke test uses a standard Chat Completions request. The model field must match --served-model-name from your launch command.

from openai import OpenAI
import time

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-required",  # vLLM does not enforce auth by default
)

start = time.perf_counter()
response = client.chat.completions.create(
    model="deepseek-v3",  # must match --served-model-name
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
    max_tokens=32,
    temperature=0.0,
)
ttft_proxy = time.perf_counter() - start  # includes network; use vLLM metrics for precision

print(response.choices[0].message.content)
print(f"Approximate wall-clock TTFT: {ttft_proxy:.3f}s")
print(f"Prompt tokens: {response.usage.prompt_tokens}, "
      f"Completion tokens: {response.usage.completion_tokens}")

A successful response with 2 + 2 = 4 confirms model loading, FP8 inference path, and the OpenAI-compatible API are all operational.

Record TTFT, throughput, and resident memory before tuning

Pro Tip: Before tuning any performance parameter, record three baseline numbers: TTFT (time to first token) from vLLM's Prometheus metrics at /metrics, generation throughput (tokens/sec) under a sustained load using vllm bench throughput, and peak VRAM via nvidia-smi dmon -s mu. The Red Hat benchmark reference point for context — vLLM v0.7.1 with MLA and FP8 optimizations delivers "up to 3x throughput and 10x memory capacity improvements" for DeepSeek models — represents a ceiling under favorable conditions, not a guaranteed baseline. Measure your actual workload on your actual hardware before tuning batch sizes, KV cache fraction, or context limits.

FAQ: what readers usually ask before deploying this stack

Can DeepSeek-V3 run on vLLM?

Yes. vLLM's documentation lists DeepSeek-V3 as a supported MoE architecture, and the DeepSeek-V3 repository explicitly states vLLM v0.6.6+ supports FP8 and BF16 inference. This is documented behavior, not a community workaround.

Does vLLM support FP8?

Yes, with hardware constraints. vLLM documents FP8 W8A8 dynamic quantization that requires no calibration data. The supported GPU families for hardware-accelerated W8A8 are Hopper (H100, H200) and Ada Lovelace (L40S, RTX 4090). Broader mentions of AMD MI300X support in the DeepSeek-V3 repo are accurate, but AMD support requires the ROCm-specific vLLM backend and separate validation.

What GPU is needed for FP8 inference?

NVIDIA H100 (Hopper architecture) is the primary recommended target per vLLM's FP8 docs. Ada Lovelace GPUs are also officially supported. AMD MI300X is mentioned in both vLLM docs and the DeepSeek-V3 repo but requires the AMD/ROCm backend path.

Is H100 better than A100 for FP8 vLLM serving?

Yes, categorically. The A100 (Ampere) lacks hardware FP8 tensor core support, so --quantization fp8 on an A100 does not engage hardware-accelerated W8A8 kernels. H100 has native FP8 execution. For FP8 W8A8 serving specifically, H100 is not merely faster than A100 — it is the intended target hardware. Serve DeepSeek-V3 on A100 in BF16 if H100 is unavailable.

How do I reduce memory usage when serving DeepSeek-V3?

Three levers, in order of impact: (1) use --quantization fp8 on a Hopper GPU — FP8 weights consume less memory than BF16; (2) add --kv-cache-dtype fp8 to reduce KV cache memory per token; (3) reduce --max-model-len to cap the KV cache allocation. Reducing tensor parallelism is not a viable option for DeepSeek-V3's weight footprint — you need enough GPUs to hold the shards.

Watch Out: The H100 vs A100 distinction is not about speed alone — it determines whether FP8 W8A8 executes at all. Engineers who provision A100 nodes for this workload expecting FP8 benefits will receive BF16 performance at FP8 configuration cost. Always verify architecture with nvidia-smi --query-gpu=name,compute_cap --format=csv before writing the launch command.

Sources & References

Keywords: DeepSeek-V3, vLLM, FP8 W8A8, BF16, PagedAttention, continuous batching, prefix caching, NVIDIA H100, AMD MI300X, Hopper GPUs, Ada Lovelace GPUs, tensor parallelism, KV cache, OpenAI-compatible API

Was this guide helpful?

The weekly brief.

One email each Sunday with what we tested, what we'd buy, and what to skip. No filler.

Share: X · LinkedIn · Reddit

Related reading