deepseek-ai/DeepSeek-V3.1

Overview

DeepSeek-V3.1 is a hybrid MoE model that supports both thinking and non-thinking modes. You can dynamically switch between the two modes from the client by passing extra_body={"chat_template_kwargs": {"thinking": True|False}}.

Prerequisites

• vLLM version: 0.17.0
• Config: GPUs = 8x VA16, Precision = FP8, TP = 32, max-model-len = 64K

Example config only. Refer to above for others.

Start Docker Container

docker run \
  --privileged=true \
  --name vllm_service \
  --shm-size=256g \
  --ipc=host \
  -p 8000:8000 \
  -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
  bash

Launching the Server

FP8 on 8x VA16, Reasoning, Tool Calling and MTP Speculative Decoding

vllm serve deepseek-ai/DeepSeek-V3.1 \
  --trust-remote-code \
  --tensor-parallel-size 32 \
  --max-model-len 65536 \
  --enable-auto-tool-choice --tool-call-parser deepseek_v31 \
  --chat_template tool_chat_template_deepseekv31.jinja \
  --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}' --no-async-scheduling  \
  --reasoning-parser deepseek_r1 \
  --enforce-eager

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
    extra_body={
      "chat_template_kwargs": {"thinking": True}  # Set to True for 'thinking' mode, False for 'non-thinking' mode
    },
)
print(resp.choices[0].message.content)

References

• DeepSeek-V3.1 on Hugging Face
• vLLM tool chat template for DeepSeek-V3.1