deepseek-ai/DeepSeek-R1
DeepSeek-R1 is a 671B-parameter MoE reasoning model built on the DeepSeek-V3 architecture, trained with large-scale reinforcement learning for strong chain-of-thought capabilities.
Open-weights RL-trained reasoning model with native FP8. Verified on 8x/16x VA16
View on HuggingFaceGuide
Overview
DeepSeek-R1 is a large language model with 671B parameters. The model has demonstrated outstanding performance across various benchmark evaluations, including mathematics, programming, and general logic.
Prerequisites
- vLLM version: 0.17.0
- Config: GPUs = 16x VA16, Precision = FP8, TP = 32, max-model-len = 128K
Example config only. Refer to above for others.
Start Docker Container
docker run \
--privileged=true \
--name vllm_service \
--shm-size=256g \
--ipc=host \
-p 8000:8000 \
-it \
-v ~/.cache/huggingface:/root/.cache/huggingface \
harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
bash
Launching the Server
FP8 on 16x VA16
export LLM_MAX_PREFILL_SEQ_LEN="102400"
export FUSE_ALL_DECODER_LAYERS="0"
vllm serve deepseek-ai/DeepSeek-R1 \
--trust-remote-code \
--tensor-parallel-size 32 \
--pipeline-parallel-size 2 --no-async-scheduling \
--max-model-len 131072 \
--enforce-eager
Client Usage
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1",
messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
max_tokens=512,
)
print(resp.choices[0].message.content)