Qwen/Qwen3-235B-A22B

Overview

Qwen3-235B-A22B is the flagship instruct MoE model in the Qwen3 series, with 235B total parameters and 22B active parameters. This guide covers deploying the model efficiently with vLLM on Vastai GPUs.

Prerequisites

• vLLM version: 0.17.0
• Config: GPUs = 4x VA16, Precision = FP8, TP = 16, max-model-len = 128K

Example config only. Refer to above for others.

Start Docker Container

docker run \
  --privileged=true \
  --name vllm_service \
  --shm-size=256g \
  --ipc=host \
  -p 8000:8000 \
  -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
  bash

Launching the Server

FP8 on 4x VA16

export LLM_MAX_PREFILL_SEQ_LEN="102400"
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
vllm serve Qwen/Qwen3-235B-A22B-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 16 \
  --max-model-len 131072 \
  --hf-overrides '{"rope_parameters":{"factor":4,"original_max_position_embeddings":32768,"rope_theta":1000000,"rope_type":"yarn"}}' \
  --enable-auto-tool-choice --tool-call-parser hermes \
  --enforce-eager 

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3-235B-A22B-FP8",
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

References

• Qwen3-235B-A22B on Hugging Face
• FP8 checkpoint
• vLLM documentation