vLLM x VastAI/Recipes
Qwen

Qwen/Qwen3-30B-A3B

Smaller Qwen3 MoE model (30B total / 3B active) with 128 experts (8 routed + 1 shared), gated delta networks architecture, and 128K context on Vastai Devices

Compact Qwen3 MoE with 3B active parameters

View on HuggingFace
moe30B / 3B131,072 ctxvLLM 0.17.0text
Guide

Overview

Qwen3-30B-A3B is the new generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models.

Prerequisites

  • vLLM version: 0.17.0
  • Config: GPUs = 1x VA16, Precision = FP8, TP = 4, max-model-len = 128K

Example config only. Refer to above for others.

Start Docker Container

docker run \
  --privileged=true \
  --name vllm_service \
  --shm-size=256g \
  --ipc=host \
  -p 8000:8000 \
  -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
  bash

Launching the Server

FP8 on 1x VA16

export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice --tool-call-parser hermes \
  --hf-overrides '{"rope_parameters":{"factor":4,"original_max_position_embeddings":32768,"rope_theta":1000000,"rope_type":"yarn"}}' \
  --enforce-eager

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3-30B-A3B-FP8",
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

References

Qwen/Qwen3-30B-A3B | vLLM Recipes