Qwen/Qwen3-0.6B
Qwen3-0.6B is a large language model with 0.6B parameters, capable of performing text-to-text generation with a wide range of tasks. The model is trained on a large amount of data, and is capable of handling long-form inputs.
Compact Qwen3-0.6B model
View on HuggingFaceGuide
Overview
Qwen3-0.6B is a large language model with 0.6B parameters,capable of performing text-to-text generation with a wide range of tasks. The model is trained on a large amount of data, and is capable of handling long-form inputs.
Prerequisites
- vLLM version: 0.17.0
- Config: GPUs = 1x VA16, Precision = BF16, TP = 4, max-model-len = 32K
Example config only. Refer to above for others.
Start Docker Container
docker run \
--privileged=true \
--name vllm_service \
--shm-size=256g \
--ipc=host \
-p 8000:8000 \
-it \
-v ~/.cache/huggingface:/root/.cache/huggingface \
harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
bash
Launching the Server
BF16 on 1x VA16
vllm serve Qwen/Qwen3-0.6B \
--trust-remote-code \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--enable-auto-tool-choice --tool-call-parser hermes \
--enforce-eager
Client Usage
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
model="Qwen/Qwen3-0.6B",
messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
max_tokens=512,
)
print(resp.choices[0].message.content)