Qwen/Qwen3-Reranker

OverView

Qwen3-Reranker is a text reranking model that provides dense vector representations of text. It is part of the Qwen3 series, which offers models of varying sizes to suit different use cases.

Prerequisites

• vLLM version: 0.17.0
• Config: GPUs = 1x VA16, TP = 4

Example config only. Refer to above for others.

Start Docker Container

docker run \
  --privileged=true \
  --name vllm_service \
  --shm-size=256g \
  --ipc=host \
  -p 8000:8000 \
  -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
  bash

Launching the Server

8B on 1x VA16

vllm serve /models/Qwen3-Reranker-8B \
  --served-model-name Qwen3-Reranker-8B \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enforce-eager

Client Usage

import requests

base_url="http://localhost:8000/v1"
model="Qwen3-Reranker-8B"

# Define the query and documents
query = "What are the key benefits of using Retrieval-Augmented Generation (RAG) over traditional fine-tuning approaches for enterprise applications?"

documents = [
    "Retrieval-Augmented Generation (RAG) enhances large language models by retrieving relevant information from an external knowledge base before generating a response. This approach significantly reduces hallucinations and improves factual accuracy, as the model is grounded in verifiable data rather than relying solely on its parametric memory. For enterprise applications, this means you can update knowledge simply by modifying the document database without expensive retraining.",
    
    "Compared to traditional fine-tuning, RAG allows for more efficient knowledge updates. Instead of retraining the entire model, you can simply add new documents to the vector store. This makes RAG particularly suitable for domains with frequently changing information, such as technical documentation, pricing updates, or regulatory compliance materials in enterprise settings.",
    
    "Fine-tuning is a common technique for adapting pre-trained language models to specific tasks. It involves updating the model's weights on a task-specific dataset. While effective, fine-tuning requires substantial computational resources and labeled data, and the model's knowledge remains static after training, making it less adaptable to changing enterprise requirements.",
    
    "The Transformer architecture, introduced in the 'Attention Is All You Need' paper, revolutionized natural language processing. It uses self-attention mechanisms to process sequences in parallel, enabling more efficient training and better handling of long-range dependencies compared to recurrent neural networks.",
    
    "RAG systems significantly reduce operational costs for enterprises by eliminating the need for continuous model retraining. When business policies or product information changes, companies can simply update their knowledge base documents rather than undergoing expensive fine-tuning processes that require specialized ML expertise and GPU resources.",
    
    "Enterprise RAG implementations provide built-in citation capabilities, allowing users to verify information sources directly. This audit trail is crucial for compliance-sensitive industries like finance and healthcare, where traditional fine-tuned models cannot provide source attribution for their generated responses."
]

instruction = "Please rank the following documents based on their relevance to the query."

def qwen3_rerank(
    base_url: str,
    model: str,
    query: str,
    documents: list[str],
    instruction: str = "", 
    top_n: int = 5
):
    url = f"{base_url.rstrip('/')}/rerank"

    payload = {
        "model": model,
        "query": query,
        "documents": documents,
        "instruction": instruction, 
        "top_n": top_n
    }
    
    resp = requests.post(url, json=payload)
    resp.raise_for_status()
    return resp.json()

response = qwen3_rerank(
    base_url=base_url,
    model=model,
    query=query,
    documents=documents,
    instruction=instruction,
)
print("\n========== Reranker result ===========\n")

for i, item in enumerate(response["results"]):
    print(f"Rank: {i+1}, score: {item['relevance_score']}, Document: {item['document']['text'][:40]}...\n")

References

• Qwen3-Reranker-0.6B on Hugging Face
• Qwen3-Reranker-4B on Hugging Face
• Qwen3-Reranker-8B on Hugging Face