vLLM x VastAI/Recipes
Qwen

Qwen/Qwen2-Audio-7B-Instruct

Qwen2-Audio a large audio-language model capable of accepting diverse audio signal inputs and performing audio analysis or generating direct textual responses based on speech instructions.

Compact Qwen2 Audio-only model

View on HuggingFace
autoregressive7B8,192 ctxvLLM 0.17.0multimodal
Guide

Overview

Qwen2-Audio-7B-Instruct a large audio-language model capable of accepting diverse audio signal inputs and performing audio analysis or generating direct textual responses based on speech instructions.

Prerequisites

  • vLLM version: 0.17.0
  • Config: GPUs = 1x VA16, Precision = BF16, TP = 4, max-model-len = 8K

Example config only. Refer to above for others.

Start Docker Container

docker run \
  --privileged=true \
  --name vllm_service \
  --shm-size=256g \
  --ipc=host \
  -p 8000:8000 \
  -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  harbor.vastaitech.com/ai_deliver/vllm_vacc:latest \
  bash

Launching the Server

BF16 on 1x VA16

export VNNL_CONV1D_DLC=1
vllm serve Qwen/Qwen2-Audio-7B-Instruct \
  --trust-remote-code \
  --enforce-eager \
  --tensor-parallel-size 4 \
  --max-model-len 8192

Client Usage

Text Chat

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen2-Audio-7B-Instruct",
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

Automatic Speech Recognition

from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1",api_key="EMPTY")

def make_audio_url(audio_file):
    with open(audio_file,"rb") as f:
      audio_base64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:audio/wav;base64,{audio_base64}"

audio_file = ".guess_age_gender.wav"

content=[
    {'type':'audio_url','audio_url':{"url": make_audio_url(audio_file)}},
    {'type':'text','text':"Please transcribe the audio. Just give me the transcription result without any explanation."}
]
messages = [
    {"role": "system", "content": "you are a helpful assistant"},
    {"role": "user", "content": content}
]


response = client.chat.completions.create(
    model="Qwen/Qwen2-Audio-7B-Instruct",
    messages=messages,
    max_tokens=1024,
    temperature=0.6,
)

print(response.choices[0].message.content)

References

Qwen/Qwen2-Audio-7B-Instruct | vLLM Recipes