Qwen3-Omni training with offline-extracted audio-enabled video#

The default Qwen3-Omni recipe (docs/examples/qwen3_omni_moe.md) feeds raw video files to the framework; VeOmni then decodes frames and pulls the audio track out of the container at training time.

This recipe shows the offline-A/V path:

each audio-enabled video has already been decoded into a list of frame bytes plus its matching audio bytes by a preceding data pipeline;
the video and audio are still a single A/V unit aligned in time;
the Qwen3-Omni processor interleaves their tokens via the standard use_audio_in_video=True path — same final input as the raw-video recipe, just with the decode work moved offline.

Use this when frame sampling, shot detection, or audio extraction is done ahead of training and you don’t want to re-decode every epoch.

Looking for independent video + audio (e.g. a silent video plus an unrelated voice query)? That’s a different shape — see the Standalone audio turns section below; the same recipe also accepts standalone audio via sample["audios"].

1. Data shape#

Each sample is a dict whose videos entries are paired-A/V dicts:

{
  "videos": [
    {
      "frames": ["<png-bytes-frame-0>", "<png-bytes-frame-1>", "..."],
      "audio":  "<wav-bytes>",
      "video_fps": 2.0,
      "audio_fps": 16000
    }
  ],
  "conversations": [
    {"from": "human", "value": "<video>\nWhat is happening in the clip?"},
    {"from": "gpt",   "value": "Someone is speaking near a car."}
  ]
}

Key points:

videos[i] is a dict with at least frames (List[bytes] of PNG/JPEG-encoded frames) and audio (WAV-encoded bytes, or a 1-D np.ndarray of mono samples). video_fps falls back to mm_configs.fps. For audio_fps:
- WAV bytes → audio_fps is read from the WAV header automatically; you can still override by setting audio_fps explicitly.
- 1-D ndarray → audio_fps is required (we raise rather than silently drop the audio downstream).
The single <video> marker per A/V item binds the entire dict — there is no separate <audio> marker for the paired audio. The Qwen3-Omni processor sees a non-empty per-video audio_length and emits an interleaved <vision_bos><audio_bos> … <|video_pad|> / <|audio_pad|> chunks … <audio_eos><vision_eos> sequence (the standard omni path).
The registered preprocessor is qwen_omni_offline_av — set data.source_name: qwen_omni_offline_av in the YAML config.

Standalone audio turns#

If a turn carries audio that is not paired with any video (a voice query, a sound effect, narration over a silent clip, …), use the existing sample["audios"] field and an <audio> marker:

{
  "videos": [{"frames": [...], "audio": "<wav-bytes>"}],
  "audios": ["<voice_query.wav>"],
  "conversations": [
    {"from": "human", "value": "<video>\nDescribe this. Also listen: <audio>"},
    {"from": "gpt",   "value": "..."}
  ]
}

<video> consumes the paired-A/V dict; <audio> consumes the standalone clip — independent token positions. Same applies to <image> / sample["images"].

2. Offline extraction utility#

To convert an existing .mp4 corpus into the paired-A/V format above, decode frames at the target fps and pull the audio track once per clip. Example using torchcodec + soundfile:

from io import BytesIO

import PIL.Image
import soundfile as sf
import torch
from torchcodec.decoders import AudioDecoder, VideoDecoder


def extract_av(video_path: str, target_fps: float = 2.0, audio_sr: int = 16000) -> dict:
    # frames
    vdec = VideoDecoder(video_path, device="cpu")
    meta = vdec.metadata
    src_fps = meta.average_fps
    n = max(1, int(meta.num_frames * target_fps / src_fps))
    idx = torch.linspace(0, meta.num_frames - 1, n).round().long().tolist()
    frames_tensor = vdec.get_frames_at(idx).data  # (T, C, H, W) uint8 RGB
    frames_bytes = []
    for f in frames_tensor:
        img = PIL.Image.fromarray(f.permute(1, 2, 0).numpy())
        buf = BytesIO()
        img.save(buf, format="PNG")
        frames_bytes.append(buf.getvalue())

    # audio
    adec = AudioDecoder(video_path, sample_rate=audio_sr)
    audio_tensor = adec.get_all_samples().data.mean(dim=0)  # (T,) mono
    audio_buf = BytesIO()
    sf.write(audio_buf, audio_tensor.numpy(), audio_sr, format="WAV")
    audio_bytes = audio_buf.getvalue()

    return {
        "frames": frames_bytes,
        "audio": audio_bytes,
        "video_fps": target_fps,
        "audio_fps": audio_sr,
    }


def build_sample(video_path: str, prompt: str, answer: str) -> dict:
    return {
        "videos": [extract_av(video_path, target_fps=2.0)],
        "conversations": [
            {"from": "human", "value": f"<video>\n{prompt}"},
            {"from": "gpt", "value": answer},
        ],
    }

Persist samples in any format your dataset class consumes (Parquet, JSONL with base64-encoded bytes, Energon shards, …). If you store frames or audio as files on disk, read them into bytes at sample-construction time — the loader only consumes the in-memory bytes/ndarray shapes.

3. Configuration#

A ready-made config lives at configs/multimodal/qwen3_omni/qwen3_omni_offline_av.yaml. The key behavioral differences from the default qwen3_omni.yaml are:

data:
  source_name: qwen_omni_offline_av    # uses the new preprocessor
  mm_configs:
    use_audio_in_video: True            # paired A/V → interleaved tokens

Token interleaving is decided per video position by the processor at veomni/models/transformers/qwen3_omni_moe/processing_qwen3_omni_moe.py:159 (use_audio_in_video = audio_length != 0). For the offline-A/V recipe each video dict carries its own audio, so audio_length > 0 and the per-position decision is “interleave” — matching the raw-video path.

The shipped data manifest at configs/multimodal/data/qwen3_omni_offline_av.yaml is a placeholder — open it and replace /path/to/your_offline_av_dataset with your real source path. Mix multiple sources by adding more entries to sources / names and rebalancing weights.

4. Launching training#

bash train.sh tasks/train_vlm.py \
    configs/multimodal/qwen3_omni/qwen3_omni_offline_av.yaml \
    --model.model_path Qwen3-Omni-30B-A3B-Instruct

Point model_path at the stock HF checkpoint — no offline MoE merge required. The runtime CheckpointTensorConverter registered on the Qwen3-Omni modeling class folds per-expert HF safetensor keys into VeOmni’s fused gate_up_proj / down_proj layout at load time. See docs/transformers_v5/transformers_v5_moe_weight_loading.md for the format matrix and how to convert a VeOmni-format checkpoint back to per-expert HF keys for inference engines.

5. Smoke-testing the data path#

A self-contained smoke test ships at tests/data/multimodal/test_qwen3_omni_offline_av.py. It verifies:

The qwen_omni_offline_av preprocessor splits <video> / <audio> / <image> markers in declaration order.
_dict_to_video_audio on a {"frames": [...], "audio": <wav-bytes>} dict produces a 4-D video tensor and a non-empty mono audio array.
(Optional, gated on QWEN3_OMNI_MODEL_PATH) The full process_sample_qwen_omni pipeline produces an input_ids whose <|video_pad|> and <|audio_pad|> token runs are interleaved (the standard Qwen3-Omni omni layout), and video_grid_thw matches the sampled frame count.

Run:

# Cheap unit checks (no model weights needed)
pytest tests/data/multimodal/test_qwen3_omni_offline_av.py -v -s

# Full processor path (needs the Qwen3-Omni processor on disk)
QWEN3_OMNI_MODEL_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \
    pytest tests/data/multimodal/test_qwen3_omni_offline_av.py -v -s

The end-to-end test does not need the 30B model weights — only the processor / tokenizer / config files from the HF checkpoint.

Qwen3-Omni training with offline-extracted audio-enabled video

Contents