LTX-2.3 training guide#

Download model#

Download the LTX-2.3 transformer weights and the Gemma3 text encoder:

# LTX-2.3 transformer weights
python3 scripts/download_hf_model.py \
    --repo_id Lightricks/LTX-2.3 \
    --local_dir /path/to/models

# Gemma3 text encoder (required for conditioning)
python3 scripts/download_hf_model.py \
    --repo_id google/gemma-3-12b-it-qat-q4_0-unquantized \
    --local_dir /path/to/models

The download helper appends each Hugging Face repository name to --local_dir, producing /path/to/models/LTX-2.3 and /path/to/models/gemma-3-12b-it-qat-q4_0-unquantized in this example.

Prepare Dataset#

Use the built-in preprocessing pipeline to split videos, generate captions, and compute latents/embeddings:

Step 1: Split scenes (optional)#

Split raw videos into scene clips using PySceneDetect:

python veomni/models/diffusers/ltx2_3/ltx_condition/preprocess_dataset.py split-scenes \
    --video_dir /path/to/raw/videos \
    --output_dir /path/to/output/clips

Step 2: Generate captions#

Auto-caption video clips using a multimodal model (Qwen2.5-Omni by default):

python veomni/models/diffusers/ltx2_3/ltx_condition/preprocess_dataset.py caption \
    --input_dir /path/to/output/clips \
    --output /path/to/output/clips/dataset.json

The captioner writes each media_path as a filename relative to the dataset file’s directory. Keep dataset.json beside the clips so later stages resolve those paths correctly.

Step 3: Generate reference videos (optional, for IC-LoRA)#

Generate Canny edge reference videos before preprocessing so their paths are written to the dataset file:

python veomni/models/diffusers/ltx2_3/ltx_condition/preprocess_dataset.py compute-reference \
    --input_dir /path/to/output/clips \
    --dataset_file /path/to/output/clips/dataset.json

Step 4: Compute text embeddings and VAE latents#

Compute text embeddings + VAE latents from the dataset file:

python veomni/models/diffusers/ltx2_3/ltx_condition/preprocess_dataset.py preprocess \
    --dataset_file /path/to/output/clips/dataset.json \
    --gemma_model_path /path/to/models/gemma-3-12b-it-qat-q4_0-unquantized \
    --checkpoint_path /path/to/models/LTX-2.3 \
    --resolution_buckets "960x544x49" \
    --with_audio

The shipped AV configs set condition_model_cfg.with_audio: true, so their preprocessing command must include --with_audio. For video-only training, set that config field to false and omit the flag. For IC-LoRA, append --reference_column reference_path to the command above; this encodes the reference videos generated in Step 3.

Step 5: Pack precomputed files#

Pack precomputed .pt files into parquet shards for offline training:

python veomni/models/diffusers/ltx2_3/ltx_condition/preprocess_dataset.py save-parquet \
    --precomputed_dir /path/to/output/clips/.precomputed \
    --output_dir /path/to/output/parquet_output \
    --pad_to_multiple_of 8 \
    --with_audio

Use --with_audio whenever the training config has audio enabled. Reference latents are included automatically when reference_latents/ exists.

Output directory structure:

output/
├── clips/
│   ├── dataset.json          # Captions + clip-relative media paths
│   ├── *.mp4                 # Scene-split video clips
│   ├── *_reference.mp4       # Optional IC-LoRA reference videos
│   └── .precomputed/
│       ├── audio_latents/    # Optional, for AV training
│       ├── conditions/       # Gemma text embeddings
│       ├── latents/          # VAE-encoded video latents
│       └── reference_latents/# Optional, for IC-LoRA training
└── parquet_output/           # Parquet shards (from save-parquet)
    ├── shard_0000.parquet
    ├── shard_0001.parquet
    └── ...

Update config paths#

Before training, update the model and data paths in the config file:

# configs/dit/ltx2_av_lora.yaml
model:
  model_path: "/path/to/models/LTX-2.3"
  condition_model_path: "/path/to/models/LTX-2.3"

data:
  train_path: "/path/to/output/parquet_output"

The Gemma path is used by the preprocessing command; the shipped offline training configs consume the precomputed embeddings and do not reload Gemma.

Start training#

Audio-Video LoRA (default)#

bash train.sh tasks/train_dit.py configs/dit/ltx2_av_lora.yaml

Audio-Video LoRA (Low VRAM)#

For GPUs with limited VRAM, use the low-memory configuration with reduced LoRA rank (16 vs 32):

bash train.sh tasks/train_dit.py configs/dit/ltx2_av_lora_low_vram.yaml

Video-to-Video (IC-LoRA)#

For video-to-video transformations (e.g., depth-to-video, style transfer), use the IC-LoRA configuration. This requires reference videos in your dataset:

bash train.sh tasks/train_dit.py configs/dit/ltx2_v2v_ic_lora.yaml

LTX-2.3 training guide

Contents