Transformers v5 Notes

Contents

Transformers v5 Notes#

This section documents VeOmni’s integration with HuggingFace transformers==5.9.0 (the only supported transformers version).

Included Notes#

  • Fused attention interface: documents the unified Flash/Flex facade, config routing, native BlockMask contract, and Ulysses restrictions.

  • Flash Attention custom-name handling: explains why _lazy_imports fails for VeOmni custom attention names and how the local hub-kernel loader adapter resolves it.

  • MoE weight loading: explains how VeOmni expects MoE expert weights to be laid out and documents qwen3_moe handling.

  • Testing a new model: SOP for adding test cases in test_models_patch.py and test_e2e_parallel.py when onboarding a new model.

The modeling code generation workflow (patchgen) has graduated out of this section — it is no longer tied to the transformers v5 cutover and now lives at design/patchgen.md.

Contents