Taming Outlier Tokens in Diffusion Transformers
Apple researchers identify a hidden inefficiency in Diffusion Transformers that could reshape how image-generation models are built.
Why it matters
Apple's research surfaces a technical problem (outlier tokens) in modern DiT architectures used for image generation. Understanding and fixing this could improve efficiency and quality — actionable for practitioners building or optimizing generative models.
The key facts
11 to knowOutlier tokens identified in both encoder and denoiser of RAE-DiT pipelines
Phenomenon appears especially in intermediate layers of DiTs
Prior ViT work on outliers extended to generative model context for first time
Published by Apple ML research team
Focus on Representation Autoencoder (RAE) DiT architecture
Study addresses underexplored role of outliers in generative models
Outlier tokens appear in both encoder and denoiser of RAE-DiT pipelines
Phenomenon especially pronounced in intermediate layers of DiTs
Pretrained ViT encoders produce outlier representations that carry limited local information
Research from Apple Machine Learning (published August 2026)
Addresses inefficiency in modern image generation architectures
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: We study outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work has shown that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models…