Explore MEGABYTE from Meta AI, a novel multi-scale transformer architecture designed to tackle the challenge of modelling sequences of over one million bytes. Traditional large transformer decoders scale poorly to such lengths due to the quadratic cost of self-attention and the expense of large feedforward layers per position, limiting their application to long sequences like high-resolution images or books.

MEGABYTE addresses this by segmenting sequences into patches, employing a large global model to process relationships between patches and a smaller local model for prediction within patches. This design leads to significant advantages, including sub-quadratic self-attention cost, the ability to use much larger feedforward layers for the same computational budget, and improved parallelism during generation. Crucially, MEGABYTE enables tokenization-free autoregressive sequence modelling at scale, simplifying processing and offering an alternative to methods that can lose information or require language-specific heuristics.

The architecture demonstrates strong performance across various domains, competing with subword models on long context language modelling, achieving state-of-the-art density estimation on ImageNet, and effectively modelling audio from raw files. While promising, the current experiments are conducted at a scale below the largest state-of-the-art language models, indicating that future work is needed to fully explore scaling MEGABYTE to even larger models and datasets.

Learn how MEGABYTE is advancing the frontier of efficient, large-scale sequence modelling.

[https://proceedings.neurips.cc/paper_files/paper/2023/file/f8f78f8043f35890181a824e53a57134-Paper-Conference.pdf]