Ai2 shows how to convert existing AI models to read text as raw bytes

Ai2, the Allen Institute for AI, has published research in Nature on a way to change how an existing AI language model reads text. Instead of word chunks, the model works with raw bytes, the basic codes computers use for characters. The method needs only a relatively short extra training run, not a full rebuild. Ai2 also released new models and its training recipe for free.

How most models read text

Most language models do not process text directly. They first break it into subwords, which are word chunks from a fixed vocabulary. That works well, but it can make models less flexible with spelling, unusual strings, rare words, or text that does not fit those chunks cleanly.

Byte-level models take another route. They work from the byte sequences behind letters, punctuation, symbols and other characters. Ai2 says this can improve a model's grasp of fine details, from whitespace to different writing systems, without tying it to a fixed vocabulary.

What changed

Until now, matching the performance of subword models usually meant training a byte-level model from scratch. That is expensive, and it makes it hard to keep up as subword models improve. Ai2 calls its alternative byteifying. It takes a capable subword model and converts it into a byte-level one.

Ai2 first did this with its own Olmo models, producing Bolmo 1B and Bolmo 7B. The Nature paper shows the process also works on other families. Applied to Qwen 3 8B and Llama 3 8B, it produced Bwen 8B and Blama 8B. Both come close to matching the models they came from. Bwen 8B is the strongest so far, ahead of Bolmo 7B on Ai2's combined tests.

What is available

• New checkpoints (saved versions of the models) on Hugging Face
• Stage 1 checkpoints, which keep the original model frozen while the new byte-level parts train
• The full training recipe, open to anyone

Who it affects

This is research, not a product. The people who gain first are researchers and developers. Some have already adapted Bolmo, including a Bolmo 7B version tuned for Russian-to-English poetry and lyric translation.

For everyone else, it points to a possible future. Ai2 says models might adapt how they represent text across languages and tasks. Because bytes also underlie other digital data, it says the same ideas could eventually apply to images and audio.

What is not known yet

Bwen 8B and Blama 8B come close to their originals, but the material does not say they match them. Researchers are still studying the tradeoff between the efficiency of byte designs like Bolmo and fine-grained character understanding. No product plans or dates were shared.