Encoder-decoder models

Both stacks are built from the same transformer blocks. The decoder differs in two places, a mask on its own attention and a second attention layer pointed at the encoder's output.

Splitting the work in half has two consequences.

The input is read once, in both directions, and the result is reused for every output token. A decoder-only model with a KV cache also reads its prompt once, in one pass called prefill, and reuses the stored attention keys and values at every step after that, so both shapes pass over the input once. The difference is in how the input is read and what each output step costs. The encoder lets every input token see every other one, where a decoder-only model reads its prompt left to right. And each output step in an encoder-decoder runs only the decoder stack, which can be much smaller than the encoder.

That second point is the reason the halves can be sized independently. Google's T5Gemma release pairs a 9B encoder with a 2B decoder for exactly this reason, because summarization needs a deep reading of a long input and writes a short output.

T5's pretraining removes random spans from a sentence, replaces each with a numbered sentinel token, and asks the decoder to write the removed spans back in order. The encoder sees the damaged sentence, the decoder writes the missing pieces.

BART (Lewis et al., 2019) does the same job with a different damage function, shuffling sentences and replacing spans with a single mask token. Its paper describes the architecture as one that "can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes", which is the clearest one-line summary of where the shape sits.

Fine-tuning follows the same interface. Each example is an input string and a target string, the loss is the decoder's next-token loss over the target, and the class in transformers is AutoModelForSeq2SeqLM. Classification becomes a target of one word, summarization a target of one paragraph, and the training loop doesn't change between them.

Speech to text

Whisper is an encoder-decoder whose encoder reads a spectrogram and whose decoder writes text. openai/whisper-large-v3 is downloaded about 4.7 million times a month, which makes it one of the most used encoder-decoders in the world.

Translation

Dedicated translation models stayed with the shape. facebook/nllb-200-distilled-600M covers 200 languages and gets about 1 million downloads a month, under a CC-BY-NC-4.0 license that rules out commercial use. Google's MADLAD-400 models are Apache-2.0 and cover more than 400 languages.

Summarization pipelines that already work

facebook/bart-large-cnn still pulls about 1.3 million downloads a month. It's a 2019 model fine-tuned on news summarization, and for the narrow job of turning an article into three sentences it's cheaper than an API call and needs no prompt.

Fine-tuned text-to-text jobs

Flan-T5 is the common starting point when the output is short text and you have examples. Extraction into a fixed format, normalization of messy strings, query rewriting for search. google/flan-t5-base at 248M parameters runs on a CPU.

Encoder-decoder LLMs

T5Gemma and T5Gemma 2 are the current attempt to bring the shape back for general-purpose work, and the versions section below covers what they are and what they cost you in licensing.

What T5Gemma changed

Google published Encoder-Decoder Gemma (Zhang et al., April 2025), which builds an encoder-decoder by copying the weights of a pretrained decoder-only model into both halves, switching the encoder's self-attention from causal to bidirectional, and continuing training with a span-corruption objective. The paper reports that under a similar inference budget the result reaches "comparable (often better) pretraining performance but substantially better finetuning performance than their decoder-only counterpart".

T5Gemma shipped on July 9, 2025, built from Gemma 2. T5Gemma 2 followed on December 18, 2025, built from Gemma 3, with three sizes from 270M-270M up to 4B-4B, a 128K context window, support for over 140 languages and image input. Its two architecture changes both save parameters. The embeddings are tied across the encoder and the decoder, and the decoder's self-attention and cross-attention are merged into one layer. Both releases carry the Gemma license, which has use restrictions that Apache-2.0 models don't.

The size of the model and what it was trained on usually count for more than the shape, and the summary in the next section shows what a 248M model does with an instruction it wasn't fine-tuned for.

AutoModelForSeq2SeqLM loads every text model in the table above the same way. Whisper takes audio, so it has its own class. This one turns a support ticket into a one-sentence summary.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "google/flan-t5-base"                  # 248M, Apache-2.0
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

prompt = ("Summarize this support ticket in one short sentence: "
          "Hi, since yesterday's update the export button on the reports "
          "page does nothing when I click it. I need the CSV for a meeting "
          "on Friday.")
ids = tok(prompt, return_tensors="pt")            # 45 tokens in
out = model.generate(**ids, max_new_tokens=40)
print(tok.decode(out[0], skip_special_tokens=True))
# -> Exporting reports does not work on the CSV page.

The summary gets the problem right, and it also invents a "CSV page" that the ticket never mentions. That's what a 248M model does with an instruction it was never fine-tuned on, and fine-tuning on pairs of tickets and their summaries is the usual next step, and the prompt below runs that comparison on your own input and target pairs.

• It can't take a new instruction as well as a large decoder can. Flan-T5 follows short instructions it was tuned for, and anything outside that needs fine-tuning.
• The input length is the encoder's limit. T5 and BART stop around 512 to 1,024 tokens, so long documents get chunked. T5Gemma 2's 128K is the exception.
• The ecosystem moved on. Serving stacks, quantization formats and agent frameworks are built around decoder-only models, so an encoder-decoder deployment does more of its own plumbing.
• Two stacks are more to fine-tune. You're training an encoder, a decoder and the cross-attention between them, where a decoder-only fine-tune touches one stack and usually only a LoRA adapter inside it.
• Licenses vary more than they look. NLLB's checkpoints are non-commercial, and T5Gemma carries the Gemma terms, where T5 and Flan-T5 are Apache-2.0.
• Small old checkpoints are small old models. A 2019 or 2022 model at a few hundred million parameters gets facts wrong and has no knowledge past its training data.