Every time you want a transformer to reason better, the standard move is to make it bigger. More layers, more parameters, more GPUs, more money. That works—but it's a blunt instrument.
Here's the tip: use a Recurrent Looped Transformer (RLT) to get deeper computation without adding parameters. Instead of stacking new layers, an RLT runs the same transformer block over and over, reusing its weights each pass. You get more reasoning steps for free on the parameter budget.
The official project page lives at yifanzhang-pro/recurrent-looped-tranformer (yes, "tranformer"—more on that typo later). In the next few minutes, you'll learn what RLT actually is, why it's worth your attention, and how to start experimenting with the repo today.
Key Takeaway: Depth of computation and number of parameters are not the same thing. RLT decouples them.
A Recurrent Looped Transformer is a transformer that loops over the same block multiple times. You feed in your input, run it through one transformer block, then feed the output back through that same block, and repeat for some number of iterations.
The critical difference from a standard transformer is weight sharing across loops. A 12-layer transformer has 12 distinct sets of weights. An RLT with one block looped 12 times has one set of weights applied 12 times. Same computational depth, a fraction of the parameters.
This isn't a brand-new idea. It connects directly to Universal Transformers (Dehghani et al., 2018), which combined recurrence with transformers, and to Adaptive Computation Time (Graves, 2016), which lets a model decide how many steps to take per input. RLT sits in that lineage—recurrence inside a transformer, with the loop count as a knob you can tune or learn.
There are three concrete reasons to care.
Parameter efficiency. Sharing weights across looped steps means you can cut model size substantially while preserving depth. For anyone deploying models under memory constraints, that's a real advantage.
Adaptive computation. The number of loops doesn't have to be fixed. A halting mechanism can let easy examples exit after two loops and hard ones run ten. You spend compute where it's needed instead of uniformly across every input.
Better iterative reasoning. Tasks that require refining an answer step by step—solving a Sudoku grid, working through multi-step math, synthesizing and verifying code—map naturally onto a loop. The model gets to "think again" using the same machinery, which is exactly what iterative refinement looks like.
Key Takeaway: RLT trades uniform width for reusable depth. That's a good trade on reasoning-heavy tasks.
1. Go to the repository. Visit github.com/yifanzhang-pro/recurrent-looped-tranformer. Note the spelling: it's "tranformer," not "transformer." The typo is in the repo name itself and doesn't affect anything inside.
2. Explore what's there. The project page is set up as an official hub—expect code, model checkpoints, and documentation aimed at reproducing results. It may also link out to papers, blog posts, or demos. Start with the README, then move on to the training scripts.
3. Train with gradient checkpointing. Backpropagating through looped steps is memory-hungry because you're effectively unrolling the loop. Gradient checkpointing—recomputing activations during the backward pass instead of storing them—is the standard way to keep this manageable. If you hit OOM errors, reach for it first.
4. Experiment with halting. Try a fixed loop count first to establish a baseline, then add a halting mechanism that decides when to stop per input. Compare compute spent versus accuracy gained. This is where the adaptive-computation payoff shows up.
5. Test on the right tasks. Algorithmic problems, math reasoning, and program synthesis are the natural fits. Language modeling with long-range dependencies is another. Don't expect magic on tasks that don't need multi-step inference.
Key Takeaway: Start with a fixed loop count, add gradient checkpointing early, then layer in halting once you have a baseline.
"It's just a deeper transformer." No. A deeper transformer has more distinct weights. RLT reuses the same weights—that's the entire point.
"Looping means it processes tokens sequentially like an RNN." Not true. Within each loop, the transformer still processes the sequence in parallel via attention. The recurrence is across loop iterations, not across tokens.
"The repo name typo is a red flag." It's a typo in a URL, not a bug in the code. Navigate to the correct spelling and move on.
"RLT is identical to a Universal Transformer." They're closely related—both use recurrence and weight sharing—but RLT is a specific implementation with its own design choices around looping and halting. Treat Universal Transformers as prior art, not an equivalence.
What is a Recurrent Looped Transformer? A transformer architecture that applies the same block repeatedly in a loop, sharing weights across iterations to gain computational depth without adding parameters.
How is RLT different from a standard transformer? Standard transformers use distinct weights per layer. RLT reuses one set of weights across looped steps.
What is the purpose of the official project page? To host code, checkpoints, and documentation for reproducing and building on RLT results.
Can RLT be trained end-to-end? Yes, via backpropagation through the looped steps. Gradient checkpointing is commonly needed for memory efficiency.
What tasks benefit from RLT? Iterative reasoning tasks: algorithmic problems, math, program synthesis, and long-range language modeling.
Is RLT the same as a Universal Transformer? Related but not identical. Universal Transformers are prior work combining recurrence with transformers; RLT is a specific take on the idea.
Where can I find the official project page?
github.com/yifanzhang-pro/recurrent-looped-tranformer.
Are there pre-trained models available? The project page is set up to include checkpoints. Check the repo for what's currently released.
The core idea is simple: you don't need more parameters to get more reasoning—you need more passes through the weights you already have. RLT makes that concrete, with weight sharing, adaptive loop counts, and a natural fit for step-by-step problems.
Pick a task that requires multi-step inference—a Sudoku solver, a math word problem, a small program synthesis benchmark—and try it with a looped setup. Compare parameter count and accuracy against a standard transformer of equivalent depth. The difference is usually instructive.
Visit the official project page at github.com/yifanzhang-pro/recurrent-looped-tranformer to access code, checkpoints, and documentation. Start experimenting with RLT today and share your results.