Every time your processor needs data that isn't in its cache, it stalls. The memory hierarchy—L1, L2, L3 caches, then DRAM—exists precisely to hide this latency, but no cache is perfect. When the CPU misses, it waits hundreds of cycles for data to arrive from main memory. That wait is wasted computation.
Hardware prefetching is the art of predicting what data the processor will need before it asks for it, fetching it into cache ahead of time. Done well, prefetching can hide most of that memory latency. Done poorly, it pollutes the cache with useless data and actually slows things down.
For decades, prefetcher design has been a craft. Experts study memory access patterns, hypothesize about program behavior, write heuristics, and then spend months tuning them against benchmarks. It's slow, expensive work, and the design space is enormous.
ArchAgent v2 is a framework that treats prefetcher design not as a craft, but as a search problem. It uses large language models (LLMs) to propose candidate prefetching policies, evaluates those policies in a cycle-accurate simulator, and then uses reinforcement learning to iteratively refine its proposals. The goal: automate the entire design loop that a human expert would perform manually.
This isn't a toy demonstration. ArchAgent v2 was entered into the Data Prefetching Championship (DPC), the premier competition for prefetching algorithms, and performed competitively against dozens of human-designed systems.
The DPC is the perfect proving ground for AI-driven hardware design. It provides a standardized simulator (ChampSim), a fixed set of benchmark traces, and a clear performance metric. Every submission runs under identical conditions. If your prefetcher is better, the numbers show it.
This article walks through how ArchAgent v2 works, what happened when it competed in DPC-3, and what the results tell us about the future of AI in computer architecture.
We'll start with the fundamentals of prefetching, then explain the DPC competition structure, dive deep into ArchAgent v2's architecture and workflow, present the case study results, and finally discuss the broader implications—and limitations—of using AI agents for hardware design.
Modern processors can execute multiple instructions per cycle, but they can't execute instructions that are waiting on data. When a load instruction misses all caches and has to go to DRAM, the penalty is typically 200-300 cycles. During that time, the processor may stall completely.
Prefetching attacks this problem by predicting future accesses and issuing memory requests early. If the prediction is correct, the data arrives in cache before it's actually needed, and the processor never stalls.
The math is compelling. A prefetcher that achieves a 20% reduction in average memory latency can improve overall performance by 10-15% on memory-intensive workloads. That's a bigger win than most other microarchitectural optimizations.
The classic prefetchers are based on observable patterns:
Each of these has strengths and weaknesses. Stride prefetchers are simple and cheap but useless on irregular code. Markov prefetchers are flexible but generate lots of useless prefetches. Real-world prefetchers are almost always hybrids, carefully tuned to specific workload characteristics.
Designing a good prefetcher is hard for several reasons:
The result is that state-of-the-art prefetchers take months or years of expert effort to develop. The DPC was created to formalize this process and give researchers a common benchmark.
Three metrics dominate prefetcher evaluation:
The critical insight: hit rate is not the goal. A prefetcher that fetches aggressively will get high hit rates but may evict useful data or saturate memory bandwidth. The DPC judges prefetchers on geometric mean speedup, which directly measures end-to-end performance improvement.
Key Takeaway: Prefetching is about hiding memory latency, not maximizing hit rate. A good prefetcher improves IPC by fetching the right data at the right time, not by fetching everything.
The Data Prefetching Championship began in 2019 as a way to standardize prefetcher evaluation. DPC-1 (2019) established the format: a common simulator, a common benchmark set, and a competitive leaderboard. DPC-2 followed in 2021 with more benchmarks and stricter rules. DPC-3 (2023) expanded further, adding cloud workloads and attracting over 100 submissions from academia and industry.
The competition has evolved alongside the field. Early editions were dominated by hand-crafted heuristics from veteran researchers. By DPC-3, a significant portion of submissions used machine learning in some form—a sign of the field's direction.
Every DPC submission runs in ChampSim, a trace-based microarchitectural simulator. ChampSim models a 4-core out-of-order processor with a 256KB L2 cache and a 2MB L3 cache. It's fast enough to run dozens of benchmarks in reasonable time, but detailed enough to capture the interactions between prefetcher, cache hierarchy, and memory controller.
ChampSim is open source and widely used outside the DPC, making it a de facto standard for prefetching research.
DPC-3 uses 40 traces: a mix of SPEC CPU 2017 benchmarks (covering integer and floating-point workloads) and cloud traces from Google and Alibaba data centers. These traces were collected from real systems and represent diverse access patterns—from regular array traversals to chaotic hash table lookups.
The cloud traces are particularly important because they represent modern data-center workloads with irregular, multi-threaded access patterns that differ significantly from traditional HPC benchmarks.
Each prefetcher is run on all 40 traces, and the speedup over a no-prefetcher baseline is computed for each trace. The final score is the geometric mean of these speedups. Geometric mean is used instead of arithmetic mean because it treats all benchmarks equally—a prefetcher that does well on 39 benchmarks and terribly on one won't be unfairly penalized, nor will one that dominates a single benchmark.
The best human-designed prefetcher in DPC-3, called Berti, achieved a geometric mean speedup of approximately 1.12x. ArchAgent v2 achieved approximately 1.08x.
Key Takeaway: The DPC is a controlled experiment. Every prefetcher faces identical conditions, and the geometric mean speedup provides a fair, reproducible ranking.
Human prefetcher designers explore the design space by intuition and trial. ArchAgent v2 formalizes this as a search: generate a candidate design, evaluate it, learn from the results, and generate a better candidate. The framework doesn't try to be clever about how to prefetch—it lets the search process discover good policies.
The LLM component of ArchAgent v2 generates candidate prefetcher implementations. Given a description of the problem (the benchmark characteristics, the current best performance, and a library of known prefetching techniques), the LLM proposes a new policy, often combining existing techniques in novel ways.
This is a critical design choice. The LLM isn't being asked to invent prefetching from scratch—it's being given a toolbox of known techniques and asked to assemble them into a working policy. This is analogous to how a human expert would work, but the LLM can explore combinations much faster.
The LLM proposes, but it doesn't learn. The reinforcement learning component does. After each candidate is simulated, the framework computes a reward based on performance (geometric mean speedup across benchmarks). This reward is fed back into the system, which uses it to guide the next round of proposals.
The key insight is that the RL loop operates on the performance signal, not on the prefetcher's internal decisions. The framework doesn't try to explain why a prefetcher works—it just knows that certain proposals led to good results and biases future proposals accordingly.
ArchAgent v2 is designed to be portable. The agent logic (LLM + RL loop) is completely separate from the simulator interface. To evaluate a candidate, the framework compiles the proposed prefetcher into ChampSim, runs the traces, and extracts performance metrics. This separation means ArchAgent v2 could be adapted to other hardware design problems—cache replacement, branch prediction, memory scheduling—by swapping the simulator and reward function.
The process begins with a prompt to the LLM. The prompt includes: - A description of the benchmark suite (e.g., "40 traces from SPEC CPU 2017 and cloud workloads") - The current best performance achieved so far - A library of known prefetching techniques (stride, Markov, signature-based, etc.) - Constraints (e.g., hardware budget, table sizes)
The LLM generates a complete prefetcher implementation in C++ that plugs into ChampSim's interface.
Each candidate is compiled and run against the full benchmark suite. This is the expensive step—each trace takes about an hour of simulation time, and there are 40 traces. ArchAgent v2 typically runs candidates in parallel across multiple machines.
The reward function is the geometric mean speedup across all traces. This is the same metric used by the DPC, ensuring that the framework is optimizing for the competition's actual goal. The reward is a single number, which makes the RL problem tractable.
The reward feeds into the RL loop, which updates a policy model that influences the next LLM prompt. If a candidate performed well, the framework might prompt the LLM to explore similar designs. If it performed poorly, the framework steers away from that direction.
The process repeats for hundreds of iterations. ArchAgent v2 required approximately 500 simulation runs to converge to a competitive policy. The final policy is the candidate with the highest geometric mean speedup, selected from all evaluated designs.
Key Takeaway: ArchAgent v2 is not a single algorithm—it's a design process that combines LLM creativity with RL-guided search. The LLM proposes, the simulator evaluates, and the RL loop learns.
ArchAgent v2 was entered into DPC-3 under standard competition rules: ChampSim simulator, 40 benchmark traces, and the geometric mean speedup metric. The framework ran on a cluster of machines, parallelizing simulation across traces. Each candidate required approximately 40 hours of simulation time (40 traces × 1 hour each), but parallelization reduced wall-clock time to a few hours per candidate.
ArchAgent v2 achieved a geometric mean speedup of approximately 1.08x over the baseline. The best human-designed prefetcher, Berti, achieved 1.12x. This places ArchAgent v2 in the top tier of DPC-3 submissions—competitive with, though not exceeding, the best human effort.
The gap between 1.08x and 1.12x is meaningful but not enormous. ArchAgent v2 was competitive with the vast majority of human designs, many of which were the result of years of expertise.
The framework didn't just replicate existing designs—it generated genuinely novel combinations:
The most striking result is time. ArchAgent v2 converged to its final policy in days. Human experts typically take months to develop and tune a competitive prefetcher. The framework didn't match the best human effort, but it came close in a fraction of the time—and it can be run again with different constraints or objectives.
Key Takeaway: ArchAgent v2 demonstrates that AI-driven design can produce competitive hardware optimizations without human expertise. It didn't win, but it proved the concept.
ArchAgent v2's primary strength is automation. It explored hundreds of design combinations that a human would never have time to try. The LLM's ability to combine known techniques in novel ways proved valuable—many of the best candidates were hybrids of existing prefetchers with new parameter settings.
The framework also benefited from its reward function. By optimizing geometric mean speedup directly, it avoided the trap of maximizing hit rate at the expense of performance.
The biggest limitation is computational cost. 500 simulation runs, each taking about an hour, requires substantial computing resources. Most academic labs couldn't afford this without dedicated clusters.
The framework also inherits ChampSim's limitations. If the simulator doesn't accurately model real hardware behavior, ArchAgent v2's optimizations may not translate to real systems. This is a general problem in computer architecture research, but it's particularly acute for AI-driven design, which has no intuition to compensate for simulator inaccuracies.
Berti, the DPC-3 winner, achieved 1.12x speedup. ArchAgent v2 achieved 1.08x. The gap likely comes down to Berti's sophisticated handling of memory bandwidth constraints—a subtle problem that ArchAgent v2 didn't fully explore. Human experts have deep intuitions about hardware constraints that are hard to encode in prompts or rewards.
The DPC-3 experiment suggests several lessons:
The ArchAgent v2 framework is general. The same LLM + RL loop could be applied to cache replacement policies, branch prediction, memory scheduling, or even instruction scheduling. Each problem has a simulator, a benchmark set, and a performance metric—the framework's requirements are met.
ArchAgent v2 wasn't alone. DPC-3 saw a significant increase in ML-based prefetchers, with many teams using neural networks or reinforcement learning in some capacity. The field is clearly moving toward ML-driven design.
The main challenges are computational cost, simulator accuracy, and the difficulty of encoding hardware constraints (area, power, timing) into the reward function. The opportunities are equally clear: AI agents can explore design spaces far faster than humans, and they don't have preconceptions about what "should" work.
ArchAgent v2 is open source, and the DPC provides a reproducible evaluation framework. This is crucial for the field—anyone can run the framework, verify the results, and build on them.
Key Takeaway: ArchAgent v2 is a proof of concept, not a final product. The framework's architecture is general, and its methodology can be applied to other hardware design problems.
ArchAgent v2 lost to Berti. AI is a tool, not a magic solution. The best results come from combining AI's exploration capability with human expertise in framing problems and interpreting results.
A prefetcher with 95% hit rate can be worse than one with 80% if it pollutes the cache or wastes bandwidth. ArchAgent v2 optimized for speedup, not hit rate, and this was the right choice.
This article discusses hardware prefetching, where the CPU predicts and fetches automatically. Software prefetching (explicit instructions in code) is a different technique with different tradeoffs. Both are useful, but they solve different problems.
The LLM in ArchAgent v2 doesn't understand computer architecture. It generates code based on patterns in its training data and the prompt it receives. The RL loop is what makes the framework work—it filters out bad proposals and amplifies good ones.
ArchAgent v2 took a fundamentally different approach to prefetcher design: instead of a human expert iterating manually, it used LLMs to propose designs and reinforcement learning to refine them. The result was a competitive prefetcher that achieved 1.08x speedup—not the best in DPC-3, but close, and achieved in days rather than months.
The implications extend far beyond prefetching. If AI agents can design competitive prefetchers, they can probably design competitive cache replacement policies, branch predictors, and memory schedulers. The hardware design process is ripe for automation.
ArchAgent v2 doesn't replace human experts. It augments them. The framework explores design spaces that humans don't have time to explore, and it does so in a reproducible, systematic way. As simulation tools get faster and LLMs get better, this approach will only become more powerful.
The next Data Prefetching Championship will be interesting. If AI agents continue to improve, they might not just be competitive—they might win.
ArchAgent v2 is a framework that uses large language models and reinforcement learning to automatically design hardware prefetchers. It generates candidate prefetcher policies, simulates them, evaluates their performance, and iteratively refines them.
The DPC is a competition series that evaluates prefetching algorithms under standardized conditions. Participants submit prefetchers that are run on a common simulator (ChampSim) with a fixed set of benchmark traces, and ranked by geometric mean speedup.
Prefetching hides memory latency by fetching data before it's needed. It can improve performance by 10-15% on memory-intensive workloads, making it one of the most impactful microarchitectural optimizations.
Traditional prefetcher design relies on human experts who manually create heuristics and tune parameters. ArchAgent v2 automates this process: an LLM proposes designs, a simulator evaluates them, and reinforcement learning guides the search.
No. ArchAgent v2 achieved a geometric mean speedup of approximately 1.08x, placing it competitively but behind the winner, Berti, which achieved 1.12x. It was one of the top-performing submissions, but not the winner.
The DPC uses ChampSim, a trace-based microarchitectural simulator that models a 4-core out-of-order processor with L1, L2, and L3 caches.
Yes. The framework's architecture is modular—the agent logic is separate from the simulator interface. It could be adapted for cache replacement, branch prediction, memory scheduling, or other optimization problems.
The main limitations are computational cost (hundreds of simulation runs) and dependence on simulator accuracy. It also didn't outperform the best human-designed prefetcher, suggesting that human expertise still has value.
A trace is a recording of memory access addresses from a real program execution. The DPC provides traces from SPEC CPU 2017 benchmarks and cloud workloads, which are used as inputs to the simulator.
Yes. The framework is publicly available, and the DPC provides a reproducible evaluation environment. Anyone can download the code, run the experiments, and verify the results.
Explore the ArchAgent v2 paper and code to see how AI is reshaping hardware design, and consider participating in the next Data Prefetching Championship to test your own ideas!