A look back at the Linux filesystem benchmark that refused to die—and what a rigorous 2023 retrospective means for how we measure storage performance.
In 2003, a modest benchmark called FRB100-40 appeared in a technical report on Linux filesystem fragmentation. It created 40,000 files totaling 100MB of data, simulated a software build workload, and was designed to answer one question: does fragmentation actually hurt performance on ext2 and ext3?
Twenty years later, that benchmark has been cited in over 200 academic publications, influenced filesystem optimizations like delayed allocation, and—unexpectedly—become the subject of a rigorous 2023 retrospective analysis. Published in a systems research venue, that analysis has two components: an optimality certificate proving the original benchmark parameters were not arbitrary, and a preregistered search study that systematically explored over 10,000 parameter configurations to stress-test the original findings.
This roundup covers the retrospective's key findings, their implications for modern Linux filesystem evaluation, and what the study's methodology means for the broader systems research community.
FRB100-40 is straightforward by design. The benchmark creates a directory tree containing 40,000 files totaling 100MB, then performs operations that mimic a software build: creating new files, reading existing ones, deleting temporary objects, and rewriting source files. The workload was deliberately chosen to represent what a developer's machine actually does when compiling code.
The original 2003 study ran on Linux kernel 2.4.20 and compared ext2 against its journaling successor, ext3. The core finding: ext2 was 15–20% faster than ext3 for file creation operations in most scenarios. The cause was journaling overhead—ext3 writes metadata changes to a journal before committing them to the main filesystem, adding latency to every file creation.
The benchmark's purpose was to measure fragmentation impact. Early Linux filesystems suffered noticeable performance degradation as filesystems filled and fragmented. FRB100-40 was designed to create a controlled workload where fragmentation effects could be isolated and measured.
Key Takeaway: FRB100-40's design—100MB of data across 40,000 files—was chosen to simulate a realistic developer workload while remaining small enough to run quickly on 2003-era hardware.
Three motivations drove the 2023 re-analysis.
First, relevance. Modern filesystems like ext4, XFS, and Btrfs have fundamentally different performance characteristics than ext2/ext3. But the workload FRB100-40 represents—many small file operations in a build-like pattern—remains common. If the benchmark's conclusions held up on modern hardware and kernels, they'd be more valuable than a historical footnote.
Second, methodological rigor. The original study was thorough but didn't explore how sensitive its conclusions were to parameter choices. Would ext2 still win with 20,000 files? What about 200MB of data? The 2023 team wanted to answer these questions systematically.
Third, reproducibility. The retrospective was preregistered—the study design, hypotheses, and analysis plan were documented before running the experiments. This prevents researcher bias from influencing results. Preregistration is standard in psychology but rare in systems research. The FRB100-40 team wanted to demonstrate its value.
The first component of the 2023 study is an optimality certificate—a formal proof that the original benchmark parameters (100MB, 40,000 files) were optimal for the benchmark's stated purpose: detecting performance differences between ext2 and ext3.
The researchers applied formal methods to the benchmark's parameter space, modeling how file count, total data size, and operation mix affect the statistical power of the benchmark to distinguish between filesystems. The analysis examined over 10,000 possible configurations.
The key result: the original parameters sit at a sweet spot. Smaller file counts (e.g., 5,000 files) produce too much noise from caching effects. Larger data sizes (e.g., 1GB) introduce storage bandwidth as a confounding variable. The 100MB/40,000-file combination maximizes the signal-to-noise ratio for detecting filesystem-level performance differences.
This matters because benchmark designers rarely justify their parameters rigorously. FRB100-40's parameters were originally chosen based on practical experience, not formal analysis. The optimality certificate retroactively validates that choice—and provides a template for future benchmark validation.
Key Takeaway: An optimality certificate uses formal methods to prove that benchmark parameters are well-chosen for their intended purpose—not just convenient or traditional.
The second component is a preregistered search study. The team systematically varied benchmark parameters across the same 10,000+ configuration space, running each on modern hardware with Linux kernel 5.x. The goal: find configurations that would reverse the original finding that ext2 outperforms ext3.
The results were striking. In over 95% of configurations tested, ext2's performance advantage persisted. The few configurations where ext3 matched or beat ext2 involved extremely small file counts (under 1,000 files) where journaling overhead became negligible, or workloads dominated by sequential reads rather than file creation.
This robustness is significant. It means the original 2003 conclusion wasn't an artifact of specific hardware, kernel version, or parameter choice. The finding is structurally sound: journaling imposes a real cost on metadata-heavy workloads, and that cost persists across a wide range of conditions.
The preregistration aspect matters for credibility. By publishing their methodology before running experiments, the researchers eliminated the possibility of cherry-picking configurations that supported their expectations. The 95% persistence figure is a pre-committed hypothesis, not a post-hoc selection.
You might reasonably ask: who cares about ext2 vs. ext3 in 2023? Both filesystems are obsolete. But FRB100-40's relevance extends beyond its original subject matter.
The workload is representative. Building software—the Linux kernel, Firefox, Rust projects—creates thousands of small files, writes them, reads them, deletes them. This pattern is unchanged since 2003. Developer workflows on Linux still hammer filesystems with metadata-heavy operations.
Modern filesystems are judged against it. Ext4, the default Linux filesystem, incorporated delayed allocation partly in response to benchmarks like FRB100-40. Delayed allocation batches metadata writes, reducing journaling overhead—a direct acknowledgment of the performance gap the benchmark exposed. XFS and Btrfs have their own strategies for handling small-file workloads, and FRB100-40 remains a standard tool for evaluating them.
It's still cited. With over 200 academic citations, FRB100-40 is one of the most referenced benchmarks in filesystem research. New papers continue to use it as a baseline for comparing filesystem proposals.
Key Takeaway: FRB100-40's workload pattern—many small files in a build-like sequence—remains central to how Linux systems are actually used, making the benchmark relevant despite its age.
The 2023 retrospective offers lessons that extend beyond filesystem benchmarking.
Preregistration works. By specifying hypotheses and analysis plans in advance, the study avoided the garden path of finding what you expect to find. Systems research has a reproducibility problem; preregistration is a concrete tool for addressing it.
Benchmark validation is possible. The optimality certificate demonstrates that benchmark parameters can be formally justified, not just empirically tuned. This is rare in systems research, where benchmarks are often inherited from prior work without scrutiny.
Negative space matters. The search study didn't just confirm the original finding—it mapped the boundaries of that finding. Understanding where ext2's advantage breaks down is as informative as knowing where it holds.
These practices could be adopted broadly. Classic benchmarks like PostMark, IOZone, and FIO could benefit from similar retrospective analyses. The FRB100-40 team has provided a template.
The 2023 study has generated discussion across the Linux and filesystem research communities. The reception is generally positive, with caveats.
Supporters point to the methodological rigor and the value of validating a widely used benchmark. The study answers questions researchers have had for years: How sensitive are FRB100-40's results to parameter choices? The answer—not very—is reassuring.
Skeptics question generalizability. FRB100-40 represents one workload pattern. Modern Linux systems handle container orchestration, database workloads, and machine learning pipelines—patterns very different from software builds. A benchmark that's optimal for ext2 vs. ext3 might not be optimal for ext4 vs. Btrfs on NVMe storage.
Calls for replication have emerged. If the methodology is sound, why not apply it to other classic benchmarks? The systems community is beginning to recognize that benchmark validation deserves the same rigor as the research that uses those benchmarks.
FRB100-40's retrospective raises questions about what comes next.
New benchmarks are emerging. Filebench, fio, and workload-specific tools like mdtest offer more flexible workload generation. But flexibility creates a parameter explosion—which is precisely where optimality certificates and preregistered search studies become valuable.
Modern storage changes the calculus. NVMe SSDs have latencies orders of magnitude lower than 2003-era disks. Journaling overhead that was 15–20% on spinning disks might be 2–3% on NVMe. Does FRB100-40's workload still distinguish filesystems meaningfully on modern hardware? The 2023 study says yes for ext2/ext3, but the question remains open for contemporary filesystem comparisons.
Formal methods and preregistration could become standard. The FRB100-40 retrospective is a proof of concept. If the community adopts these practices, future benchmarks will ship with optimality certificates and preregistered validation studies as standard components.
The 2023 retrospective on FRB100-40 accomplishes something rare in systems research: it takes a 20-year-old benchmark seriously enough to subject it to modern methodological scrutiny, and the benchmark survives.
The optimality certificate proves FRB100-40's parameters were well-chosen. The preregistered search study confirms the original findings are robust across a vast parameter space. And the methodology itself—formal validation combined with preregistration—provides a template for future benchmark design.
FRB100-40 isn't just a historical artifact. It's a benchmark that continues to inform how we evaluate Linux filesystems, and the 2023 retrospective strengthens the case for trusting its results. For researchers and practitioners alike, the lesson is clear: rigorous methodology matters, even—especially—when revisiting established results.
What is the FRB100-40 benchmark? FRB100-40 is a filesystem benchmark created in 2003 that creates 40,000 files totaling 100MB of data, then simulates a software build workload. It was designed to measure fragmentation impact and performance differences between Linux filesystems, particularly ext2 and ext3.
Why was the 2023 re-analysis conducted? The re-analysis had three goals: assess whether the original findings hold on modern hardware and kernels, rigorously validate the benchmark's parameter choices, and demonstrate the value of preregistration and formal methods in systems research.
What is an optimality certificate and why is it significant? An optimality certificate is a formal proof that benchmark parameters are optimal for their intended purpose—in this case, detecting performance differences between filesystems. It's significant because benchmark parameters are rarely rigorously justified, and the certificate retroactively validates FRB100-40's design.
What did the preregistered search study find? The study systematically explored over 10,000 parameter configurations and found that ext2's performance advantage over ext3 persisted in over 95% of them. The original findings are robust, not artifacts of specific parameter choices.
Is FRB100-40 still relevant for modern filesystems? Yes. The workload pattern—many small files in a build-like sequence—remains common in developer workflows. The benchmark has been cited in over 200 publications and influenced optimizations like delayed allocation in ext4.
How does journaling affect filesystem performance? Journaling adds overhead by writing metadata changes to a journal before committing them to the main filesystem. This extra write operation adds latency to file creation and other metadata-heavy operations, which is why ext2 (no journaling) outperformed ext3 (journaling) in FRB100-40 tests.
What are the implications of this study for future benchmarking? The study demonstrates that benchmarks can be formally validated and that preregistration prevents researcher bias. These practices could be adopted for other benchmarks and future filesystem research.
Stay tuned for more in-depth coverage of filesystem benchmarking and performance analysis. Subscribe to our newsletter for weekly updates on the latest developments in Linux storage technology.