AI coding agents have evolved from research curiosities into daily tools for working developers. GitHub Copilot, Cursor, and Claude Code now write entire functions, debug production issues, and refactor codebases with minimal human supervision. The pitch is simple: describe what you want, and the agent writes it. Describe what's broken, and the agent fixes it.
The question that truly matters, however, isn't whether these agents can produce code—it's whether their code is actually good. For the past few years, the dominant answer to that question has been: "Does it pass the tests?"
That answer is incomplete. A benchmark released in 2025 called SWE-Gate makes this case with uncomfortable clarity. SWE-Gate evaluates agents not just on whether their solutions pass functional tests, but on whether those solutions are readable, maintainable, efficient, and secure. The results are sobering: agents that nail functional tests routinely produce code that human reviewers wouldn't accept in a pull request.
This article compares the two evaluation philosophies head-to-head: functional test passing versus holistic code quality. We'll examine what each approach measures, where they diverge, and why the gap between them matters for anyone who relies on AI to write production software.
SWE-Gate is an evaluation benchmark designed to test software engineering agents on tasks that mirror real development work. Introduced in early 2025 by researchers from multiple academic and industrial labs, it's available as an open-source project on GitHub.
The benchmark's name is a deliberate play on "Watergate"—the authors frame it as exposing a scandal in how we evaluate AI coding agents. The scandal, as they see it, is that the industry has conflated "passing tests" with "solving the problem correctly."
SWE-Gate tasks come from 500 real GitHub issues across 50 popular open-source projects. Each task requires an agent to make changes to an existing codebase—often spanning multiple files—to resolve an issue. The evaluation operates on two layers:
Functional tests: Hidden tests check whether the agent's solution produces the correct behavior. These are similar to the tests used in SWE-bench and other functional benchmarks.
Code quality assessment: This layer goes beyond correctness and includes: - Static analysis: Automated checks for security vulnerabilities, code smells, and adherence to language best practices. - Human review: Experienced developers rate each solution on dimensions like readability, maintainability, efficiency, and integration with the existing codebase.
Human evaluators spend an average of 15 minutes per solution to complete this review. That investment reflects the benchmark's core premise: evaluating code quality requires judgment that automated test runners simply can't provide.
The result is a two-dimensional score for each agent: a functional pass rate and a code quality score (on a 1–5 scale). As we'll see, these two numbers tell very different stories.
Key takeaway: SWE-Gate isn't just another benchmark. It's a deliberate challenge to the assumption that functional correctness equals solution quality. By adding code quality metrics to the evaluation pipeline, it exposes weaknesses that test-based benchmarks miss entirely.
Functional tests check one thing: does the code produce the expected output for a given input? They're objective, automatable, and fast. A test suite gives you a clear pass/fail signal, and it's easy to compare agents by their pass rates.
These strengths explain why functional tests dominate AI coding evaluation. SWE-bench, the most widely cited benchmark in this space, measures agents on their ability to resolve GitHub issues as judged by hidden tests. The metric is simple: what percentage of tests does the agent's patch pass?
Here's the problem: tests only verify behavior under the specific conditions they cover. They don't verify that the solution is good in any broader sense. A test suite is a sample of expected behaviors, not a complete specification of what constitutes a correct, maintainable fix.
Consider these scenarios that functional tests routinely miss:
Security vulnerabilities: An agent asked to fix a bug in a Python web application might pass all provided unit tests while introducing unsafe deserialization—a vulnerability that no test catches but that exposes the system to remote code execution.
Inefficient algorithms: An agent implements a sorting function that passes functional tests but has O(n²) complexity in a project that explicitly requires O(n log n) for large datasets. The tests pass. The code ships. The application crawls.
Poor integration: An agent modifies a function to satisfy a test but uses global variables and breaks the module pattern the rest of the codebase follows. The change works in isolation but creates maintainability debt for every future developer.
Style violations: An agent's code passes all tests but ignores the project's naming conventions, indentation rules, and documentation standards. It gets rejected in code review—or worse, merged and left as a permanent eyesore.
These aren't hypothetical edge cases. SWE-Gate's data shows they're the norm.
The deeper issue is overfitting. Agents that are trained and evaluated primarily on test passing learn to optimize for that signal. They generate code that satisfies the specific assertions in the test suite while ignoring broader requirements.
This produces what SWE-Gate's authors call "test-passing but semantically incorrect" solutions. The code works for the tested cases but is wrong in ways that matter—wrong for production loads, wrong for security, wrong for maintainability.
The stats from SWE-Gate make the point starkly:
In other words, the overlap between "passes tests" and "is good code" is much smaller than the industry has assumed.
Key takeaway: Functional tests verify behavior under sampled conditions. Code quality encompasses readability, maintainability, efficiency, security, and integration. Passing the former says nothing about the latter. SWE-Gate quantifies this gap: high test pass rates coexist with low quality scores in most agents.
SWE-bench, introduced at ICLR 2024, set the standard for evaluating AI agents on real-world coding tasks. It pulls issues from popular open-source repositories and checks whether agent-generated patches pass the project's test suite. SWE-bench was a major step forward because it moved beyond toy problems to actual codebases.
But SWE-bench and SWE-Gate differ in fundamental ways:
| Dimension | SWE-bench | SWE-Gate |
|---|---|---|
| Primary metric | Functional test pass rate | Test pass rate + code quality score |
| Quality assessment | None (binary pass/fail) | Static analysis + human expert review |
| Task complexity | Single-issue resolution | Multi-file changes, integration with existing architecture |
| What it measures | Correctness under tested conditions | Correctness + production readiness |
| Human involvement | Minimal (test setup only) | Significant (15 min per solution for expert review) |
SWE-bench's strengths: It's objective, reproducible, and scalable. Test results don't require human judgment, so anyone can run the benchmark and compare results. This objectivity made SWE-bench the de facto standard for comparing coding agents.
SWE-bench's weaknesses: It optimizes for exactly what it measures—test passing. Agents that perform well on SWE-bench may do so by overfitting to test patterns. The benchmark provides no signal about code quality, security, or maintainability. A patch that passes tests but introduces a critical vulnerability scores identically to a patch that passes tests and follows best practices.
SWE-Gate's strengths: It provides a two-dimensional view of agent performance. By including human review and static analysis, it captures dimensions of code quality that automated testing misses. This makes it a more realistic predictor of how agent-generated code will fare in actual production environments.
SWE-Gate's weaknesses: Human review introduces subjectivity and cost. Evaluators may disagree on quality ratings, and the benchmark's results are harder to reproduce than pure test-based benchmarks. The 15-minute-per-solution review time also limits scalability.
The authors of SWE-Gate don't argue that functional tests are useless. Rather, they argue that functional tests are necessary but insufficient. A benchmark that only measures test passing creates a perverse incentive structure: agents optimize for the metric, and the metric doesn't capture what production software actually requires.
SWE-Gate's two-layer evaluation creates a more complete picture. An agent that scores 85% on functional tests but 3.1/5 on code quality (the best performance in the benchmark) is clearly not ready for production use. That information is invisible in a test-only benchmark.
Key takeaway: SWE-bench asks "Can the agent produce code that passes tests?" SWE-Gate asks "Can the agent produce code that a senior engineer would approve?" These are different questions, and the answers diverge significantly.
SWE-Gate evaluated several state-of-the-art agents, including GPT-4, Claude, SWE-agent, and Codex. The results reveal a consistent pattern: high functional pass rates, mediocre quality scores.
The best-performing agent achieved an 85% functional pass rate but scored only 3.1 out of 5 on code quality. That's a solid C+ from human reviewers—not the kind of grade that inspires confidence in production deployment.
More telling is the gap between the two metrics. When SWE-Gate's authors separated agents by their training focus, they found that agents specifically optimized for test generation (like SWE-agent) showed a 20% drop in code quality scores compared to general-purpose agents. These agents were better at passing tests and worse at writing good code—the overfitting effect made visible.
The security regression: One agent was asked to fix a bug in a web application's file upload feature. It passed all functional tests but used pickle for deserializing user-supplied data—a well-known security vulnerability. The tests didn't cover this case, so the agent's solution looked correct. Human reviewers flagged it immediately.
The inefficient implementation: Another agent was tasked with optimizing a data processing pipeline. Its solution passed all tests but introduced an O(n²) algorithm where the codebase required O(n log n). The tests used small datasets, so the inefficiency was invisible. In production, the code would have been catastrophically slow.
The maintainability failure: An agent fixed a bug in a library function but did so by adding a global state variable that broke the library's thread-safety guarantees. The tests passed. The code was fundamentally incompatible with the project's architecture.
These aren't isolated incidents. SWE-Gate's data shows that only 12% of agent solutions were considered production-ready by human reviewers. The gap between functional correctness and production quality isn't a minor discrepancy—it's a chasm.
SWE-Gate's authors point to training data as a key factor. Agents trained primarily on test-generation tasks learn to reverse-engineer solutions that satisfy test assertions. They become skilled at pattern-matching test expectations rather than understanding the underlying problem and designing robust solutions.
This isn't a failure of the agents themselves—it's a failure of the evaluation metrics that guided their training. When test passing is the only reward signal, test passing is what agents optimize for.
Key takeaway: Across all agents evaluated, functional test passing and code quality scores diverged significantly. The best agent scored 85% on tests but only 3.1/5 on quality. Agents trained specifically for test generation showed the largest quality deficits.
Holistic evaluation: SWE-Gate measures what matters for production software—not just correctness but quality, security, and maintainability. This makes its results more actionable for teams deciding whether to deploy AI-generated code.
Real-world tasks: The benchmark uses actual GitHub issues from popular projects. Tasks require multi-file edits, integration with existing libraries, and understanding of architectural patterns. This is representative of real development work, not synthetic toy problems.
Open-source: SWE-Gate is publicly available on GitHub. Researchers can reproduce results, extend the benchmark, and contribute new tasks. This transparency is essential for building trust in evaluation metrics.
Human-in-the-loop: The human review component adds a dimension that automated metrics can't capture. Experienced developers can identify issues—like security vulnerabilities or architectural mismatches—that no static analysis tool catches.
Subjectivity: Human quality ratings are inherently subjective. Different reviewers may grade the same code differently. SWE-Gate attempts to mitigate this with standardized rubrics, but the results still carry evaluator bias.
Cost and scalability: The 15-minute-per-solution human review limits SWE-Gate's scalability. Running the benchmark on many agents or many tasks requires significant human labor.
Limited scope: With 500 tasks from 50 projects, SWE-Gate covers less ground than larger automated benchmarks. The tasks skew toward Python and popular open-source projects, which may not represent all software engineering domains.
Potential bias: The choice of projects, the framing of tasks, and the evaluation criteria all reflect the authors' priorities. A benchmark that emphasizes code quality may undervalue agents that are optimized for rapid prototyping or exploratory coding, where test passing might matter more than perfect architecture.
SWE-Gate correctly identifies that test passing is an incomplete metric. But it also has blind spots. It doesn't measure factors like: - Developer experience: How much human intervention does an agent require to produce good code? - Iteration speed: How quickly can an agent respond to feedback and improve its solutions? - Generalization: How well do agents perform on tasks outside the benchmark's 50 projects?
These dimensions matter for real-world adoption but fall outside SWE-Gate's scope.
Key takeaway: SWE-Gate is a significant improvement over test-only benchmarks, but it's not a complete solution. Human review adds valuable insight at the cost of scalability and objectivity. The benchmark should be viewed as one tool among several for evaluating AI coding agents.
SWE-Gate's data makes a compelling case. Agents routinely pass functional tests while producing code that fails basic quality standards. The median quality score of 2.5/5 for test-passing solutions means most agent-generated code would be rejected in a serious code review.
The gap between test passing and code quality isn't small—it's the difference between "works in the demo" and "works in production." That difference carries real costs: security breaches, performance failures, and maintainability debt that compounds over time.
When evaluation metrics reward test passing exclusively, agents learn to game the metric. They produce code that satisfies assertions while ignoring everything else. This isn't a hypothetical concern—SWE-Gate shows it happening with state-of-the-art agents.
The broader danger is that the entire AI coding ecosystem—model training, agent design, benchmark development—optimizes for the wrong target. Teams choose agents based on benchmark scores that don't predict real-world performance. They deploy AI-generated code that passes tests but introduces vulnerabilities and architectural debt.
For researchers: Develop benchmarks like SWE-Gate that include quality metrics alongside functional correctness. Invest in automated quality assessment tools that can scale beyond human review. Study the relationship between training data and code quality to understand why overfitting occurs.
For developers: Treat benchmark test pass rates with skepticism. Evaluate AI coding agents on their actual output, not just their test scores. Establish code review processes that catch quality issues in AI-generated code before it reaches production.
For tool builders: Design agents that optimize for code quality, not just test passing. Incorporate feedback loops where agents learn from human code review. Build evaluation into development workflows so quality issues are caught early.
SWE-Gate represents a step toward more realistic evaluation, but it's not the final word. The ideal benchmark would combine: - Objective functional testing at scale - Automated quality analysis for security, efficiency, and best practices - Human review for judgment-dependent dimensions - Real-world tasks that reflect production complexity
Such a benchmark would give developers the information they need to make informed decisions about AI coding tools. Until then, SWE-Gate provides the strongest evidence yet that test passing is necessary but nowhere near sufficient.
Key takeaway: The verdict is clear: code quality must be part of the evaluation equation. Functional tests measure whether code works under sampled conditions. Production software requires code that is secure, maintainable, efficient, and well-integrated. SWE-Gate shows these dimensions are largely orthogonal—and that ignoring quality has real consequences.
SWE-Gate is a benchmark introduced in 2025 that evaluates software engineering agents on both functional test passing and code quality. It includes 500 tasks from real GitHub issues across 50 open-source projects, with quality assessed through static analysis and human expert review.
Functional tests only verify behavior under the specific conditions they cover. They don't catch security vulnerabilities, inefficient algorithms, poor integration with existing codebases, or violations of project conventions. SWE-Gate shows that agents routinely pass tests while producing code that human reviewers rate poorly.
SWE-Gate uses two methods: static analysis tools that check for security issues and code smells, and human review where experienced developers rate solutions on a 1–5 scale across dimensions like readability, maintainability, efficiency, and architecture. Each human review takes about 15 minutes per solution.
Tasks come from real GitHub issues in 50 popular open-source projects. They require agents to understand existing codebases, make multi-file changes, handle edge cases, and integrate solutions with the project's architecture and libraries.
SWE-Gate has evaluated state-of-the-art agents including GPT-4, Claude, SWE-agent, and Codex. The best-performing agent achieved an 85% functional pass rate but scored only 3.1/5 on code quality.
The main findings are: (1) agents that pass functional tests have a median quality score of only 2.5/5; (2) only 12% of agent solutions were considered production-ready despite 80% passing tests; (3) agents trained specifically on test generation show a 20% drop in quality scores, indicating overfitting.
Researchers can use SWE-Gate to evaluate their own agents, study the relationship between functional correctness and code quality, and investigate why agents overfit to test passing. The open-source repository allows for reproducing results and extending the benchmark.
Yes, SWE-Gate is open-source and available on GitHub. The repository includes the task data, evaluation scripts, and documentation for running the benchmark on new agents.
SWE-Gate has several limitations: human review introduces subjectivity and cost, the benchmark covers a limited set of projects (mostly Python), and it doesn't measure factors like developer experience, iteration speed, or generalization beyond its task set.
SWE-bench measures only functional test passing on real GitHub issues. SWE-Gate uses similar tasks but adds code quality assessment through static analysis and human review. SWE-Gate can be seen as a more holistic version of SWE-bench that addresses its predecessor's blind spots.
The software engineering community has spent years optimizing AI coding agents to pass functional tests. SWE-Gate demonstrates that this optimization has produced agents that can satisfy test suites while writing code that no competent engineer would approve.
The gap between test passing and code quality isn't an artifact of the benchmark's design—it's a real phenomenon with real consequences. Code that passes tests but contains security vulnerabilities, uses inefficient algorithms, or violates architectural patterns isn't production-ready. It's a liability.
The path forward requires a shift in how we evaluate AI coding agents. Functional tests remain essential, but they can't be the only metric. Code quality must be part of the equation—through human review, automated analysis, or ideally both.
The tools we build will reflect the metrics we optimize for. If we continue to reward test passing alone, we'll get agents that are excellent at passing tests and poor at writing good software. If we expand our metrics to include quality, we'll get agents that produce code worth shipping.
The choice is clear. The question is whether the AI community will make it.
Dive deeper into SWE-Gate by exploring the open-source repository and reading the full paper. Share your thoughts on how we should evaluate AI coding agents—join the conversation in the comments below!