Training a large language model to be helpful, harmless, and honest usually means backpropagating through billions of parameters. That works—until it doesn't. When the model lives behind an API, when the reward is a human rating rather than a differentiable loss, or when GPU memory is the binding constraint, standard gradient descent stops being an option. Zeroth-order optimization offers a way out: estimate the gradient using nothing but function evaluations, then take a step. This article explains how that works, where it fits in the LLM alignment pipeline, and what it costs.
Preference alignment is the process of steering a pretrained language model toward outputs that humans actually prefer. The standard recipe—reinforcement learning from human feedback (RLHF)—has three stages: collect human comparisons, train a reward model on those comparisons, then optimize the policy against the reward model using an RL algorithm like PPO. Direct Preference Optimization (DPO) collapses the last two stages into a single supervised loss. Both approaches share one assumption: you can compute gradients of the objective with respect to the model's parameters.
That assumption fails in more situations than you might expect.
Zeroth-order optimization (ZOO) sidesteps all of this. Instead of computing an analytic gradient, it estimates one by probing the objective: perturb the parameters, measure how the loss changes, and use that finite difference as a descent direction. The method needs only forward passes.
Key Takeaway: Zeroth-order optimization replaces gradient computation with function evaluations, making alignment possible when gradients are unavailable, unreliable, or too expensive to store.
Zeroth-order optimization is a class of algorithms that minimize a function using only its values, never its derivatives. The term "zeroth-order" refers to the order of information used—zeroth derivative, i.e., the function value itself.
Suppose you want to minimize a scalar function $f(\theta)$ but can't compute $\nabla f(\theta)$. You can still ask: "If I nudge $\theta$ in direction $z$, does $f$ go up or down?" The answer gives you a noisy estimate of the directional derivative along $z$.
The simplest version uses finite differences. Pick a random direction $z$ (often drawn from a Gaussian), then compute:
$$\hat{g} = \frac{f(\theta + \epsilon z) - f(\theta)}{\epsilon} \cdot z$$
This is a one-sided estimate. A two-sided version evaluates $f(\theta + \epsilon z)$ and $f(\theta - \epsilon z)$, which reduces bias. Either way, you get a vector that points roughly downhill, and you can plug it into any first-order optimizer's update rule.
You might wonder why we don't just perturb each coordinate individually. That would require $d$ function evaluations per gradient estimate, where $d$ is the number of parameters—prohibitive for a model with billions of weights. Random directions collapse that cost: a single perturbation gives you a projection of the gradient onto a random vector, and in expectation, that projection points in the right direction.
The trade-off is variance. One random direction is a noisy guess; averaging over many directions reduces the noise.
| Property | First-Order (SGD, Adam) | Zeroth-Order |
|---|---|---|
| Information used | Gradients | Function values |
| Cost per step | 1 backward pass | 2+ forward passes |
| Memory | Activations + gradients | Forward activations only |
| Convergence rate (non-convex) | $O(1/\sqrt{T})$ | $O(1/\sqrt{T})$ with larger constants |
| Function evals per iteration | 1 | $O(d)$ for full gradient, $O(1)$ per random direction |
The memory savings come from skipping the backward pass entirely. You still need to run the forward pass twice (once for the perturbed parameters, once for the original), but you never store the intermediate activations that backprop requires.
The convergence theory for zeroth-order methods was developed largely in the 2010s. Ghadimi and Lan (2013) showed that stochastic zeroth-order methods achieve the same $O(1/\sqrt{T})$ convergence rate as stochastic first-order methods for non-convex smooth functions, albeit with constants that depend on the dimension. Nesterov and Spokoiny (2017) extended this to convex settings and provided bounds on the number of function evaluations required to reach a given accuracy.
The dimension dependence is the central caveat: to estimate a full gradient to a given accuracy, you need roughly $O(d)$ function evaluations. For a model with $d = 10^9$ parameters, that's a lot of forward passes. The practical trick—used by MeZO and similar methods—is to never estimate the full gradient. Instead, take a single random direction per step and accept the variance. The optimizer still makes progress, just noisily.
Key Takeaway: Zeroth-order methods trade gradient precision for accessibility. They converge at the same asymptotic rate as first-order methods but require more function evaluations and tolerate higher variance.
Applying zeroth-order optimization to preference alignment means treating the alignment objective as a black box. You can evaluate it—by scoring outputs with a reward model, comparing against human preferences, or computing a task metric—but you can't differentiate through it.
This loop requires only forward passes. No backpropagation, no stored activations, no access to the reward model's internals.
Preference alignment often involves objectives that are inherently non-differentiable. Human raters give thumbs up or thumbs down. Safety classifiers output binary labels. Task metrics like exact match or F1 are computed after decoding, which involves discrete token selection. In all these cases, the reward signal is a scalar you can observe but not differentiate.
Zeroth-order methods handle this natively. The reward function can be anything—a Python function, a human in the loop, a proprietary API—as long as you can evaluate it.
Because zeroth-order optimization never touches the model's internals, it works with any model you can query. That includes:
This model-agnostic property is what makes the paradigm attractive for production systems, where the policy and reward may come from different vendors or run on different hardware.
Key Takeaway: Zeroth-order alignment treats the reward as a black box, enabling optimization when the reward is non-differentiable, proprietary, or based on human feedback.
MeZO, introduced by Malladi et al. in "Fine-Tuning Language Models with Just Forward Passes" (NeurIPS 2023), is the most widely cited instantiation of zeroth-order optimization for LLMs. It adapts the classic ZOO idea to the scale and structure of transformer models.
MeZO maintains a single random seed per step. At each iteration:
The key engineering trick is that $z$ is never stored. It's regenerated from the seed whenever needed. That keeps memory overhead near zero.
Standard fine-tuning with AdamW stores:
MeZO eliminates gradients, optimizer states, and stored activations. The paper reports up to 50% memory reduction compared to standard fine-tuning for large language models. For a model that barely fits on a single GPU, that difference is the gap between training and not training.
MeZO isn't just a memory hack. On several NLP tasks—including sentiment classification, natural language inference, and summarization—it achieves performance comparable to AdamW while using only forward passes. The trade-off is speed: each step requires two forward passes instead of one forward and one backward, and the gradient estimates are noisier, so more steps are often needed.
Key Takeaway: MeZO makes zeroth-order fine-tuning practical by regenerating random perturbations from a seed, eliminating the need to store them, and cutting memory usage by up to half.
Parameter-efficient fine-tuning (PEFT) methods like LoRA and prompt tuning reduce the number of trainable parameters. That's a natural fit for zeroth-order optimization, which suffers when the parameter dimension is large.
LoRA inserts low-rank matrices into the model's attention layers. Instead of training all $d$ parameters, you train a much smaller set—often 0.1% to 1% of the original count. Zeroth-order optimization over this reduced space is more tractable because the dimension $d'$ is smaller, which directly reduces the variance of gradient estimates and the number of function evaluations needed.
The combination works like this: freeze the base model, apply LoRA adapters, then run MeZO-style updates on the adapter weights only. You get the memory benefits of both approaches—no backprop through the base model, and far fewer parameters to perturb.
Prompt tuning optimizes a set of continuous embeddings prepended to the input. The base model stays frozen. This is already a low-dimensional optimization problem, and it's a natural fit for zeroth-order methods because:
For API-based models where you can't touch the weights at all, prompt tuning is often the only option. Zeroth-order optimization makes it work even when the reward is non-differentiable.
Key Takeaway: Combining zeroth-order optimization with PEFT reduces the search space, which lowers variance and makes black-box alignment practical even for very large models.
Use zeroth-order optimization when:
Stick with first-order methods when:
Key Takeaway: Zeroth-order methods win on accessibility and memory, lose on speed and variance. Choose them when gradients are unavailable or memory is tight, not when you have a clean differentiable path.
Suppose you want to align GPT-4 to your company's tone and policies. You can't fine-tune it directly, but you can optimize a prompt. Treat the prompt embeddings as parameters, define a reward function that scores outputs against your criteria (e.g., a classifier or a set of rules), and run zeroth-order optimization over the prompt. Each step: perturb the prompt, generate outputs, score them, update.
For open-weight models, you can optimize soft prompts—continuous embeddings that steer the model's behavior—without touching the base weights. Zeroth-order optimization is a natural fit because the prompt is a small parameter set and the reward may be non-differentiable.
If a vendor provides a reward API but not the model weights, you can't backprop through it. Zeroth-order optimization lets you query the reward and update your policy anyway.
The MeZO paper demonstrated fine-tuning large models for sentiment classification using only forward passes. The setup: a dataset of movie reviews labeled positive or negative, a pretrained LLM, and a cross-entropy loss. MeZO matched AdamW's accuracy while using significantly less memory.
Key Takeaway: Zeroth-order alignment is most useful when you're optimizing a small parameter set—prompts, adapters, or a policy head—against a reward you can only query.
For non-convex smooth functions, stochastic zeroth-order methods converge at $O(1/\sqrt{T})$, matching first-order methods asymptotically. The constants are larger, and the number of function evaluations per iteration scales with the dimension.
Estimating a full gradient to accuracy $\delta$ requires $O(d/\delta^2)$ function evaluations. That's the bad news. The good news is that in practice, you don't need a full gradient—a single random direction per step is enough to make progress, and the dimension dependence shows up as variance rather than a hard barrier.
MeZO and related methods have shown competitive performance on:
In preference alignment specifically, zeroth-order methods have demonstrated win rates against baselines that are within a few percentage points of gradient-based methods like PPO and DPO, though they typically require more training steps.
The gap narrows when:
Key Takeaway: Zeroth-order methods converge at the same asymptotic rate as first-order methods, but the constants and function-evaluation costs are higher. PEFT and variance reduction close much of the practical gap.
Misconception: Zeroth-order is always less efficient. False. It's less sample-efficient per step, but it can be more memory-efficient and is the only option when gradients are unavailable. For small parameter sets, the overhead is modest.
Misconception: It can't handle high-dimensional problems. It can, but variance grows with dimension. PEFT and single-direction updates make high-dimensional problems tractable in practice.
Misconception: It only works for convex problems. The theory covers non-convex smooth functions, which is what neural network training involves.
Misconception: It requires access to model internals. No. Zeroth-order methods need only the ability to evaluate the objective. That's the whole point.
Misconception: It's not theoretically grounded. Ghadimi and Lan (2013) and Nesterov and Spokoiny (2017) provide rigorous convergence guarantees. The theory is well-established.
Key Takeaway: Zeroth-order optimization is not a hack. It's a principled method with convergence guarantees, and its limitations are well-understood.
Research is moving in several directions:
The most likely near-term deployment is in constrained environments: fine-tuning on a single GPU, aligning API-based models, or optimizing against non-differentiable rewards. As those use cases grow, so will the toolkit.
Key Takeaway: Zeroth-order alignment is not a replacement for gradient-based methods. It's a complement for situations where gradients are unavailable or impractical.
What is zeroth-order optimization in the context of LLMs? It's a method for optimizing a model's parameters using only function evaluations—no gradients. You perturb the parameters, measure the change in the objective, and use that to estimate a descent direction.
How does zeroth-order optimization differ from first-order methods like gradient descent? First-order methods use analytic gradients computed via backpropagation. Zeroth-order methods estimate gradients from function values, which requires more evaluations but no backward pass.
Can zeroth-order optimization be used for RLHF? Yes. You can optimize the policy against a reward model using zeroth-order updates, which is useful when the reward model is non-differentiable or proprietary.
What are the advantages of zeroth-order methods for LLM alignment? Memory efficiency, compatibility with API-based models, and the ability to handle non-differentiable rewards.
What are the disadvantages? Higher variance, slower convergence, and more function evaluations per step.
Is zeroth-order optimization scalable to large language models? Yes, especially when combined with PEFT. MeZO demonstrated fine-tuning of models with billions of parameters using only forward passes.
What is MeZO and how does it relate to zeroth-order optimization? MeZO is a specific zeroth-order optimizer designed for LLMs. It uses two forward passes per step and regenerates random perturbations from a seed to save memory.
Can zeroth-order methods be combined with existing alignment techniques like DPO? They can be used as an alternative to DPO when gradients aren't available, or combined with PEFT to reduce the search space.
What are some practical applications of zeroth-order alignment? Fine-tuning API models, optimizing soft prompts, aligning with proprietary reward models, and any task where the reward is non-differentiable.
How does the performance of zeroth-order alignment compare to first-order methods? It's competitive on many tasks, especially with PEFT, but typically requires more iterations and may have higher variance.
Ready to align your LLM without gradients? Explore the MeZO repository and start experimenting with zeroth-order optimization today.