The problem
Most tuning of prompts and language-model pipelines is done by feel. Someone edits a prompt, reads a handful of outputs, and decides whether it seems better. That approach has three weaknesses.
- It is slow. Every change requires a person to read and judge results.
- It does not scale. A pipeline with several steps has many prompts that interact, and adjusting them one at a time by hand rarely finds a good combination.
- It optimizes the wrong thing. Judging from a few examples quietly tunes the system toward whichever cases the person happened to look at.
What we built
An optimization suite that treats prompts and pipelines as things that can be improved systematically. Instead of adjusting by intuition, it tunes the system against measurable objectives. It is built on TextGrad, with evaluation at its core, in Python.
How it works
The idea borrowed from machine learning
Neural networks are trained using gradients: a signal that says, for each parameter, which direction to move it to reduce error. TextGrad applies an analogous idea to text. When a pipeline produces an output, feedback on that output, often generated by a language model acting as a critic, is turned into specific suggestions for how to change the prompts and components that produced it.
Those suggestions play the role of gradients. Applied repeatedly, they move the prompts toward versions that perform better against the objective.
Evaluation first
Optimization is only as good as what it optimizes toward. So every change is judged against a defined objective: a measurable test of whether the output is actually better. Without that, an optimizer will happily improve a number that does not reflect what anyone cares about.
Whole pipelines, not single prompts
Because feedback can be traced back through a pipeline, the approach can improve several connected components together rather than tuning one prompt in isolation.
Why the objective is most of the value
The most useful part of adopting systematic optimization is often the discipline it forces before any optimization happens. To use it at all, a team has to write down what “better” means. That conversation surfaces disagreements that intuitive tuning hides. Once the objective is explicit, improvement stops being a debate and becomes a process.
Where it can go wrong
- Gaming the metric. An optimizer will find ways to score well that do not reflect real quality. The objective has to be hard to game, and somebody has to look at the results.
- A biased critic. When a language model generates the feedback, its blind spots become the optimizer’s blind spots.
- Overfitting to test cases. A prompt tuned against a small set of examples can do worse on everything else. Evaluation sets must be representative and held out.
When this approach fits
Systematic optimization earns its cost when a pipeline is important enough to measure, runs at enough volume that small improvements matter, and has an objective that can genuinely be written down. For a one-off prompt used occasionally, careful manual iteration is still the pragmatic choice.
Questions to ask before building one
If you are considering a system like this, these are the questions we would want answered first.
- Can you write down what better means? If the objective cannot be stated and measured, systematic optimization has nothing to aim at.
- Do you have representative test cases? Optimization is only as good as the examples it is judged on, and they must reflect real use.
- Who reviews the results? A person should check that a higher score really means better output.
- Is the pipeline important enough? The investment pays off for high-volume, business-critical pipelines, not occasional prompts.