What happens when you let a self-improving harness run unattended on telecom

On May 3rd we pointed Reflex at tau-bench telecom, one of the harder customer-service benchmarks in the public set, full of long procedural workflows that punish agents for skipping steps, and walked away. We wanted to know what a self-improving loop could do with no human in the optimization path. Sixteen iterations later, the agent's verified val accuracy had moved from 0.400 to 0.875, the regression set had grown from empty to fourteen permanently-locked tasks, and the loop had autonomously discovered ten distinct prompt sections: protocols a domain expert would recognize, written in plain English, justified by the actual judge transcripts that motivated them.

This is the story of that run. Numbers are real, baked from the telecom-claude-sdk-002 state directory.

The setup

Haiku 4.5 was the agent under test, on both sides of the conversation, as agent and user-simulator. Opus 4.7 played every meta role: the judge that scores the rollouts, the diagnoser that reads each failed transcript, the analyzer that synthesizes themes across failures, and the proposer that drafts the actual patch. Two pools of tasks: 74 in the training pool that the loop is allowed to learn against, 40 in a held-out validation pool we never let it see.

Apart from picking the domain, we did nothing. No prompt engineering, no theme curation, no manual rejection of bad patches. The harness ran end-to-end and wrote its own playbook.

The trajectory

The shape of the run is the canonical improver curve: a steep early run-up, a long plateau, and two late edge-case lifts that look small but matter.

The first three iterations did most of the heavy lifting. They all converged on the same theme (call it t001, the data-limit-and-refuel-pre-transfer pattern) and chained +0.20, +0.075, +0.075 in a row. By iteration 3 the agent was at 0.75. Then iteration 4 plateaued. Three patch attempts, none of them survived the regression gate.

What happened next is the most interesting moment of the run. The proposer did not keep grinding on t001. It pivoted.

The proposer had t001 and t002 in front of it every attempt. When the dominant theme stopped paying, it switched levers, and unlocked +0.10 in a single shot.

That single iteration, iter 5, the pivot to t002 (phone-number-to-line-id resolution), was the difference between a 0.75 plateau and a 0.85 one. It is the kind of move a junior engineer would not make: the obvious thing was to keep optimizing the failing theme. The harness had something better than instinct, which is a shortlist.

The long plateau, and what broke it

Iterations 6 through 10 are a wall. Five iterations, fifteen patch attempts, zero accepted. The score sits at 0.85 and the proposer keeps drafting fixes that look reasonable on paper and break two or three regression-locked tasks the moment they hit the gate. None ship.

Then iter 11 lands a one-op patch on t003 (suspended-account handling: bill-payment path before troubleshooting). It is worth +0.010 absolute. Three iterations later iter 14 lands another, on t004 (the refund-vs-refuel boundary), worth +0.015. The kind of behavior that only surfaces when the dominant patterns are already fixed. The kind of edge case a human reviewer would need many transcript-hours to even notice.

What the loop actually wrote

The artifact of all this is not a score. It is a playbook: ten prompt sections written in plain English, each justified by the failure cluster it closed. Every one of them survived every subsequent iteration through the regression gate. They are systemic protocols, not one-off hacks.

Read those out loud. They sound like the field manual a senior support engineer would write for a new hire. They are not what most prompt-optimization papers produce, which is usually some opaque blob of "be more careful" or a model-tuned suffix that nobody can audit. These are protocols. We can read them. So can compliance.

The regression set is the real product

The number we are most proud of in this run is not 0.875. It is 14.

Fourteen tasks, out of the 74 in the training pool, so roughly 19%, moved from "the agent fails this" into "the harness preserves passing behavior on this, forever, against every future patch." Each accepted iteration grows this set, and the gate rejects any patch that breaks more than 20% of it. That is what makes improvements compound rather than cancel.

One detail that mattered: the analyzer

This run used the new LLM-based failure analyzer in place of the old mechanical primary-tag clusterer. We did not expect it to matter as much as it did. Two effects:

Cross-tag pattern detection. t001, the data-limit-and-refuel theme that drove most of phase one, almost never had missed_required_action as its primary tag in the underlying judge output. It was the secondary tag in nearly every failure. The old clusterer would have shown that pattern at frequency=1 and the proposer would not have bothered. The analyzer correctly read the actual content of the failures and hoisted the universal pattern.

Multi-theme-aware patches. Iter 1's accepted patch is the cleanest example. The proposer's own rationale stated it was targeting t001 but bundling supporting fixes for t002, t003, and t004, because the same failures repeatedly exhibited those secondary patterns. One four-op patch addressed four themes at once. The earlier mechanical run on the same domain plateaued at 0.575. A 0.275 ceiling lift came purely from changing how failures get synthesized.

Where this run stops, and why

By iteration 14 the gate had become a wall. Every drafted patch broke at least three of the fourteen regression-locked tasks, past the 20% threshold. Diminishing returns, of the meaningful kind: the score is not stuck because the proposer is bad, it is stuck because the system is correctly refusing to ship overfits.

To go higher we would need one of three things: a genuinely novel theme the analyzer has not surfaced (likely model-bound, low fixability for a self-improvement loop); a looser regression threshold (we will not do this); or a larger training pool that exposes failure modes outside the current 74. For Haiku-as-agent on telecom in this configuration, 0.875 is a defensible asymptote.

What we are taking from this

Four things, roughly in order of how surprised we were.

The first is that +0.475 absolute val accuracy is achievable with zero human prompt engineering. We did not edit a system prompt. We did not curate a single rule. The loop ran unattended and the curve happened.

The second is the regression set. Average accuracy is a noisy headline number. Fourteen tasks the harness now demonstrably preserves: that is the actual moat. Maintained-test-suite, not snapshot-leaderboard.

The third is that the patches are interpretable. They are not weight deltas, prompt suffixes, or in-context tricks. Every section is a concrete protocol rule a domain expert would recognize, with the failure transcripts that motivated it preserved alongside it. The system is not searching prompt-space randomly. It is reading judge output and codifying observable failure modes into rules a human could approve.

The fourth is the long tail. Iterations 11 and 14 each added less than 0.02 to the headline score. They are easy to dismiss in a chart. But each of them represents a class of failure (suspended accounts, refund versus refuel) that only surfaces once the dominant patterns are gone. That is exactly the work human engineers spend disproportionate hours on, and the work that doesn't get done when the team is shipping features. We watched the loop do it overnight.