Building Effective Autoresearch Systems: 2

This is the second of two posts about autoresearch systems: software that proposes a change, runs an experiment, reads the result, and keeps the change if it worked. The first post asked whether you can believe what these systems tell you. This one is about how you would build one that deserves to be believed.

Here is what we are going to go through, in the order the system does it.

It starts by deciding what the campaign actually is: what is being optimised, what is allowed to change, and what would count as cheating. Then it picks something to change and tells the proposer about it, which means choosing carefully what the proposer is allowed to see and exactly what kind of edit it may make. Then it has to choose which candidate to build on, keep the search from collapsing onto one idea, and decide where to spend the next hour of compute.

Then it measures. That part splits into four: filtering out the obvious losers cheaply, keeping the answers out of the candidate’s reach, choosing which data it is allowed to measure on, and coping with the fact that sometimes the marker is another language model.

Then it decides whether the result was real or just noise, and whether it can afford to find out. Then it has to survive a crash at hour nine, prove three weeks later where a number came from, remember what it learned in a way that does not rot, and finally measure whether the whole loop is working at all.

Seventeen parts. Every one of them has a characteristic way of going wrong and a small set of mechanisms that stop it. Most of those mechanisms are a few dozen lines. Almost none of them are written down in a paper, and they do not know about each other, so the same idea has been invented four times under four names and refined by nobody.

Before the tour, one example of the kind of thing we are talking about.

Somewhere in ASI-Arch, in a file called candidate_manager.py, there is a function that throws away results for being too good.

It takes the loss a candidate architecture achieved, compares it to the reference, and if the improvement is more than about ten percent it refuses the result. Not “flags it for review”. Refuses it. The comment says: allow loss delta within 0.1 range, but no upper limit for benchmark.

The first time you read that it looks insane. You built a machine to find improvements and then you taught it to reject the big ones. The second time you read it you realise it is the most experienced line of code in the file. Somebody watched a loop announce a twenty percent gain, went to look, and found the candidate had shortened the evaluation. Then it happened again. And at some point they stopped debugging individual incidents and wrote down the general rule: in this system, a result that good is evidence of a leak, not a discovery.

That is what the rest of this is. Not the big architectural ideas behind self-improving systems, which you can read about anywhere, but the small hard-won rules that people who actually ran these loops for months ended up writing into their code.

Here is the shape of the thing we are walking through.

A word on how to read it. Every mechanism below is real, and comes from a working system rather than from a proposal. But a mechanism being real does not mean it works where it lives. A fair number of these ship broken, switched off, wired to nothing, or subtly undermined by something else in the same codebase, and where that is true I say so. The idea is still good. The implementation just is not evidence for it.¹¹One more thing before you start borrowing. An idea is free to reimplement and a file is not: a mechanism described here in a sentence can be rebuilt however you like, but lifting the actual code means taking its licence with it. A couple of the projects below are explicitly non-commercial or enterprise-licensed, and I say so where I know.

Start at the top of the loop.

Deciding what the campaign is before it starts

Almost every failure further down this list has an ancestor here. A loop that will run for nine hours needs to know, before it starts, what it is optimising, what it is allowed to touch, what counts as done, and what would count as cheating. Write none of that down and you get the classic outcome: a system that improved a number for reasons nobody can reconstruct.

Hash the whole agreement, and check it at both ends. The strongest version of a campaign contract does not just write down the rules, it hashes them. Everything the campaign depends on goes into one object: the evaluator command, the check commands, the protected paths, what each metric actually means and in which direction, and the noise model. The user accepts that object once, and its hash travels with the campaign. The clever second half is a consistency rule at the other end: the metric a worker logs must equal the metric the evaluator parsed. If those two ever disagree, something in between has rewritten the result, and you find out immediately rather than three hundred candidates later. Seen in codex-autoresearch.

Or at least fill in a form and have a human sign it. The lighter version is a preflight spec you complete before the first mutation: the objective, the core metric, the secondary metrics, the exact evaluation command, an explicit timeout, the success and stopping criteria, the writable scope, and the baseline scores. Then a person approves it. It reads like bureaucracy until you notice that the explicit timeout alone saves you from the most common overnight failure, which is one candidate silently eating the entire budget while the other three hundred never run. Seen in ASI-Evolve and MLGym.

Write down the ways it could cheat, in advance. If you can name a shortcut before the run, you can require the system to prove it did not take it. So the contract carries a list of things that would look like success and are not: throughput measured on truncated prompts, say, or accuracy computed after dropping the hard cases. Verification then has to mark each listed shortcut as rejected, violated, unresolved or not applicable, with evidence for the call. This is the difference between hoping your system does not cheat and requiring it to file a short report on the specific ways it did not. Seen in get-physics-done.

Freeze the analysis plan before any data comes in. The plan says which metric is primary, what would count as success, when to stop, whether the system has seen this data before, and whether this campaign is exploratory or confirmatory. It is hash-chained and frozen before a single evaluation runs. If that sounds familiar, it should: this is preregistration, including the exploratory-versus-confirmatory distinction that psychology spent a decade arguing about, arrived at independently by people building code-editing agents. The prior-exposure flag is the part I would copy first, because it is the only honest way to handle a benchmark your model has probably already read. Seen in k-dense-byok.

Give every candidate the same compute, and hold the clock yourself. Two structurally different candidates are only comparable if they were given the same resources, so each trial gets an identical compute budget. For a serving system the equivalent is a fixed request trace or a fixed number of GPU-seconds. The part to get right is where the clock lives: enforce it from outside the candidate. Seen in Karpathy’s autoresearch, which enforces it inside the very file the agent is allowed to edit, leaving the budget one mutation away from being edited.

Give the proposer one small surface to write on. One file, with explicitly marked editable and fixed regions, and everything else read-only. This is the single highest-value constraint in the early part of the system, because it collapses an unbounded edit space into something a person can reason about and a reviewer can diff. Take it one step further than either source does, though: keep the verifier out of the proposer’s readable tree as well, not just its writable one. Reading the test is most of the way to passing it. Seen in Karpathy’s autoresearch and autoagent.

Standardise the shape of a run, not just its budget. One run command. A fixed output directory per run. A results schema that carries means and standard errors. A precomputed baseline sitting in run zero. A snapshot of the code saved with each run. The schema requiring standard errors is doing quiet work there: a results format with nowhere to put the uncertainty is a results format that will be read as if there were none. Seen in AI-Scientist.

Make every stage declare what it consumes and produces. Each stage of the pipeline names its input files and its output files, and the executor checks them before the next stage starts. It turns a vague “the pipeline broke somewhere” into a named stage and a named missing file, which matters a great deal at three in the morning when you are reading a log rather than watching it. Seen in AutoResearchClaw.

Let the search space describe itself. A hand-maintained registry of tunable knobs drifts out of date the moment somebody adds a feature. Instead, a config field marks itself as optimisable and carries its own search space, with an allow-list deciding what is eligible. The registry cannot go stale because there is no registry. Add a deny-list covering the evaluator subtree when you build this, or the search space will eventually discover the thing that grades it. Seen in NeMo-Agent-Toolkit.

Give every search engine exactly one way in. Register each patchable field by path, then apply and restore a configuration under a lock. Every engine now reaches into the system under test through one seam instead of each growing its own way in, and a crashed trial cannot leave the system half-configured, which is the kind of bug that shows up as unexplained variance rather than as an error message. Seen in EvoAgentX.

Represent the system as a tree you can address and edit by part. Instead of treating the thing under optimisation as text, treat it as a selector-addressable tree of components. Typed operators then edit one subtree at a time, recording both the action taken and the actions that preceded it. Two things fall out for free: genuinely structural moves rather than string edits, and per-operator credit assignment, because you know exactly which operator touched what. Seen in SAMMO, which computes that credit and then never uses it. Wire it into operator selection and you have a loop that learns which kind of edit is working.

Declare parameters where they are used, and tell the model what they mean. Rather than a central schema, each parameter is declared at its point of use with a plain-language context string attached that the proposer reads. So the proposer is not staring at alpha: float, 0.0 to 1.0, it is reading a sentence about what alpha does. Trials are then coordinated ask-and-tell over a shared SQLite study in WAL mode, which is a pleasingly boring way to get parallel search without a scheduler. Seen in Optim-Agent.

Split a pipeline into stages and search each one separately. You get per-stage artefacts and a per-stage best, which means an interpretable leaderboard for each part of the system instead of one number for the whole thing. When something regresses you know where. The obvious missing piece, which no version I found implements, is an iterated pass: once later stages change, the optimum for an earlier stage usually is not what it was, and nothing goes back to check. Seen in AutoRAG.

Search the cheap dimension first and freeze it. Numeric search runs first, its winners are frozen, and only then does prompt search start, resuming with a trial-number offset so the two phases share one history. The ordering is the insight. Prompt changes are expensive and high-variance, numeric ones are cheap and low-variance, and searching both at once means your cheap dimension gets drowned out by noise from your expensive one. Seen in NeMo-Agent-Toolkit.

Put every search strategy behind one signature. Register strategies by config type behind a single entry point, roughly “run this base config over this space with this optimiser config”. Swapping a random search for an evolutionary one, or for a bandit, then costs a config line instead of a refactor. That matters more than it sounds, because the honest answer to “which search algorithm should I use” is usually “try three”, and any friction there means you will try one. Seen in NeMo-Agent-Toolkit and GEPA.

Or get the same decoupling from four interfaces. Problem, Organism, Evaluator, Mutator. That is the whole seam. The detail worth stealing is inside Problem, where failure cases are split into trainable and holdout sets and weighted by failure type. Most systems treat every failure as equally interesting. They are not, and weighting them is a one-line way to say so. Seen in darwinian_evolver.

Run the campaign in phases with different jobs. Get it working, then tune, then explore, then ablate. Each phase gets its own node budget and starts from the previous phase’s best result as its root. The structure is good because those four phases genuinely want different search behaviour: getting something working rewards persistence, exploring rewards variety. One change I would make is to compute the promotion statistics between phases in code. Seen in AI-Scientist-v2, which asks the model whether it is time to move on, which is exactly the kind of judgement a model is bad at and a comparison is good at.

All of that fixes the campaign in place. The next question is what, inside that frozen space, is actually worth changing.

Picking what to change next

Most loops answer this with a shrug: let the model decide. That works until the search space is bigger than a single file, at which point what the model finds interesting and what is actually limiting your result stop being the same thing. The techniques here are all ways of taking the decision away from the model and giving it to a measurement.

Profile first, and let the measurement pick the target. Profile the workload, rank candidate targets by their share of total cost, attack the top few, and retire a target once it reaches its threshold, with each target running on its own clock. The ranking is Amdahl weighting, which is the correct answer to “what should I optimise” and has been for fifty years: a thing that is three percent of your runtime cannot make you more than three percent faster no matter how brilliant the fix. Without this the loop optimises what is interesting, which is what a language model is drawn to and what nothing in a default system pushes back on. Seen in autokernel, which describes this priority scheme and never actually computes it, so take the idea and not the code.

Or just mutate whatever is scoring worst. If your system is decomposed into separately scored parts, the next thing to change is the lowest-scoring one. That is the entire rule, it costs you one call to min, and as a default it is hard to argue with. Seen in atlas-gic, as an idea only, since the repository does not ship its loop.

Weight the gap by how much it matters, and change one thing at a time. Pick the dimension with the largest weight multiplied by headroom, so a heavily weighted dimension that is nearly maxed out loses to a lightly weighted one with room to move. Then change exactly one dimension per round. That second half is what makes the first half useful: if you change three things and the score moves, you have learned that three things together do something, which is almost nothing. One change per round means every result is attributable to a cause. Seen in darwin-skill.

One atomic change per step, and fix which way is up once. Each step makes a single improvement, and the direction of the objective is set once at the start rather than read from each node’s own report of how it did. The second half is the subtle one. If a node can tell you which way is better, then somewhere in a long campaign a node will get it backwards, and your search will spend an hour confidently walking downhill with no error anywhere. Seen in AIDE.

Treat sibling nodes as competing answers to one question. When a search branches, the siblings are alternative options for a single decision, and the search only descends onto a sibling that has been promoted. Without the rule, a tree search will happily expand three siblings in parallel and then try to reconcile three divergent subtrees that each assumed a different answer to the same question. Enforce it in the controller rather than in the prompt. Seen in OpenResearch, which leaves it to the prompt, which is to say it leaves it to chance.

Make having an idea free and pursuing one expensive. Ideas start life as briefs with no branch attached. Only a promoted brief gets a git branch and a worktree, and each implementation of it is an attempt report filed underneath. The three tiers mean the cost of having an idea is nearly zero, the cost of pursuing one is a branch, and the record of how many times you tried is built into the file layout rather than reconstructed afterwards. Most systems conflate all three and end up with a repository full of branches nobody can explain. Seen in DeepScientist.

Target chosen, you now have to tell the proposer about it. This is the part of the system with the most mechanisms and the most ways to ruin everything downstream, because everything you put in that context window is something the proposer can optimise against.

What the proposer is allowed to see

Here is the thing that took me longest to appreciate. The context window is not an input to the proposer, it is the proposer’s entire universe. Anything in there is something it can optimise against, and anything not in there may as well not exist. So the mechanisms in this part of the system divide into two kinds: careful choices about what evidence to pack in, and increasingly aggressive rules about what to keep out.

Start with the packing, and with the mechanisms that run before the proposer has seen anything at all.

Making the proposer commit before it sees the result

The cheapest quality signal in the whole loop is a prediction made before the answer arrives. A model shown the result first will explain it, fluently and in hindsight, every single time. A model that wrote down what it expected can be caught being wrong, and being able to count how often it was wrong is worth more than any amount of well-argued retrospective.

Make every run record what it was testing, and keep that record after the revert. Three fields do most of the work: the hypothesis the run was testing, the reason it was rolled back if it was, and a hint about what to try next. The crucial part is that the record survives the revert. Most systems delete the history of a failed branch along with the branch, which leaves the loop free to rediscover the same dead end forever. A fourth field, a pointer back to an earlier discarded run, asks whether new evidence reopens something previously binned, which is the closest thing in this whole catalogue to a system changing its mind on purpose. Seen in pi-autoresearch.

Make the proposer predict which tasks it will fix before it finds out. Before evaluation, the proposer names the tasks it expects this change to fix and the tasks it considers at risk. Then the verdict is not a judgement at all, it is set arithmetic over which tasks actually flipped: effective, mixed, or harmful, computed rather than assessed. I love this one. It converts “did this help?”, a question a model will answer optimistically, into a set comparison, which has no opinions. One gap to close: compute part of the risk set from the blast radius of the files that changed, because if declaring risk costs the proposer anything, it learns to declare less of it. Seen in agentic-harness-engineering.

Make each proposal state a hypothesis you could falsify. A proposal is required to carry four things: a falsifiable claim, where it sits on the explore-versus-exploit axis, which candidate it builds on, and which components it touches. The explore-or-exploit declaration is the unusual field and the useful one, because it lets the controller hold a deliberate portfolio rather than discovering after the fact that everything it ran this hour was a safe tweak. Seen in GEPA.

Make the worker bet before it acts. The worker records a prediction before running anything, and the controller then runs gates for anomaly, novelty and what one project calls a kill shot, to catch a loop that is circling or one that has concluded too early. The prediction does something subtle: a system that writes down what it expects can be caught being surprised, and being surprised is the signal you most want and least often capture. Seen in Agent-Native-Research-Artifact.

Those four are all about the proposer’s own words. The rest of the packing is about the evidence you hand it.

What to show it about the past

Every loop shows the proposer something about what has already been tried, and the choice is genuinely hard: too little and it repeats last week’s mistakes, too much and the signal drowns. The systems differ on four questions, which is what the mechanisms below sort into. Which past attempts do you show, in what order, described in how much detail, and with which names taken off?

Put the current best immediately next to the instruction. Show the top handful of prior solutions sorted ascending by score, so the incumbent ends up in the position the model attends to hardest, right before the ask. The most extreme version of this idea strips everything else away: no scores, no harness, no instructions at all. It shows the model two versions of a function, renames the second one’s docstring to “Improved version of version zero”, and asks for version two. That is the entire prompt. The instruction is not in the instruction, it is in the naming convention. Seen in OPRO and, in that stripped form, FunSearch.

Show it the examples the incumbent actually fails. Not a random sample, the ones it gets wrong most often. Random examples mostly teach the proposer about cases it already handles, which is a way of spending tokens to learn nothing. Seen in OPRO and PromptWizard.

Report which specific tasks a change rescued and which it broke. For each candidate, list the task IDs it fixed and the ones it regressed relative to its parent, and feed that back as evidence. An aggregate score says a change was worth 1.2 points, which is a fact you cannot reason from. A flip table says it fixed these four and broke that one, which is a sentence a proposer can act on. Seen in Raven.

Carry the losers forward as explicit negative constraints. Rejected strategies, pruned lessons and their measured scores go into the next prompt, held in an append-only ledger of outcomes. The append-only part matters more than it sounds: a ledger you can rewrite is a ledger that will eventually forget the inconvenient entries, usually during a compaction pass nobody reviewed. Seen in agentic-harness-engineering, EvoSkill and Arbor.

Show the family, not the individual, so it reasons about differences. The analysis step sees the parent, the grandparent and the siblings with their scores, which pushes it towards reasoning about deltas instead of absolute numbers. Models are much better at “this is worse than that” than at “this is good”. Rank the siblings by score when you build it. Seen in ASI-Arch, which ranks them by child count instead, telling the model what the search has already explored rather than what worked.

Let the reasoning travel down the tree with the code. Store a written analysis next to each candidate’s code and score, and when a parent is sampled its analysis comes along. The thinking propagates with the genes, so the proposer inherits not just what its parent did but why somebody thought it would work, which is the part that generalises. Seen in ASI-Evolve.

Build the prompt from this branch only. Take the attempts and measured failures on the path from the root to this node, and nothing else. Sibling branches never contaminate each other. Without this rule, a tree search gradually turns into one long conversation in which every branch has read every other branch, and the branch independence you were counting on quietly stops existing. Seen in LanguageAgentTreeSearch.

Send the shape of the curve, not the whole log or a single number. Compress each metric’s full trajectory to about ten points plus the maximum, mean and minimum. You keep the shape, which is what actually carries the diagnosis, at a fraction of the tokens. Most systems either dump the entire log or send one final number, and a final number throws away the difference between “converged” and “diverged, then got lucky at the end”. Seen in Eureka.

Hide the grader’s name. Show the gate metric to the proposer under a bland label, something like task_score, so the evaluator’s identity does not leak. If the proposer knows which grader it faces, it will write for the grader rather than for the task. This is a one-word change and it is blinding, in the clinical sense, arrived at by people optimising reward functions for robots. Seen in Eureka.

That is the evidence going in. The harder half is what you do with the evidence coming back, because most of what comes back is a failure, and a failure is not by itself an instruction.

Turning a failure into an instruction

The step that does the converting is reflection, and it is the one nearly everybody implements and almost nobody structures. Left alone, a model will produce a paragraph of sympathetic prose about trying harder next time, which reads like learning and changes nothing. Every mechanism here is an attempt to make the output of a failure into something the next attempt is not able to politely ignore.

Lay the whole step out under fixed headings. Code, inputs, other context, outputs, feedback. With that structure, one call can assign blame across many parameters at once instead of one at a time. Two refinements are worth copying: comment exceptions directly into the source at the failing line, so the error arrives attached to the code that caused it rather than in a stack trace the model has to reunite with the file, and attach constraint strings to parameters so the proposer knows what it is not allowed to do to each one. Seen in Trace.

Have the evaluator return more than a number. A scalar is a terrible return value from an evaluator, and three projects independently decided so. One returns a per-component score plus text. One returns an array of scores plus labelled insights. One returns metrics plus an artefact side-channel, with size-split storage so large artefacts do not bloat the database and redaction so secrets do not travel. All three are saying the same thing: the evaluator knows more than one number’s worth, and throwing the rest away is a choice you are making. Seen in DSPy, AlphaEvolve and OpenEvolve, the last of which fetches the first few artefacts in its database rather than the parent’s, so the proposer can end up reading somebody else’s evidence.

Split diagnosis from fix into two calls, and forbid the first one to propose anything. The benefits stack up fast. The critique becomes an inspectable artefact you can read and argue with, it is cacheable, and the writer never sees the raw trace, only the diagnosis. That last one is a leak barrier hiding inside an architecture decision: a model that diagnoses and proposes in one breath will write the diagnosis that justifies the fix it already wanted. Seen in LMOps, TextGrad and ACE.

Keep each critique attached to the conversation it came from. And give every optimisable artefact a role description saying what it is for. A critique detached from its context is an instruction with no scope, and instructions with no scope accumulate into a prompt that contradicts itself by generation forty. Seen in TextGrad.

Stage the lesson as the model’s own previous turn, not as a correction. Turn a raw failure into a short note about why it failed and what to try, then insert that note into the conversation as though the assistant had written it. The framing is the technique. A model reading its own prior reasoning continues it. A model reading a user telling it that it was wrong either defends itself or over-corrects. Same information, different conversational position, measurably different behaviour. Seen in Reflexion.

Distil reflection into two lists: what to avoid and what to try. Run a structured reflection asking whether the diagnosis was correct, whether the fix worked, what to avoid and what to try next, then feed the last two forward as explicit lists. Simple, legible, effective, and it will drown you by generation thirty unless you add the two things the source does not: deduplicate the entries, and decay them. A list of things to avoid that only ever grows eventually forbids everything. Seen in KernelAgent.

Vary how much history you show by what you are asking for. Debugging gets eight related experience cards; improving and crossover get three. Debugging genuinely needs more context than proposing does, and treating them identically means either starving the debugger or drowning the proposer. The same system also scrubs scores out of the prompt text, which is the blinding instinct again. Seen in OpenRSI.

Sample failures round-robin, batch them, and ask what they have in common. Taking the worst failures lets one noisy category monopolise attention, so sample across categories instead. Batching several together pushes the model towards a general cause rather than a specific patch. And when the context overflows, fall back to the single shortest failure, which is a graceful degradation that still says something rather than nothing. Seen in EvoSkill.

Route by what actually happened, and spend the good model on the good lineage. A parent that errored gets a bug-fix prompt; a parent that ran gets an improve prompt; stronger parents get a stronger model tier. Spending your expensive model on your most promising lineage is straightforwardly correct. Declare the model roles explicitly if you build this. Seen in OpenAlpha_Evolve, whose environment-variable fallbacks can silently invert the tiers, and a loop quietly running its cheap model on its best lineage is a very hard thing to notice.

Everything so far has been about getting the right things into the window. The rest of this part is about getting things back out, because a context that only ever accumulates stops working somewhere around hour three.

Keeping the window clean

These split by what they evict and when. Some never let a thing in at all, some give it an expiry date, and the most aggressive ones throw the whole conversation away on purpose once it has done its job. All of them are deliberately losing information, which feels wrong right up until the first time you watch a run spend its last ten thousand tokens re-reading a stack trace from hour one.

Keep raw logs out of the window entirely. Send the full output to a file and surface only a summary, using a simple METRIC name=value line protocol on standard output plus tail-only output for anything else. This is a leak barrier as much as a token saving: raw evaluator logs are where ground truth lives, and a proposer that can read them can learn the test. Seen in Karpathy’s autoresearch and pi-autoresearch.

Show the remaining budget on every single turn, and auto-submit on exit. Remaining steps and remaining budget go in every turn, only the last few observations are kept, and any budget exit automatically submits the most recent artefact. The auto-submit is the part people skip and then regret: without it, a run that hits its limit mid-thought returns nothing at all, and you have paid full price for zero information. Seen in MLGym.

Give large tool outputs an expiry date. A dumped file carries a time-to-live measured in steps, after which it drops out of history automatically. Something essential at step four is dead weight at step forty, and manual pruning never happens because nobody schedules it. Seen in Agent Laboratory.

Or work in time-boxed windows and forget the raw steps at the end of each. Same instinct as the expiry, applied on a schedule rather than per item, with the pleasant property that the summary gets written while the details are still in context rather than reconstructed later from what survived. Seen in infiAgent.

Make plans typed slots rather than prose. Plans become tool calls with fixed names, merged deterministically. A plan in prose can only be read. A plan in typed slots can be diffed against the last one, checked for missing pieces, and rejected by a controller without asking a model’s opinion about whether it looks complete. Seen in AI-Researcher.

Rewind the conversation after a successful fix. The worker going forward sees a clean first answer rather than the three failed attempts that preceded it. It feels like cheating and it is not. Failed attempts sitting in history bias everything after them, and you have already extracted the lesson; keeping the wreckage costs tokens and accuracy. Seen in data-to-paper.

Notice how the second half of that list runs: expire it, box it, rewind it, keep the logs out entirely. A context management strategy is mostly a deletion strategy. Which brings us to the moment the proposer actually writes something, where the same instinct shows up as constraints on what an edit is allowed to be.

How a change is actually made

An edit is not just a diff. It is a move in a game, and the set of moves you allow shapes everything the search can reach. Get this part wrong in the permissive direction and your loop spends the night rewriting files wholesale; get it wrong in the restrictive direction and it can only ever fiddle. The mechanisms here are about naming the moves, ordering them sensibly, and making each one mechanically checkable.

Name the moves so you can measure them. Four distinct operations, kept separate: explore something genuinely new, recombine two backbones, mutate the structure, and mutate parameters only. The value is not the taxonomy itself, it is that once the moves are named you can measure them, budget them, and notice that one of them has produced nothing for two hundred generations. Seen in LLM4AD.

Order those moves over the life of a run. Exploration operators like induction and paraphrase first, then gradient-style refinement, then crossover between parents that sit far apart in score-vector space. Crossover last is the right call: crossing two near-identical parents produces a third near-identical parent, so you want it saved for when the population has genuinely spread out. Seen in promptimizer.

Give a prompt optimiser an explicit menu, including a delete. Add a constraint, add an anti-pattern, reorder by priority, add an example, delete a rule that over-optimises. That last one is the one everyone forgets. Every prompt optimiser I have looked at accumulates rules, because adding always looks safe, and the deletion move is what stops a prompt from turning into a thicket of contradictory special cases by generation fifty. Seen in autoresearch-skill.

Draft a few, then flip a coin between debugging and improving. Draft until you have a handful of drafts, then with probability one-half debug a random buggy leaf up to depth three, otherwise improve the best node. Scale the counts to the run: drafts at about fifteen percent of total steps, debugging at about ten. The coin flip is doing something people underrate, because a loop that always improves the best node abandons every broken-but-promising branch, and most of the interesting ideas arrive broken. Seen in AIDE and AI-Scientist-v2.

Sample the kind of edit from a fixed mix. Diff sixty percent of the time, full rewrite thirty, crossover ten. Keeping a meaningful share of full rewrites is what stops a diff-only search from being trapped forever inside the structure of its first working version, which is a real and very quiet failure mode. Seen in ShinkaEvolve.

Fan out across parent, diagnosis, model and sample, but profile once and time alone. Expand the cross-product, computing one shared profile per parent rather than per candidate, and hold a per-GPU mutex around measurement. That engineering detail is the important bit: timing measurements taken while another candidate shares the GPU are not measurements, they are a story about scheduling. Seen in KernelAgent.

Anchor edits, protect a core, and shrink the edit budget over the run. Edits become anchored diff operations rather than free-form rewrites. A strategic region is protected so only an epoch-level process may touch it. And the edits-per-step budget decays as the run proceeds. Three ideas in one: anchoring makes edits mechanically checkable, the protected region gives you a stable core that tactical edits cannot erode, and the decaying budget is an annealing schedule, big moves early and small moves late. Seen in SkillOpt.

Require an edit to quote the exact span it replaces. Either it applies mechanically or it visibly fails. No fuzzy matching, no closest-reasonable-location, no silently patching the wrong function that happened to look similar. This is the cheapest correctness win in the whole part and I would put it in any system that lets a model edit files. Seen in autonovel.

Check that a rewritten prompt still contains all its variables. The recipe is mask the variables, rewrite, assert every one is still present, escape, unmask. Without it you eventually get a prompt that scores beautifully because the hard part of the input is no longer being substituted into it, and you will spend a day being delighted before you find out. Seen in agents and promptimizer.

Cap how much the artefact is allowed to grow. A hard absolute size limit plus a ceiling of about twenty percent growth over the baseline, checked before any expensive evaluation. Unconstrained optimisers grow their artefacts monotonically, because adding text almost never hurts on the training signal and hurts almost everywhere else. Seen in hermes-agent-self-evolution and darwin-skill.

Cap how many things one move may change. At most two fields per move, with a threshold and a patience parameter. Same logic as changing one dimension at a time when choosing targets: attribution is worth more than speed, because a search that moves fast and cannot explain itself has to redo all of its work the moment something regresses. Seen in SimpleMem.

And cap the meta-level too, with a floor. When you let a model edit the search space itself, limit it to about five freeze-or-unfreeze changes per cycle, and require at least four dimensions to stay active. Without a floor, a meta-level optimiser will eventually freeze everything, because a search space of size one has no variance and variance is what it is being punished for. Seen in Bilevel-Autoresearch.

Land an edit only if the program still runs, and bound the repair. If it does not run, revert and send the error to a repair call with a fixed attempt limit. Bounded is the operative word: an unbounded repair loop is how a run spends four hours and its entire budget trying to fix a syntax error it keeps reintroducing. Seen in Agent Laboratory.

Keep the output contract next to the prompt. Treat prompts as config with a declared order, loop and extraction key, which turns them into diffable artefacts with an explicit statement of what they are supposed to produce. A prompt whose expected output shape is written down somewhere other than inside the prompt is a prompt you can validate against. Seen in agents.

Ask for exactly one object, and read it with a tolerant parser. Workers return a single object under a strict schema, with fields like summary, key changes and key learnings, extracted by a parser that tolerates surrounding prose. The alternative is having the worker write its answer to a named file that the harness reads, which sidesteps parsing entirely. Either is fine. What is not fine is asking a model for structured output and then parsing whatever it feels like sending. Seen in gnhf and SIA.

Clamp numeric proposals, re-prompt once, then fall back to real maths. Validate and clamp whatever number the model suggests; if it is invalid, re-prompt once; if it is still invalid, fall back to a classical sampler rather than to a random draw. The ladder is right and the last rung should be better still: make the fallback a proper Bayesian optimiser or tree-structured Parzen estimator, with the model acting as a prior rather than as a replacement. A language model is good at suggesting where to look and bad at the arithmetic of narrowing down, which is exactly the division of labour a hybrid gives you. Seen in Optim-Agent.

Let the proposer smoke-test its own draft on data it has already seen. Before paying for real tournament evaluation, the proposer runs its draft on a familiar slice and revises. Using seen data is the right choice here, counter-intuitively, because this is a smoke test and not a measurement: you are asking “is this obviously broken”, and for that question contamination is free. Seen in RoboPhD.

Check novelty before you check for bugs. Two cheap screens before evaluation: is this novel against the archive, and does it have obvious bugs. Ordering novelty first is correct, because debugging something you were about to discard as a duplicate is pure waste. Seen in ADAS.

Those last two are already doing something different from the rest: they are filters, spending a little to avoid spending a lot. Hold that thought, because it comes back as an entire part of the system later. First, the loop has to decide which parent it is mutating in the first place.

Choosing which candidate to build on

This is the part where the field has quietly converged on a formula, and almost nobody says so out loud. Four different projects, working independently, ended up with the same shape: a sigmoid over the score, divided by something that punishes a parent for having lots of children. The score says exploit. The child count says explore. One line, both pressures, no model call.

Here is that family, and what each variant fixed about the last.

The original: a sigmoid over accuracy, divided by one plus the child count. Sample a parent with probability proportional to sigmoid(λ·(acc − 0.5)) / (1 + children). The sigmoid sharpens the difference between good and bad candidates without letting one outlier dominate, and the child count stands in for novelty when you have no embedding to measure novelty with, which is most of the time. Seen in the Darwin Gödel Machine, whose default “best” selection option actually selects the worst, so read the code before trusting the flag.

Then: make the midpoint follow the frontier. The obvious flaw in the original is the hardcoded 0.5. The fix is sigmoid(10·(score − mean(top-3))) × exp(−(children/8)³), where the midpoint tracks the current frontier, so selection pressure rises as the population improves instead of flattening out once everything is above a fixed threshold. The cubed exponential child penalty is gentler than division early and much harsher late, which is roughly the schedule you want. Seen in HyperAgents.

Then: make it scale-free. sigmoid(10·(α − median)/max(MAD, 1e-6)) × 1/(1 + children), computed over the intersection of the archive, the island and the programs that actually ran. Using the median and the median absolute deviation instead of the mean and standard deviation means one catastrophic outlier cannot rescale the entire selection curve. And because it works in robust z-units, the same single knob behaves the same way whether your metric is an accuracy in the nineties or a loss in the thousands. Seen in ShinkaEvolve.

And: bump the child counts inside a batch. Track the sigmoid midpoint to a score percentile, and increment child counts within a batch rather than between batches. Without that second part, sixteen parallel selections all see the same stale child count and all pile onto the same parent. It is the kind of bug that only appears when you go parallel and presents as a mysterious loss of diversity rather than as an error. Seen in darwinian_evolver.

A different line entirely: count how often a parent is the best on something. A parent’s probability is proportional to the number of validation instances on which it is co-best, after pruning candidates whose every win is shared with someone else. This keeps specialists alive. A candidate that is mediocre overall but the outright best on eleven awkward instances survives, where any aggregate-score rule would have binned it generations ago, and specialists are where recombination gets its raw material. Seen in GEPA and evo, the latter adding a task intersection and a probability floor.

Ask three questions and add them up. Score a parent as its own result, plus the variance of its children’s rewards, plus a term that decays with how often it has been visited. That is “is it good”, “is it uncertain” and “has it been over-visited” in one number, and every input comes from rows the controller already stores, so it costs nothing to compute. Replace the self-reported score with a measured one if you build it. Seen in OpenRSI, which runs on the candidate’s own report and is CC BY-NC licensed, so reimplement rather than copy.

Weight score, improvement and novelty, and infer novelty from imports. Roughly 1.0·score + 0.4·Δ + 0.25·novelty, every term normalised, then softmax-sampled. The delightful part is how novelty gets computed: 1/√(1 + family_count), where a candidate’s family is inferred from which libraries it imports. No embeddings, no model call, and it genuinely captures “this candidate is trying a different kind of approach”. Normalise every term onto one scale when you build it, or your weights are not the weights you think they are. Seen in OpenRSI and QuantaAlpha, the latter mixing in a raw correlation against a label count without rescaling.

Or keep it crude and auditable. p = 0.3·uniform + 0.7·softmax(0.2·(s − max s)) over the top k. Two constants, and the first one is an exploration floor you can point at in a design review, which is worth more than an elaborate scheme nobody can audit. Seen in AFlow and EvoAgentX. A note for anyone reading AFlow’s paper: it presents this as Monte Carlo tree search, and there is no tree search in it.

Sample behaviour clusters rather than programs, and anneal repeatedly. Cluster programs by their per-test score signature, then softmax-sample clusters with a temperature that cools as T = 0.1·(1 − (n mod 30000)/30000). Two ideas worth separating. Sampling clusters means a hundred near-identical high scorers get one vote between them instead of a hundred. And the modulo makes the temperature sawtooth: it cools, resets, cools again, so the search anneals over and over rather than freezing once and staying there. Seen in FunSearch.

Or branch on a random number and be done with it. Below 0.2 pick uniformly from the island; below 0.9 pick uniformly from the island’s archive members; otherwise pick fitness-proportionally. Crude, legible, and tunable by somebody who has never read a paper on selection pressure, which is a real advantage in a codebase other people have to maintain. Seen in OpenEvolve.

If you have no population at all, alternate on a schedule. Switch on a fixed cadence between conditioning on the most recent attempts and conditioning on the best so far. It is diversity maintenance for systems that never had a population, which is most real deployments, and it costs one counter. Seen in AutoPrompt.

Pick the best feasible parent, not the best parent. The next parent is the best run that satisfies every hard guardrail metric. Constraints first, objective second. If you have hard constraints, and you usually do, this ordering is the only correct one, and it is surprising how many systems optimise the objective and then check the constraints afterwards. Seen in FAROS.

Rank by head-to-head duels, and replay the ratings from scratch. Candidates meet on shared samples, and the Elo table is recomputed from the full match history rather than updated incrementally. Replaying is slower and completely worth it: an incrementally updated rating depends on the order the matches happened to arrive in, which makes it impossible to audit and impossible to reproduce. Seen in RoboPhD.

Every one of those formulas has a child-count term, a cluster or an island in it, because every one of them is fighting the same enemy. Left alone, a scored population collapses onto one lineage and stops discovering anything. That fight deserves its own part of the system.

Keeping the search from eating itself

A loop that only exploits converges, and converged is a polite word for stuck. The mechanisms here fall into two halves: structural ones that keep a population spread out by construction, and trigger ones that notice a plateau and do something about it. The second half has a rule worth applying generally: the trigger should live in the controller and fire on a counter, never in the prompt and never on a model’s opinion about whether things feel stuck.

Keep an archive of the best candidate per kind of behaviour, not the best candidate overall. A candidate’s niche is a deterministic function of its measured behaviour, for instance a cell in a latency-by-cost grid, and fitness decides who owns that cell. You end up with a collection of specialists rather than one champion, which is both more useful and much harder to collapse. The centroidal Voronoi variant needs only a niche count rather than a per-dimension grid, and degrades gracefully when you have more than two or three descriptors. Seen in OpenELM and EvoAgentX.

Compute the niche from what the patch touched, not from what the model called it. This is the difference between a diversity archive and a creative writing exercise. Ask a model to label its own approach and you will get variety in the labels long after the approaches themselves have converged. Bind the coordinate to something mechanical. Seen in Raven.

Use the pass-fail signature as the unit of both deduplication and diversity. Two programs that pass and fail exactly the same tests are the same program for search purposes, whatever their source looks like. For configuration search the equivalent is the per-workload metric vector. This is the cheapest useful behaviour descriptor there is, because you already computed it. Seen in FunSearch.

Or embed the code itself when behaviours are too similar to separate. Build syntax-aware code embeddings, standardise them, run PCA down to at most ten dimensions and K-means into a handful of niches. It is the heavyweight option, and worth it when your candidates are large programs whose test signatures are nearly identical. Seen in LLM4AD.

Weight Pareto dominance by how different the code is. For a multi-objective front over latency, quality and cost, this keeps the front spread out without anybody hand-tuning a scalarisation. Hand-tuned scalarisations are where multi-objective search goes to die, because the weights quietly encode an answer to the question you were trying to ask. Seen in LLM4AD.

Run islands, migrate on a ring, and tag the migrants. Five islands; every fifty generations the top programs copy to both ring neighbours; identical code is deduplicated exactly. The tag is the detail people miss: migrants are marked and never migrate again. Without it, one strong program ping-pongs around the ring until every island is running the same thing, which is the precise opposite of what islands are for. Seen in OpenEvolve.

At the population cap, evict the ones that own no niche first. And never evict the incumbent best. Evicting purely by score quietly destroys your archive, because the candidate holding a weird empty corner of behaviour space is usually not a high scorer, and it is exactly the one you kept the archive for. Seen in OpenEvolve.

Periodically wipe half the islands and reseed them. Kill the weaker half and restart each from a random survivor’s best, so half the population is always young. Brutal and effective. Change one thing if you copy it: trigger the reset on evaluation count rather than on wall-clock time. Seen in FunSearch, which uses four hours of wall-clock, meaning the same experiment on a faster machine is a different experiment.

Assign the explore-exploit trade-off to workers instead of resolving it per decision. Each parallel worker sits at its own fixed point on the spectrum, migration is gated on stagnation, and a quota of restarts begin with no parent at all. Holding a portfolio at all times is a genuinely nicer answer than oscillating between modes, and it makes the trade-off visible as a configuration rather than buried in a constant. Seen in LLM4AD_Next.

Give workers explicit roles. Lanes and postures, including dedicated reviewer and technical-writer roles. Forced role diversity, in place of hoping that identical agents with identical prompts will somehow diverge. They will not. Seen in CORAL.

Alternate the kind of hint when you cannot measure novelty. Structural hints one round, guidance-level hints the next. If you have no novelty metric, you can at least alternate the category of change you ask for, and that gets you most of the way there for none of the machinery. Seen in agentic-harness-engineering.

On a plateau, convene a council and rank its proposals blind. After a set number of iterations with no improvement, generate several single-change proposals under fixed philosophical lenses, rank them anonymised, and cascade-evaluate the best one. The anonymised ranking is the good part and needs one fix: rank with a separate model, or at least a fresh context, because a model ranking proposals it just wrote is not anonymising anything. Seen in auto-researchtrading.

Force a structural change after too many parameter tweaks, and protect it. Three parameter-only iterations, or two empty ones, and the next change must be structural. Then protect that new structure for several iterations before any rollback is allowed. The protection window is what makes it work: a structural change almost always looks worse immediately, and without protection it gets reverted before it can pay off. Put the rule in the controller. Seen in AutoSOTA and ARIS, both of which leave it to prompt memory, and prompt memory forgets.

Inject reflection on a heartbeat, not on request. Interval and plateau-anchor triggers, with an epsilon and a cooldown, deterministically push reflection, a pivot, or consolidation into long-lived workers. Deterministic is the word that matters. A long-running agent that decides for itself when to step back and reflect will never do it, in much the same way that you never do. Seen in CORAL.

Swap the whole search algorithm when progress stalls. Rotate between engines on stagnation, with all of them drawing on one shared budget. This is the payoff for putting every engine behind one registry earlier: once swapping the search algorithm costs a config line, swapping it automatically becomes a reasonable thing for the controller to do. Seen in GEPA.

Check new ideas against the ones that already failed, lexically and semantically. Hybrid keyword and vector similarity at a threshold around 0.7, with bounded regeneration when it hits. Hybrid matters because lexical and semantic matching fail in different directions, and an idea rephrased is still the same idea. Seen in InternAgent.

Keep, on each parent, a list of what has already been tried on it. Fed back to the proposer and checked again before evaluation. Belt and braces, which is right, because a proposer told not to repeat itself will still repeat itself. Compare semantically when you build it. Seen in AFlow, which matches exact strings, and no two model outputs are ever exactly equal.

Hash each plan, and back off when the same one turns up three times. Three lines, and it catches the specific failure where a loop has stopped thinking and started pacing. You will meet this on your first long run. Seen in auto-deep-researcher-24x7.

Diversity keeps the search alive. What it does not do is tell you where to spend the next hour of GPU time, and by this point you have far more plausible things to run than budget to run them with.

Deciding where to spend the next run

Allocation is where classical computer science shows up and quietly outperforms everything clever. A bandit over which model proposes next, or which island gets the next evaluation, is a few dozen lines of well-understood maths, and it beats any amount of asking a language model where it thinks the budget should go.

Run a cost-aware bandit over which model writes the next proposal. Reward is the improvement over the better of the parent and the baseline, clipped at zero. Rewards decay at 0.95 so old evidence fades. There is an exploration floor of 0.2 so no model is ever fully abandoned. A cost coefficient of 0.5 means an expensive model has to be meaningfully better rather than just better. And pending submissions count as pulls, which is the detail I would least want to be without, because it is the difference between a bandit and a stampede when several workers pick at once. Seen in ShinkaEvolve.

Decide whether to widen or deepen with a single Thompson draw. The most fundamental decision in tree search, whether to add a sibling or go deeper, gets made by sampling from two posteriors held at each node under a normal-inverse-gamma prior: one for generating a new child, one for continuing. One sample from each decides. About twenty-five lines, no model call, and it replaces the pile of hand-tuned widening heuristics that usually occupies that slot. Seen in treequest.

Pool the rewards by action name across the whole tree. Store rewards per action rather than per node, and the same machinery becomes a drop-in bandit over which model proposes next: the action “ask the big model” accumulates evidence everywhere it is tried instead of starting from scratch at every node. Seen in treequest.

Run a bandit over islands, and clamp each island’s exploration. Decayed, globally normalised improvement feeds a UCB over islands, and each island’s exploration intensity comes from a moving average of squared relative gains, clamped to between 0.15 and 0.5. The clamp is the practical part: an island that got lucky once would otherwise be assigned near-zero exploration and stop being an island at all. Seen in skydiscover.

Put the bandit over the kind of work, not the specific item. Linear Thompson sampling chooses between categories of work, using the last experiment’s metrics as context. Choosing the category is an under-used place to put a bandit, because categories accumulate evidence far faster than individual items do. Seen in RD-Agent, where a trailing space in a metric key zeroes two of its context features, which is the sort of bug that turns a contextual bandit into an ordinary one with no error anywhere.

Put one interface in front of several best-arm algorithms. Candidates and examples in, scores out, backed by brute force, UCB, successive rejects or successive halving. You then trade evaluation budget against selection confidence by changing a string, without touching the search at all. There is a classic bug to fix: unpulled arms must start at positive infinity so everything gets tried once. Seen in LMOps and APE, the latter returning zero for unpulled arms, which sorts them last, and the arms you never pulled are exactly the ones you needed to.

Fade exploration as the budget runs out. Multiply the exploration term by the fraction of budget remaining: norm(Q) + λ₀·eval_remain·√(ln(N+1)/n), with eval_remain = 1 − used/max, plus progressive widening. Standard tree search has no way to express this, and exploring on your last ten evaluations is just losing. Seen in LLM4AD.

Or decay the exploration constant on a fixed schedule. Hold it at 1.414 until step forty-five, then drop it by 0.01 per step to a floor of 0.5. Fine as a schedule, and not fine in the source, where the reward collapses to plus or minus one because the best-metric variable is never assigned, so the bandit is choosing between win and lose with no sense of magnitude. Seen in ML-Master.

Reserve the children before the call, and lock the subtree. Reserve the expected children before making the model call, and lock subtrees so parallel workers stay in disjoint branches. If you are running tree search with more than one worker you need something like this, or your workers will spend the night rediscovering each other’s nodes. Seen in ML-Master.

Learn which components to switch on, then freeze the policy for the real evaluation. Each on-or-off component is an independent logit trained by policy gradient, and the learned policy is thresholded into a single deterministic configuration for the sealed evaluation, with the per-component probabilities published. Two good instincts there: the final evaluation is deterministic rather than sampled, and the probabilities are visible so you can see which components the optimiser was actually confident about. Add a moving-average baseline and a cost term. Seen in GPTSwarm, whose main path has neither, and policy gradient without a baseline is mostly variance.

Search on cheap minibatches, then promote the leader to a full evaluation. Run the numeric search over minibatches and periodically promote the best-by-mean configuration that has not yet had a full evaluation. Hold the minibatch fixed within each comparison when you build it. Seen in DSPy, because comparing two candidates on different random minibatches measures the minibatch.

When your evaluator only gives preferences, use duelling bandits. Duelling Thompson sampling chooses which pair to send to the judge next, then Copeland scoring over the win matrix produces the ranking. About ninety lines. An LLM judge really is a pairwise-preference oracle and nothing more, and this is the right machinery for one: it spends judge calls on the comparisons that are still uncertain rather than on the ones you already know the answer to. Seen in prompt-ops.

Stop spawning candidates that cannot win, and spend the rest confirming. Near the end of a budget, any candidate that provably cannot overtake the leader in the evaluations remaining is not worth starting. Spend what is left re-testing the actual contenders instead. It converts a budget you were going to waste into extra confidence about the result you are going to ship. Seen in RoboPhD.

That last one is a specific case of a much more general and much more valuable idea: work out what a candidate cannot achieve, and stop paying to find out.

Not paying full price for obvious losers

Evaluation is where the money goes. Everything in this part is about spending less of it without spending less rigour, which sounds like a contradiction and is not, because most of what you evaluate is obviously bad and you do not need a careful measurement to establish that.

Stop the moment a candidate cannot win, and lose nothing by it. You have scored some items out of a total. Assume every remaining item scores perfectly. If that best case still loses to the incumbent, nothing you learn from the rest of the run can change the outcome, so stop.

if (total + remaining) / n < incumbent_score:
    stop()          # cannot win, even if everything left is perfect

The beautiful thing about this is that it throws nothing away. It is not a heuristic, a cheap proxy or a screening model. A candidate killed by this bound provably could not have won, so the decision is identical to the one you would have made after paying in full, and on a stream of mostly-mediocre candidates it saves an enormous fraction of your evaluation budget for exactly zero loss of information. Two corrections when you build it: store the partial figure as a partial score rather than as the candidate’s score, or a stopped candidate looks like a terrible one in your records, and assert your metric’s real upper bound instead of assuming one, because if the metric can exceed one the bound stops being admissible and you start discarding winners. Seen in AdalFlow.

Put a cheap, deliberately generous screen in front of the expensive evaluator. Obvious losers die on a handful of items and never reach the protected evaluator at all. The better of the two versions I found runs a generous screen followed by a paired full confirmation, which means the screen’s errors are all in the safe direction. Make the screen big enough that its false-negative rate is genuinely small; four items is not enough to be generous about anything. Seen in AdalFlow and Raven.

Discount a partially measured candidate before ranking it. Multiply a candidate scored on only part of the suite by the fraction of the suite it actually saw, before comparing it to fully evaluated ones. It is a pessimistic prior on incomplete information, and it stops a half-measured candidate from topping the leaderboard on the strength of the easy half. Seen in HyperAgents.

Keep a cheap lane whose results do not count. A second evaluation path that does not enter the leaderboard and does not advance the plateau counters. Every system needs somewhere to poke at things without the poking counting as evidence, and without that lane, exploratory runs quietly pollute the record your stopping rules are reading. Seen in CORAL.

Let smoke mode shrink counts and never change algorithms. A quick pass may reduce how many items or steps it runs, but it must not swap in a different code path. Then a smoke pass actually predicts the real run, which is the only reason to have one. Smoke modes that take a shortcut test the shortcut. Seen in Claw-AI-Lab.

Run the entire pipeline on five samples before the campaign starts. Unglamorous, and it catches the missing credential, the wrong path and the unsupported runtime before you have burned a night on them. Seen in AutoRAG.

Forecast the cost, and ask whether the search can pay for itself. Extrapolate a run’s cost from a sample of fifty queries. Then go further and estimate break-even: whether the optimisation can recover its own cost at all. That second question is one almost nobody asks, and for a lot of small campaigns the honest answer is no. Seen in APE and SAMMO.

Fingerprint the artefact and never measure the same thing twice. Normalise the compiled output by stripping comments and renaming registers and labels by order of first occurrence, hash it, and keep the fastest candidate per hash. A lighter version hashes the content of the editable scope instead. Once you start logging how often this fires you discover it is a lot. Seen in KernelAgent, codex-autoresearch and autoresearch-mlx.

Require a child to actually change an output before it earns a full evaluation. Stronger than hashing the source, because a rename or a reordering will defeat a text hash and will not defeat this. Mutations that change nothing observable are extremely common, and without this check you pay full price to find out. Seen in darwinian_evolver.

Put the split in the score cache’s key. Key on the triple of candidate hash, split, and example ID, so a cached score can never cross splits. This is one of those defences whose value is invisible until the day it saves you: without the split in the key, a validation score can be served for a test query and your held-out set is silently no longer held out. Seen in GEPA and SkillOpt.

Check you are ready before you commit any budget. Credentials work, a training split is declared, the score type matches the evaluation type, every runtime is supported. It is a preflight checklist, and it exists because every item on it has failed for somebody at hour six. Seen in Kiln and future-agi.

Wait on the scheduler, not on the model. Between launching work and reflecting on it, block on process or scheduler state, and wake the model only at a terminal state. No polling the model to ask whether it is done yet. If your agent loop is idling at a cost per minute, this is the fix. Seen in auto-deep-researcher-24x7.

Run expensive guards only on the candidates that won. You do not need to prove that a losing candidate did not cheat. Only winners need auditing, and only winners get audited, which makes it affordable to audit them properly. Seen in codex-autoresearch.

Every mechanism above is a way of not measuring something. Which leaves a rather important question about the measurements you do take, and it is the one part of this system where being wrong is not expensive, it is fatal.

Putting the score somewhere the candidate cannot reach it

The first post was about one failure: most of these loops let the thing being tested touch the thing doing the testing. This is the part of the system that stops it, and it has more mechanisms in it than any other, which tells you how many different ways the boundary leaks.

It leaks in ways that are genuinely hard to anticipate. The candidate can read the tests. Or it cannot read them, but it can read the logs, which quote them. Or it cannot read anything, but it can write to the directory the results land in. Or it cannot write there, but it gets reverted and then its own patch is replayed later and puts the change back. Each of these was somebody’s week.

The good news is that the mechanisms are mostly small, and they nearly all follow from one sentence: unreachable by construction beats policed by convention. Once you take that seriously, most of the rest of this part writes itself.

Where the evaluation runs

The crudest version of that sentence is also the most effective one: put the measurement somewhere the candidate has no route to. What varies below is how far away “somewhere” is, running from a separate process on the same machine, through a sealed container, out to a server that holds the labels and will only ever hand back a number.

Never send the answers. The strongest version of this whole part is also the simplest: the solver process never receives the private tests at all. Not “the solver is instructed not to read them”, and not “the solver is sandboxed away from them”. The bytes are never sent. There is nothing to audit, nothing to misconfigure and nothing to regress, because the leak path does not exist rather than being closed. Every other mechanism in this part is a patch for a case where this was not possible. Seen in AlphaCodium.

Make every engine evaluate through one server, and never register the test split with it. The server meters budget and rejects unknown example IDs with an HTTP 400. So a request for a test item is not refused on policy grounds, it is refused because the server has never heard of that ID, which is a much stronger guarantee than a rule. Seen in GEPA, which undercuts itself a little by letting its agent engines see train and validation merged into one readable pool, so do not copy that part.

Hand out evaluation work as leases. Acquire, evaluate, submit with a lock token. That puts evaluators outside the engine’s trust boundary entirely and lets them scale independently, and the lock token means a result can only be submitted by whoever holds the lease for that unit of work, which closes the hole where two workers both claim the same job and one silently overwrites the other. Seen in AlphaEvolve on Google Cloud.

Seal the evaluator in its own container and withhold the confirmation result. The trusted evaluator runs in its own container, reachable by candidates over one read-only socket. It vetoes on environment drift and peak memory, and checks token, dollar and wall-clock budgets before any GPU work starts. The aggressive part is withholding the confirmation outcome from the proposer entirely, and it is right: if the proposer can see the confirmation result, the confirmation result becomes a training signal. Seen in autocontext.

Hide the metric and strip the candidate of any handle on its environment. The scored metric is a human-written function the proposer never sees, and the candidate itself is compiled into a pure function from inputs to outputs with no access to anything else. The thing being optimised has no way to reach anything except its own inputs, which makes an entire class of shortcut structurally unavailable rather than merely forbidden. Seen in Eureka, which does still execute model-written code unsandboxed, a separate problem you should not inherit.

Give each test its own process, no network, and a stop-then-kill timeout ladder. A separate process or container per test, with the network disabled and a reliability guard in place. The ladder matters because a process that ignores a polite termination request needs to meet a less polite one on a schedule rather than hanging your run. Add memory and CPU limits, which none of the versions I found do, or one runaway candidate takes the host down with it. Seen in OpenAlpha_Evolve, agents and AlphaCodium.

Mask the labels and blackhole the dataset hosts. Mount trajectories read-only, mask the label files, and block the dataset’s home at the network level so the answers cannot simply be downloaded again. That third one is the move people forget: if your data has a public home, a sandboxed candidate with a network connection is not sandboxed. Seen in a-evolve.

Keep the meta-level code out of the image entirely. For a system whose whole purpose is editing itself, the agent cannot modify what is not in the container, so exclude the meta-level code from the build context and run the scorer on the host in fresh containers with the test paths stripped. Deciding exactly which parts are off the table is the entire safety story for a self-modifying loop. Seen in the Darwin Gödel Machine.

That is where the evaluation runs. Keeping it there is a separate job, because most leaks are not holes in the wall, they are things that happen over time while the loop is moving.

Keeping the boundary intact under load

So the question stops being what the wall is made of and becomes what keeps chipping at it. Patches get replayed weeks later. Harnesses get edited by the thing they are supposed to be measuring. A tidy-up script writes into the results directory at three in the morning. Each of these is somebody’s answer to a leak that only showed up around day forty, when every individual rule was still in place and the boundary had gone anyway.

Protect paths twice: after every step, and in every replayed patch. Reverting the protected directories after each agent step is the obvious move. Stripping those same paths out of every replayed patch is not obvious at all, and you only think of it after watching a candidate’s forbidden edit get reverted and then quietly reintroduced when its diff was applied again later in the run. Seen in HyperAgents.

Do not let the agent report a number at all. Let it cite a receipt. The controller spawns the evaluation itself, binds it to a unit ID, seals the output directory and writes a hash-chained receipt. The agent is permitted only to reference the receipt. The inversion is the point: reporting stops being something the agent does and becomes something the agent points at. Seen in EvoMap.

Or more simply: make transcripts the agent cannot write the only ground truth. Same principle, much lighter implementation, and a reasonable first step if a receipt chain feels like a lot of machinery for where you are. Seen in luxas.

Treat a measurement that changed the repository as an error, not a result. A dirty working tree or a moved HEAD after measuring means something happened that you did not account for. If measuring changed the repository, you do not actually know what you measured, and every subsequent measurement is downstream of that. Seen in codex-autoresearch.

Grade in one clean room and merge from another. Grade each candidate in an immutable detached checkout, then re-execute it in a clean detached worktree at merge time. Two separate clean rooms, one for the decision and one for the commit, because the thing you graded and the thing you are about to merge have had time to diverge. Seen in CORAL and Arbor.

Require every reported number to appear verbatim in the file it cites. Before any reviewer sees it. Derived numbers have to be formulas the controller evaluates itself, and a value it cannot trace aborts the run. This eliminates a specific and very common failure where a model reports a number that is approximately, plausibly, and not actually what the file says. Seen in data-to-paper and ARIS.

Wrap runtime monitors around the candidate’s own library calls. They catch fitting the same model twice, ill-conditioned numerics, and file access outside an allow-list. Catching a double fit at runtime is a lovely piece of paranoia, because it is exactly the kind of methodological error that produces a beautiful result and no error message at all. Seen in data-to-paper.

All of that keeps the candidate away from the score. The next set assumes something got through anyway, and asks what the system should do with a number it cannot quite believe.

Refusing to believe the result

The answer is always some version of no, and what varies is the grounds. Some of these refuse on the evidence, some on the sheer size of the claim, some on the health of the thing the claim is measured against. Putting them between a good score and a promotion will cost you a few genuine wins. That is the right trade, because the alternative is a leaderboard built out of the ones that were not genuine.

Let rules veto a promotion and never mint one. Schema checks, an audit of file access against forbidden inputs, coverage checks: all of these can reject a candidate, and detailed results may veto an approval, but nothing there can ever grant one. Promotion requires the measurement. Everything else can only take it away. It is worth writing on a wall. Seen in RD-Agent and ouroboros.

Lock criteria that passed, and tell the model the lock exists. A criterion that passed, was not challenged and did not regress cannot be reworded by the model. Then tell the model the guard is there, and it stops trying. That second half is unusual and I think underrated: telling a model about a constraint it cannot evade saves everyone the budget of watching it try. Extend it to failing criteria too, which is where the temptation to reword is obviously strongest. Seen in ouroboros, which leaves that door open.

Make missing evidence widen the search rather than narrow it. Unverifiable and skipped are never a pass, and routing follows a typed status rather than a truthiness check. Almost every system I have looked at fails open somewhere, usually inside an exception handler, and fails open silently. Seen in ouroboros and get-physics-done.

Refuse improvements that are too large to believe. This is where we came in: a loop that rejects a result more than about ten percent better than the reference, on the reasoning that in this system an improvement that big is evidence of a leak rather than a discovery. A one-sided sanity check catches nonsense. A two-sided one catches cheating. Make it an explicit gate with a logged reason.²²The real check lives in candidate_manager.py and computes abs(loss - standard_loss) / standard_loss, with a comment allowing a delta within 0.1 and no upper limit for the benchmark. Its exception handler returns True, so anything it fails to parse is accepted. The guard fails open, which is the opposite of what a guard should do. Seen in ASI-Arch, where it is buried inside a manager class, and it is easy to convince yourself a check like that is running when it is quietly swallowing its own errors.

Make “scored badly” and “could not be scored” different outcomes. And never let the second one complete silently. This is the smallest mechanism in the whole part and possibly the most valuable. An evaluator that errors and returns zero will reject every good candidate you ever produce, cheerfully, all night, with nothing in the log to suggest anything is wrong. Seen in Opik.

Prove the parent was healthy before blaming the child. Before attributing a failing test to a candidate, check that the parent builds, then re-run that specific test at the parent commit in a throwaway worktree. Without it, the first pre-existing broken test in your repository starts silently rejecting every candidate that touches that area. If a parent does not build, mark everything pre-existing and do not re-inject stale failure lists, which is the trap the naive version falls into. Seen in yoyo-evolve, which calls it an innocence check.

Recompute the headline number from per-sample records, against a fingerprinted set. The evaluator never accepts a headline metric, it derives one, from per-sample records aligned to a frozen and fingerprinted evaluation set. A fingerprint change then means “not comparable” rather than producing a misleading result. Seen in FAROS.

Make a candidate climb a correctness ladder before any speed claim counts. Smoke test, then a shape-by-dtype sweep, then adversarial scales, then bitwise determinism, then odd sizes. Optimising a kernel that is wrong is the easiest speedup in the world, and this is what stops you buying it. One addition: check outputs while timing, because timing on fixed inputs is an open invitation for something in the stack to memoise the answer. Seen in autokernel.

Every mechanism so far is the system checking itself, which has an obvious limit. The last few give up on that and fetch somebody who was not involved.

Bringing in an outsider

What you are buying is independence, and these are graded by how much of it you actually get. A second opinion from the same model family shares all of the original’s blind spots. One from a different family shares fewer. A task whose answer you knew before the campaign started shares none at all, which is why the cheapest mechanism in this group is also the one hardest to fool.

Replicate independently, blind, on a model from a different family. The harness spawns a replication before review, blind to the producer’s files by path scope, on a model pinned outside the producer’s family. That last clause is the good bit. It is not enough for the replicator to be a different process; it has to be a different lineage, because two instances of the same model share the same blind spots and will agree enthusiastically about the same mistake. Count verifier liveness alongside it, or a dead verifier degrades into a rubber stamp without anyone noticing. Seen in luxas.

Mix in sentinel tasks whose answers you already know. Stratified sentinels rotated through every candidate, with stable and borderline items tested differently, plus beacons for attribution. These are positive and negative controls in the laboratory sense: the measuring apparatus is being measured continuously rather than once at setup. Seen in Raven.

Make a required step leave a trace that a later step asserts. Collapse the step into one command plus a sentinel file that a later phase checks for, so skipping it becomes mechanically visible instead of something you have to trust. Any time your process has a step that must happen, the question to ask is what artefact proves it happened. Seen in ResearchStudio.

Bound every external search to the campaign’s snapshot date. So no evidence newer than the snapshot can leak in. This is contamination control for the open web, and it is the only way a literature-searching agent can be evaluated against a period it could not have read about. Apply it to every fetcher rather than just one. Seen in AI-Researcher.

You will notice that the whole part is about one boundary. The next one is about what sits on the far side of it, and the mistakes there are quieter, because a split that has quietly stopped being held out never throws an error.

Which data you’re allowed to measure on

There is one sentence in this part that I would put above every system of this kind: a split that the acceptance rule reads every iteration is a selection set, not a held-out set. It does not matter what you called the variable. If a number from that data influences which candidate survives, that data is part of the search, and it has stopped being able to tell you anything about generalisation.

Keep three splits, not two. Data the proposer sees, data that decides selection, and data that only the final winner ever touches. Almost everyone builds two, uses the second for both jobs, and reports the second as a held-out result. Seen in APE, auto-harness and ADAS, though APE’s own simplified path violates the rule its main path enforces, which is a neat illustration of how easily the convenience shortcut wins.

Seal the test set and make unsealing irreversible. Define the deliverable as the best result on training, then seal the test split, allow it to be touched a fixed number of times, and make each unseal permanent. Making the unseal irreversible is what turns an intention into a mechanism: you get a budget of looks, and when it is gone it is gone, which is exactly how a held-out set should behave and almost never does. Seen in Raven and reef.

Pin the dataset and hash the splits before the first paid call. Pinned revisions, expected split sizes, and a checksum of the split contents, all verified before anything is spent. Datasets change under you, and a benchmark that quietly gained forty items last Tuesday makes every comparison across that boundary meaningless. The hash is how you find out in the first second rather than the last. Seen in reef.

Stratify the split on whether the baseline passed, and store the seed. A seventy-thirty split stratified on the baseline’s own pass or fail means both halves carry a comparable mix of easy and hard items, so a candidate cannot win by drawing a lucky sample of easy ones. One thing not to copy: do not drop the tasks the baseline timed out on, because dropping the hardest items quietly redefines the benchmark in your favour. Seen in auto-harness.

Promote every newly fixed task into a permanent gate. When the campaign genuinely fixes something, on a real verifier pass, that task becomes a gate for every later candidate, and anything that breaks it reverts. Your benchmark grows teeth as the campaign proceeds, which is the single best answer to the whack-a-mole failure where generation sixty re-breaks what generation twelve fixed. Seen in auto-harness and AlphaCodium.

Fix the denominator: a task that did not come back is a failure. With a bounded number of infrastructure reruns allowed first. Otherwise a candidate that crashes on the hard half of the suite posts a superb pass rate over the easy half that survived, and nothing in your pipeline objects. Seen in auto-harness and Raven.

Classify every item as stable-pass, stable-fail, unstable or infrastructure-only. Then exclude the last two from the objective in code, and report the eligible count alongside a pass-at-k estimate. Separating “this item is genuinely hard” from “this item is flaky” is one of the highest-value things you can do to a benchmark, and almost nobody does it because it requires looking at items across runs rather than within one. Seen in agentic-harness-engineering.

Refuse to report at all when validation overlaps training. Never recall validation or test items back into training, and offer a per-task no-regression check alongside. Refusing is the right response to a contaminated comparison; reporting it with a caveat is how contaminated comparisons end up in slide decks. Seen in SkillOpt.

Reject any candidate not scored on the identical seeded set. And score a legitimate task failure as zero rather than letting it crash the run. Both halves matter: a candidate measured on a different set is not comparable, and a crash that takes down the run turns a zero into a missing value, which is much worse than a bad score. Seen in agent-lightning.

Build evaluation sets out of real production logs. Mine actual session logs, scrub the secrets, then use a model to label relevance and apply a rubric. Real traffic is the best evaluation data there is, because it contains the distribution you actually serve rather than the one you imagined. Add seeded splits, deduplication and an explicit privacy policy before you do this, and never top the set up with unrelated records to hit a size target, which quietly changes what you are measuring. Seen in hermes-agent-self-evolution.

Make synthetic examples pass a validator, and feed the rejections back. Rejected samples and the reasons for rejection go into the next generation attempt. That loop is what stops synthetic data generation from producing a thousand variations of the same easy case. Keep a human-owned holdout alongside it regardless, because a fully synthetic evaluation set measures your generator. Seen in promptomatix.

Score an upstream knob by the end result it enables. Not by a local proxy. Retrieval precision is not the goal, answers are, and the two come apart more often than anyone expects. Seen in AutoRAG.

Score a training recipe by actually training with it. Run a small adapter update, measure held-out performance against a no-update baseline in the same request, and have a stateless train-then-evaluate service return the gain and the per-item flips. It is obviously correct and expensive enough that almost nobody does it. The no-update baseline in the same request is what makes it trustworthy. One fix: gate inclusion on the gain being above zero. Seen in SEAL, which keeps the best of five candidate recipes even when that best one is worse than doing nothing, so a bad generation still ships an edit.

Splits keep an honest number honest. But a growing share of what these systems measure is not a number at all, it is a judgement, and that is a different problem with a different set of defences.

When the marker is a language model

Sometimes there is nothing to measure. The output is an essay, a plan, a design, and no test suite is going to grade it. So a model grades it, and you inherit every bias a model has: it prefers the first thing it read, it prefers longer answers, it prefers its own writing, and it will happily give a different score to the same text on Tuesday.

The mechanisms here are all ways of making an unreliable instrument usable. Pair it, blind it, repeat it, constrain it, and never let it promote anything on its own.

Never compare absolute judge scores across calls. Compare within one call, and vote. An odd number of fresh judges each compare before against after in a single call, and the votes decide. This is the foundational move of the whole part: a judge’s absolute score drifts between calls and means very little across them, while its preference within one call is far more stable. Count the votes in code. Seen in darwin-skill, which leaves the counting to prose rules that contradict each other.

Subtract a deterministic penalty from the model’s rubric score. For a serving system the same shape is a hard latency or cost penalty subtracted from judged quality. You keep the model’s ability to assess things you cannot measure, and you stop it trading away the things you can. Seen in autonovel.

Replace fuzzy quality with a handful of binary, observable checks. Three to six of them, non-overlapping, each of which has to pass one test: could two judges agree on this? That question is the whole technique. Run it over your own rubric and watch half the criteria fail it. Seen in autoresearch-skill.

Force a named answer per dimension, and let any structured failure override the summary. The judge must answer each named dimension with a reason of at least a few words, and a structured fail beats a summary pass. Models write approving summaries over critical details all the time, and letting the structured field win is a two-line rule that catches it every time. Seen in yoyo-evolve.

Build the judge’s prompt from measured state, never from the candidate’s claims. Parse the verdict fail-closed and short-circuit deterministically on errors. If the judge reads the candidate’s own description of what it did, you are grading the description. Seen in Voyager.

Randomise which candidate is shown first, and un-swap the result. Or simply judge every pair in both orders. Language model judges have a well-documented preference for whatever they read first, and this is counterbalancing, straight out of experimental design. Seen in prompt-ops and robin.

Shuffle per judge, randomise the merge order, and forbid the critic from proposing the fix. Every revision must also cite the problem it claims to solve. That third rule is the critique-and-rewrite split from earlier showing up as a bias control: a critic that can propose the fix will find the problem its favourite fix happens to solve. Seen in autoreason.

Put a replica pair into every judging plan. That is the same candidate run against itself, alongside a reference-transfer pair, with categorical verdicts mapping to deterministic stop and overfit signals. The replica pair is a beautiful idea: it measures your judge’s noise floor continuously and for free, and if your judge ever prefers one copy of an identical candidate over the other, you know precisely how much of your leaderboard is weather. Seen in prompt-optimizer.

Rotate reviewers so the model accepting a change is never the one that wrote it. A recusal rule, invented independently by people building code agents. Seen in autoresearch.

Withhold the first judge’s verdict from the second, and merge conservatively. Fatal methodological findings override everything else. Showing the second judge the first judge’s verdict destroys the independence you were paying for, and a conservative merge means two judges can only end up more sceptical together than either was alone. Seen in AutoSci.

Make the judge earn the job before you optimise against it. Score judge configurations against human labels using correlation and error metrics, and only let the best-aligned configuration, above a bar, become a campaign objective. This inverts the usual order, where a judge prompt is picked by vibes and then optimised against hard for a week. Seen in Kiln.

Make the scorer as deterministic as you can. Temperature near zero with a fixed seed, on one shared evaluation subset per round. It does not make the judge correct, but it does make it consistent, which means the differences you see between candidates are differences between candidates rather than resampling noise. Seen in future-agi, under an enterprise licence, so reimplement the idea rather than copying the code.

Block promotion when a model judge is the only supporting signal. One sentence, and it is the strongest statement in this part. Judges are allowed to contribute evidence. They are not allowed to be the whole case. Seen in AWorld.

Combine independent runs with a deterministic rule, not another model. Run several independent analyst trajectories on the same data and merge them arithmetically. Using a model to summarise your models is how you lose the independence that made running several of them worthwhile in the first place. Seen in robin.

Turn pairwise preferences into a ranking with Bradley-Terry. If your judge gives you preferences and you need an order, this is the century-old, well-understood way to do it, and it beats averaging scores that were never on a common scale to begin with. Seen in robin.

Add one fixed question to every review: would this still help if this task vanished? A single sentence, aimed exactly at the failure where your system gets very good at the benchmark and no better at the job. I would put it in every rubric I write. Seen in autoagent.

All of that gets you a signal you can partly trust. It does not get you a decision, because the signal still moves when nothing else has, and telling those two apart is its own part of the system.

Deciding whether a win is real

Here is the question this part of the system exists to answer. Your candidate scored 0.847 and the incumbent scored 0.841. Do you keep it?

Nothing you have so far can answer that, because you do not know how much that number moves when nothing changes. The honest answer is that most loops never find out, accept the 0.847, and spend the next six hours building on a coin flip.

Run replicates, and make the challenger beat the incumbent’s best. At least two incumbent and two candidate replicates, grouped by content fingerprint, and every candidate replicate must beat the best incumbent replicate by more than the largest of three things: the observed spread, the declared tolerance, and the minimum improvement worth having. Comparing the challenger’s worst case against the incumbent’s best case is deliberately unfair to the challenger, which is exactly the right direction for the unfairness to point. Seen in codex-autoresearch.

Or compute an actual probability that it is better. Replicate the candidate three times, then require a bootstrap probability of at least 0.95 that it beats the incumbent’s own replicates, rejecting obvious losers early to save the cost. About a hundred and fifty lines, and it gives you a real confidence statement rather than a threshold somebody picked. Seen in autoresearch-mlx, where it is not actually wired into that project’s own loop, though the code is there.

Repeat only the close ones. This is the most economical answer I found to “repeats are too expensive”. Run the candidate once. If the difference from the incumbent is bigger than twice your noise estimate, you are done: accept or reject and move on. Only when the difference falls inside the noise band do you pay for a second seed. Each pair you run gets recorded, and once you have five of them your estimate of the noise locks so the threshold stops drifting. Decisions already made are never reclassified, which is doing more than it looks: an estimate that keeps updating will eventually reach back and change a verdict you already acted on, and then your archive contains decisions made under two different rules with no record of which is which. Seen in AutoScientists.

Or do it properly, with a sequential test and an alpha budget. Matched candidate and incumbent pairs go through a Student-t sequential confirmation with a Bonferroni correction over the number of looks, and the campaign spends a global alpha budget that shrinks geometrically, with a separate held-out lane alongside. The insight it encodes is that peeking at your results repeatedly inflates your false positive rate, and a loop that checks after every candidate is peeking thousands of times a night. Alpha spending is how clinical trials handle interim analyses, and it is the correct tool here for exactly the same reason. Seen in autocontext.

Define disagreement explicitly, and cap the uncertainty a measurement may claim. Two measurements disagree if their ratio exceeds three or their signs differ; otherwise they agree if the gap is within twice the combined standard error, with each error capped at half its own value. A disputed headline number blocks release. The cap is what stops a measurement from claiming such huge uncertainty that it agrees with everything, which is the obvious way this kind of rule gets gamed by a noisy process. Seen in luxas.

Keep the unchanged incumbent in the race as a candidate, and let it win ties. Stop after a set number of consecutive survivals, and require a real margin before anything displaces it. The incumbent as a candidate is a placebo arm: if your loop cannot reliably beat doing nothing, you want that to show up as doing nothing winning, rather than as a coin flip that happens to favour change. Seen in autoreason.

Have exactly one function that decides whether a candidate wins. Every optimiser calls it, and “a tie keeps the incumbent” is written down as explicit policy rather than emerging from whichever comparison operator somebody happened to type. If your codebase has three places that decide acceptance, it has three different acceptance criteria and you do not know which one produced any given result. Seen in Opik.

Never compare against a cached incumbent score. Baseline and candidate are evaluated together, on the same split, same backend, same fresh batch, every time. A number from last Tuesday was measured on a machine, a model version and a dataset snapshot that may no longer exist, and the difference between then and now has a name: drift. Seen in AWorld, agent-lightning and SEAL.

Keep the soft objective and the hard constraints as two separate contracts. A guard failure vetoes the change whatever the metric says. The refinement that makes it survivable is that only a green-to-red transition counts: tests that were already failing, tests that are flaky, and failures in newly added coverage never gate. Without that refinement your guard fires constantly and gets switched off within a week. Protect the check scripts when you build this. Seen in autoresearch and pi-autoresearch, the latter leaving its measurement script agent-writable, which makes the guard advisory at best.

Make an accepted change pass an independent check before it counts as done. Nothing is chosen and certified on the same data. Use a real held-out evaluator for the second step rather than a model judge, which is what one implementation does, and which puts the weakest instrument in the most consequential position. Seen in autoresearch.

Replace one threshold with a stack of named gates. Gain above a minimum delta, no cost regression, no latency regression, held-out verification on at least thirty cases or a replay. Named gates tell you why something was rejected, and a scalar never does. Do not copy the accompanying ranking formula, though, about forty-five percent of which is the model’s assessment of its own work. Seen in AWorld.

Put the stopping policy in one small object. Direction, minimum improvement, patience, target value and the hard guardrail metrics, all together. Making the policy a value rather than a scattering of conditionals means you can log it, diff it between campaigns, and answer “why did this run stop” by reading one thing. Seen in FAROS.

Score degenerate candidates catastrophically before any ranking happens. Too few trades, blow-ups, the strategies that technically post a great number by barely participating: give them a floor value and get them out of the comparison. For a serving system the equivalent is applying quality and reliability floors before you compare latency or cost. Degenerate solutions are what an optimiser finds first, every time, because they are the easiest part of the space to reach. Seen in auto-researchtrading.

State a multi-objective problem as one objective plus constraints, not as weights. Minimise cost subject to quality being at least the baseline minus a tolerance and parse validity above a threshold, with the baseline calibrated on the same data. This is nearly always better than inventing weights, because the constraints are things you can actually defend in a meeting and the weights are not. Seen in SAMMO.

Make every term of the fitness function saturate. Each term is a clamped sigmoid that flattens out at about ten percent of a reference value, so no single metric can run away with the total. A linear combination is always vulnerable to one term going enormous, and saturation is a one-line structural answer to reward hacking that does not require you to anticipate the specific hack. Seen in ASI-Arch.

Separate “it ran” from “it worked”. An executor may close execution facts: the exit code, whether a file exists. A merit verdict needs an independent judge and a durable verdict handle, and a decision closes only by re-reading a fresh artefact bound to that unit and cycle. Those are two different claims with two different evidence requirements, and most systems conflate them. Seen in ARIS and EvoMap.

Count a finding only if it recurs across independent runs. It is the crudest version of everything in this part and it is still better than what most systems do. Seen in Kosmos.

Before promoting the winner, re-run it under several declared variations of the setup. Two to sixteen of them, named in advance. This is multiverse analysis, and it answers the question that should follow every result: is this a fact about the world, or a fact about the particular arrangement of my experiment? Seen in k-dense-byok, which even arrived at the same word the statistics literature uses.

Re-verify a parallel batch after merging it. And fall back to the second-best if the merge loses. Two changes that each help can easily combine into one that does not, and if you only ever verify before merging you never find out. Seen in codex-autoresearch.

At equal score, keep the simpler candidate. It is a regulariser against harness bloat, and over a long campaign it is the difference between a system you can still read and one nobody dares touch. Measure simplicity in code, by size and configuration complexity, rather than by asking a model which one feels simpler. Seen in autoagent.

Every one of those rules costs evaluations, and evaluations cost money. Which raises the question of what happens when the money runs out at two in the morning.

Making the budget actually bite

A budget that only holds when everything works is not a budget, it is a hope. This part is about the failure modes: the crash that loses the accounting, the concurrency that lets twenty workers each spend the last hundred dollars, and the provider outage that quietly eats your clock.

Reserve budget per stage and debit it on completion, including on failure. Generation, gates, evaluation, judging: each stage reserves what it needs. The important cases are the unhappy ones. An exception debits the full reservation, and an interrupted run is archived with its reservation spent. So a crash never buys you free budget, which is otherwise the easiest way for a loop to overspend by an order of magnitude, since crashing is the one thing it does reliably. Seen in AWorld.

Count in-flight work against the cap before admitting more. Committed spend is actual spend plus in-flight proposals multiplied by their mean cost, and no new proposal starts once that reaches the cap; in-flight work simply drains. Failed proposals count towards the total, which is correct and frequently forgotten. This is the concurrency fix: without counting in-flight work, several parallel workers can each independently decide there is room. Seen in ShinkaEvolve, where the cap is unset by default, so set it.

Reserve what a job could cost, not what you expect it to cost. Timeout multiplied by the instance rate, reserved before the job is admitted. It is the only form of reservation that still holds when a job goes wrong, which is the only time a reservation matters. Seen in k-dense-byok.

Check accumulated spend before every proposal, not afterwards. And make the accountant provider-agnostic. Seen in AutoPrompt, whose cap dies silently when you swap providers, which is a particularly cruel failure because swapping providers is exactly when you would want a cap.

When the cap is exceeded, exit the process. Not a tool refusal the model can keep retrying. Not a warning in a log. The process ends. This is the only version of a budget cap that is not, ultimately, a suggestion. Seen in luxas.

Label each budget enforced, advisory or unsupported. Depending on whether trusted telemetry actually exists to enforce it. That is a genuinely mature piece of engineering honesty, and most systems present every limit identically and let you find out which ones were real during the incident review. Seen in codex-autoresearch.

On a usage limit, wait for the reset instead of paying overage. Retry the same iteration after the reported reset window, and abort token streams mid-flight at the cap rather than letting them finish. Waiting is free. Overage is not. Seen in gnhf.

Pause the clock during provider outages. So the budget measures useful work rather than elapsed misery, and a forty-minute outage does not silently convert into forty minutes less research. Seen in RD-Agent.

Make retries cost budget without counting as experiments. Incomplete runs get marked as needing a retry: they consume money but never enter the experiment ledger. Keeping those two ledgers separate is what stops a flaky infrastructure day from looking like a productive one in your records. Seen in Arbor.

Keep separate slot pools for sampling and for evaluation. With adaptive oversubscription and a per-worker cap on pending evaluations. One shared pool means a burst of cheap proposals starves the expensive evaluations, and then you have a queue of unmeasured candidates and no idea which of them are any good. Seen in ShinkaEvolve and CORAL.

Run two timeouts: one per execution and one per node. Budget the node timeout to cover code generation and post-processing, not just the run itself, and make sure a killed child cannot hang the drain loop, which is the classic way a timeout system ends up needing a timeout. Seen in AI-Scientist-v2.

Budgets, guards, gates, ledgers. All of it assumes the process is still alive to enforce them, and at hour nine on a Saturday that is not a safe assumption.

Surviving a crash at hour nine

Nobody builds this part first, and everybody builds it eventually, usually the morning after losing a night’s work to an unhandled exception in a results parser. The mechanisms divide into three: making state that survives, making failures legible, and making the loop resumable at something finer than the whole campaign.

State that survives

The question here is narrow: once the process dies, what is still true? Anything held only in memory is gone, and everybody expects that. The mechanisms below are mostly about the second category, the state that survives but is wrong, because a stale value that gets read back and believed does far more damage than one that is simply missing.

Never cache status. Re-derive it from an append-only log and check invariants on every read. Keeping a candidate implies it improved, and the guard passed, and the repository head is at the trial commit. Any read that violates that invariant is a bug that surfaces immediately rather than a corrupt cache that surfaces in your results three days later. Rebuilding from an event log is more work and it is the only version of this that is actually trustworthy. Seen in codex-autoresearch.

Let the model call exactly one operation, and let code do everything after it. The model says “finish”. Then code performs the scope check, the commit, the measurement, the decision, the revert and the record, in that fixed order. The model gets to say it is done; it does not get to say what happened. That division is the clearest statement of the whole integrity argument, expressed as a control-flow decision rather than as a policy. Seen in that same project.

Advance search state only once the state commit is durable. The on-disk copy is a post-commit mirror written by atomic rename, an aborted step rolls back, and a stale mirror is rejected rather than merged. About eighty lines, and it is the difference between a crash costing you one iteration and a crash costing you the ability to trust anything that happened before it. Seen in reef.

Persist what you are about to do before you do it. Write down the drawn parent and the minibatch before running anything, so a resumed campaign replays the same iteration and seeds reproduce across restarts. With an ask-and-tell interface, late results are accepted and stale ones dropped. Without this, a restart silently redraws, and the reproducibility you thought you had from fixing the seed was never there. Seen in reef and treequest.

Make the search algorithm a value you can save and reload. It exposes a way to load state, produce the next beam, accept a result and serialise itself, with the budget on resume expressed as a maximum number of new items. The controller then leases work in bounded increments rather than handing over control for an hour. Enterprise licence, so reimplement rather than copy. Seen in future-agi.

Put a queue between the optimiser and the workers. The optimiser enqueues task-and-resource-version pairs and reads back traces plus a terminal reward; a gateway puts the rollout ID in the URL path, so the agents themselves need no modification at all to participate. Decoupling at a queue rather than through an interface the agent has to implement is what makes this adoptable in a codebase you did not write. Seen in agent-lightning.

Reconcile continuously towards a desired state instead of reacting to events. Keep the right number of trials in flight, take suggestions from a pluggable service, and recognise exactly three terminal conditions: goal reached, maximum trials, maximum failures. This is the Kubernetes controller pattern and it is the right one here, because a missed event costs you nothing when the loop is continuously comparing what is running to what should be. Seen in Katib.

Reconciling towards a desired state only works if you can tell what state you are actually in, and the hardest part of that is saying what a failure was.

Failures that say what they were

A loop with two endings, worked and didn’t, throws away the one thing you most need the next morning: whether to go and fix your harness or your idea. Everything here puts that back, either by giving a run more ways to end than two or by making each ending carry its own reason. The nastiest failures are the ones that look like a result, and several of these exist purely to stop that.

Lease each step to a single owner, checkpoint within a generation, and return the best result on interrupt. Graceful interrupt handling that hands back best-so-far is a small kindness that makes you far more willing to stop a run, which in turn makes you far more willing to start one. Seen in ouroboros.

On a policy error, withdraw to the previous session rather than continuing. With a checkpoint per step to withdraw to. Continuing after a policy violation is how a small integrity problem becomes a whole corrupt campaign. Seen in RD-Agent.

Snapshot the random state with the archive, and refuse to resume if the environment drifted. Include the evaluator version and the config hash in the fingerprint, not just a name. The failure this prevents is subtle: a campaign that resumes cleanly against a slightly different evaluator produces a results table whose two halves are not comparable, and nothing anywhere says so. Seen in OpenELM and Raven.

Give a run six possible endings, not two. Complete, continue, retry-infrastructure, pause-for-operator, hand-off, exhausted. Failure events carry an owner, a stage, a scope and a repairable flag. Six is about right; one boolean is not, because “it failed” covers both the candidate being bad and your GPU falling over, and those want opposite responses. Seen in AWorld.

Tag failures as build, evaluation or hypothesis failures. The consequences are concrete: an evaluation failure gets retested on the same artefact, a diff that did not apply is marked failed rather than discarded, and a crash never reads as a low score. That last one is the important one, because a crash scored as zero teaches your search something false and it will act on it. Seen in evo, AutoScientists and LLM4AD_Next.

Re-read the current champion before replacing it. Classic compare-and-swap. Without it, two workers that both started when a given candidate was champion will both replace it, and the second one silently erases the first one’s better result. Seen in AutoScientists.

Write the retry count before attempting the retry. Write-ahead logging, applied to a counter, and it closes the loop where a crash mid-retry resets the budget for retrying and you quietly do it forever. Seen in get-physics-done.

Turn a failed commit into the next iteration’s repair task. And refuse to start on a dirty tree. Failure becomes work rather than an exception, which means the loop’s response to breakage lives inside the loop rather than in a try block nobody reads. Seen in gnhf.

Ask the scheduler whether a job is alive, not the job. The scheduler’s own accounting is the only liveness authority, with a bounded grace count for unknown states and a time-limit backstop. Asking the process whether it is alive is not a liveness check. Asking the thing that owns the process is. Seen in auto-deep-researcher-24x7.

Have exactly one reader of stop conditions and exactly one writer of state. One pre-iteration call evaluates every stop condition and returns continue-or-not plus the prompt; one post-iteration call is the only thing that writes run state; a single-instance guard stops two copies running. It is a lovely invariant. Keep the loop driver in a durable process rather than in a UI that can be closed. Seen in Oh-my--paper, which puts it in the UI.

Surviving is one thing. Reaching back into hour four and changing something, without paying again for hours one to three, is another, and it needs the archive to be addressable at something smaller than a whole run.

Making it resumable at a finer grain

Restarting from the beginning is technically a recovery strategy. It is just an expensive one. These mechanisms shrink the unit of work you lose when something goes wrong, from the whole campaign down to a single step, and one of them shrinks the bill at the same time by making sure a resumed step lands back on a prompt the provider has already cached.

Snapshot each run by content and record its commit. A source archive plus the commit hash per run, with worker slots refilled by waiting on the first completion and reconciling against the store. Every run is then reconstructible from the store rather than from a directory somebody may have cleaned up. Seen in OpenResearch.

Give each candidate one commit, inheriting its parent’s. With tags for the baseline, the best and the pre-iteration state, and a patch exported at the end. Using git as the archive means the evolutionary lineage and the code history are the same object, so you get diff, log, bisect and blame over your search for free. Seen in LLM4AD_Next, AutoSOTA and a-evolve.

Make attempt directories immutable, and let declared artefacts outlive a discard. A rejected candidate’s trained checkpoint is often still the best starting point available, so declared artefacts survive the discard and can reseed a later attempt, with gate phases inheriting down the tree. Seen in evo.

Steer a running campaign by writing files the controller polls. Pause, continue, redirect. No API, no dashboard, no new failure mode. You can drive it from a terminal on your phone, and it degrades to exactly nothing if you never use it. Seen in AutoSOTA.

Isolate projects, pool agents per layer, and let idle workers pull. With checkpoint-aware resume. The architecture is right; implement real resource leasing when you build it. Seen in Claw-AI-Lab, whose allocator is never actually called.

Record every model and tool call, keyed by stage, so you can replay it. This turns debugging a fourteen-hour campaign from an archaeological exercise into replaying a tape, and it is the only way to fix a bug in stage seven without paying for stages one to six again. Seen in data-to-paper.

When you redo a step, delete every later step. Obvious, and the alternative is a results directory containing a mixture of two different runs with no marker for where the boundary is. Seen in MLE-agent.

Anchor checkpoints on a prompt hash so a resume hits the provider’s cache. Checkpoint messages every turn, keyed so that resuming lands on cached context rather than paying full price for it again. On a long conversation that is most of the bill. Seen in Arbor.

Everything in that part is about the campaign surviving. The next one is about the result surviving, which is a different and slightly harder problem, because a number outlives the process that produced it and arrives somewhere later with no memory of how it was made.

Being able to prove it later

Three weeks after a campaign, somebody will ask where a number came from. The mechanisms here are the ones that let you answer without re-running anything, and the theme running through all of them is that a record is not a log. A log is what happened. A record is a structure designed in advance to answer a specific question.

Store the hypothesis next to the code and the score. Every trajectory keeps its parent IDs, phase, round, hypothesis, code and feedback, mirrored into a persistent library. Storing the hypothesis is the part that pays off later, because it lets you ask which kinds of idea worked rather than which candidates did, and only the first question generalises. Seen in QuantaAlpha.

Hash-chain the shared knowledge base at every evaluation. Each state hash points at its parent, so you can reconstruct exactly what the system believed at the moment it made any given decision, which is the only fair way to judge that decision afterwards. Seen in CORAL.

Mark provenance edges as observed or inferred. Edges derived from actual filesystem diffs are observed; everything else is inferred, and each step records an environment fingerprint. Distinguishing what you watched happen from what you reconstructed afterwards is the difference between provenance and a plausible story, and almost every provenance system I have seen silently mixes the two. Seen in k-dense-byok.

Tie every reported number back to the evaluator log that produced it. With one correction, which is the same correction as everywhere else in this post: the evaluator has to write that log, not the candidate. Seen in AutoResearchClaw.

Validate the metric contract at write time, not at analysis time. Recording a result requires metric IDs, a direction and a scope that match the baseline contract, so a mismatched result cannot enter the store at all rather than being discovered weeks later. Compute the hashes server-side rather than trusting agent-supplied fields. Seen in DeepScientist.

Give a finding a schema with a slot for the null result. Claim, statistics, methods, notebook path, code-line provenance, null-model result, failure-mode flags, expert-validation fields. The null-model field is the one to notice: the schema has a required slot for what the result would have looked like by chance, and a schema with a required slot is a schema that makes you go and get the number. Seen in Kosmos.

Give a claim a falsification criterion and a way to be withdrawn. A statement, its mandatory conditions, its falsification criteria, proof as a list of experiment IDs, number sources tagged as input, result or pending, and terminal states of refuted or withdrawn. Having refuted and withdrawn as first-class endings means the system can change its mind and leave a trace, rather than quietly dropping claims that stopped working. Seen in Agent-Native-Research-Artifact.

Label every result fully run, partially run, or claimed but not run. Sit with that third value for a second. Somebody built a three-valued enum because they needed a machine-readable way to say “the system reported this result and did not actually run it”, and they needed it often enough to standardise it. It is an enum. It costs nothing. It turns an entire category of silent failure into something that cannot be reported as a result. Seen in spark-to-paper-skills.

Give each revert a machine-readable class, and close it when it is resolved. “No progress” and “too large” demand opposite responses, so the class has to be readable by code. And the receipt closes once the issue it warned about is fixed, which is the design goal stated in the code itself: memory should hold warnings, not history. An open warning is guidance; a closed one is clutter, and the difference is what stops memory filling up with obsolete cautions. Seen in yoyo-evolve.

Record a negative result with a promote flag. The trigger, why it was wrong, the counter-action, a category, a confidence, and a flag saying whether this should become a standing rule. That last field is how a one-off lesson graduates into policy, which is the mechanism by which a loop’s experience turns into a loop’s behaviour. Seen in get-physics-done.

That promote flag is the hinge into the next part, because a record that changes future behaviour has stopped being a record and started being memory.

What the loop remembers

Memory is where most of these systems go wrong slowly. The failure is not dramatic: it is that by generation two hundred the loop is carrying four hundred lessons, half of which contradict each other and most of which were learned under conditions that no longer apply. Then the lessons start costing more than they are worth, and nothing in the system is capable of noticing.

So the good mechanisms here are almost all about the unglamorous half of memory: scoping it, deduplicating it, attributing it, and letting it die.

What a lesson actually is

None of that is possible until you decide what a lesson is, and the answer nearly everyone reaches for first is a sentence. A sentence cannot be scored, scoped, deduplicated or retired. It can only pile up. The systems that avoid the slow rot mostly avoid it right here, by giving the thing enough structure that something other than a sympathetic model can handle it later.

Make a lesson an object, not a sentence. Evidence references, target scope, generality, confidence, occurrence count and affected cases, deduplicated and re-extracted each iteration. Once a lesson has a scope and a generality field you can decide whether it applies here, which a free-text note can never support. Seen in AWorld.

Type each note, scope it, and say why it matters. Semantic, procedural or episodic; global or project-scoped; with its evidence, its source session and a hash for deduplication. It is written by a post-run worker that sees a sanitised trajectory and treats tool output as data rather than as instructions, which is a security property most memory writers do not have and should. Seen in EvoScientist.

Let notes point at each other as complements, contradictions or replacements. And make the replacement relation actually demote the superseded note at retrieval time, or you have a beautifully typed graph that still returns the obsolete note next to the one that replaced it. Seen in EvoScientist, which does not do the demoting.

Deduplicate lessons by a stable key, and promote them up a ladder. The key is something like area plus symptom, carrying a recurrence count and first- and last-seen timestamps. The ladder runs raw log, to always-loaded rule, to on-demand skill. The ladder is the good idea: not everything a system learns deserves to be in every prompt, and a promotion path lets frequency decide which things do. Seen in self-improving-agent.

Package expertise as a versioned folder, not a paragraph. Name and description frontmatter, a body, optional scripts, versioned as a git branch, with a ledger of outcomes. Versioning skills as branches means you can diff them, revert them and measure them, which turns “the agent learned something” into an artefact a person can review. Seen in EvoSkill.

Store the code but embed the description. Retrieve the top few descriptions into the prompt while the whole library stays callable, and assert at startup that the vector store matches its manifest. Embedding the description rather than the code is right, because you retrieve by what a thing is for and not by what it looks like, and the startup assertion catches the silent disaster where your index and your library have drifted apart. Seen in Voyager.

Deduplicate successes and keep every failure. Completed tasks are collapsed; failed ones are not, so the proposer can see that something has failed nine times rather than that it has failed. One line in a deduplication routine, and it changes what the system believes about the world. Seen in Voyager.

Every mechanism so far adds to memory. The hard half is the other direction.

Making memory forget

Forgetting is unglamorous and nobody demos it, which is roughly why memory stores rot. The rule worth stealing from this group is that an item earns its place by being useful rather than by being recent. Age is trivial to measure and tells you almost nothing. Usefulness is harder to measure and tells you everything, so most of the work below is in making usefulness measurable at all.

Count how often each memory item helped and how often it hurt. Store each item with a helpful count and a harmful count, and have the generator report which item IDs it actually used, which gives you per-item attribution almost for free. Then act on the counters. Seen in ACE, whose curator only ever adds, so the counters accumulate as decoration.

Rank patterns with decay by usage, not by age. A lesson that keeps getting used is still true; a lesson nobody reaches for has quietly expired. Usage is the right axis, and it is the one you already have data for. Seen in Opik.

Compact related lessons into families that decay together. Families rather than individual items means the compaction preserves the shape of what was learned instead of dropping one half of a pair of related cautions and keeping the other. Seen in codex-autoresearch.

Curate on new deltas rather than recomputing all pairs. Add, update, delete or do nothing, decided per new item. Once your memory has a few hundred entries, an all-pairs curation pass is the single most expensive thing in your loop and it runs constantly. Seen in InternAgent.

Record the method and the reasoning, not the rejected text, and clear it when the context changes. Both halves are good. Storing the reasoning rather than the artefact makes the lesson portable, and clearing on a context change stops a lesson learned about one situation from being applied to a different one forever. Seen in AdalFlow.

Lock the store, version the updates, and protect the newest entry from eviction. A file lock around read-modify-write, revision-checked optimistic updates, an inactive status instead of deletion, and a budgeted eviction that never removes the most recent item. Protecting the newest entry is what stops a cold-start eviction policy from immediately deleting the thing you just learned. Seen in infiAgent.

Only allow a retry if the last failure taught you something new. And escalate a recurring “same blocker” lesson into re-scoping the whole campaign. This is the most sophisticated stopping rule in the entire catalogue and it is still, mechanically, a set membership test. It also states the right principle: a retry that cannot teach you anything new is not a retry, it is a repetition. Seen in ResearchStudio.

Keep a deterministic results table plus an explicit list of dead ends. Injected into planning, with model synthesis reserved for phase-level promotion only. Keeping the facts deterministic and the summarising rare is the right split: the results are results, and only the interpretation needs a model. Seen in ML-Master and auto-deep-researcher-24x7, the former never actually storing its cross-task layer, so build the write path.

Store a failed direction with the conditions it failed under. And distinguish its failure scope: “could not run within budget” is a completely different fact from “ran and underperformed”, with support and contradiction counts alongside, and condition mismatches penalised at retrieval. That distinction is nearly always lost, and it matters, because an idea that failed for lack of budget has not been tested at all, and filing it next to the ideas that were tested and lost means you never come back to it. Seen in NanoResearch.

Scoping and decay keep memory from rotting. The last group is about the point where memory stops being a reference and starts steering the loop.

Memory that changes behaviour

A store nothing reads is a diary. The gap is nearly always in the same place: writing is automatic, reading is one hopeful sentence in a prompt, and nothing in between guarantees that what got written is what comes back when it matters. These close that gap from both ends, by making what is stored addressable and by making the thing that reads it part of the loop rather than part of the prompt.

Treat the prompt context like a deployment artefact. Candidate context bundles are content-addressed and kept separate from the serving context until promoted by an atomic pointer swap that can roll back. Staging and an instant rollback for your context is obviously correct and almost nobody does it. Seen in autocontext.

Rebuild the working summary from the ledger, never from the model’s recollection. Take the last several persisted runs with their hypothesis, hint and rollback reason, and reconstruct the summary from those, with a full on-disk backup before compacting at a context threshold. Summarising a summary is how a long campaign loses its own history, one paraphrase at a time. Rebuilding from the ledger each time means the summary is always one step from ground truth. Seen in pi-autoresearch and freephdlabor.

Put a mutable belief layer on top of an append-only journal. The journal holds verbatim before-and-after revisions; promotion into the belief layer happens only on closure signals; refutations become first-class dead-end nodes. You get a system that can change its mind and still show you exactly what it used to think and why it stopped. Seen in Agent-Native-Research-Artifact.

Give ideas, experiments and methods forward-only state machines. With prefixed failure reasons acting as a ban-list, edges for supports and invalidates, and context compiled per purpose with section budgets. Forward-only states mean an idea cannot quietly return to being promising after being refuted, which is otherwise a very easy thing for a hopeful system to do. Seen in AutoSci.

Require human approval before a lesson becomes a rule. Recorded in a change log. There is a real distinction between things the system noticed and things the system is now bound by, and putting a person in that specific gap is cheap. Seen in spark-to-paper-skills.

Commit the knowledge file next to the thing it describes. Directory-scoped, so the knowledge moves, renames and gets deleted along with its subject. It solves staleness by construction rather than by curation, which is the only way staleness ever actually gets solved. Seen in autoresearch.

Carry a running context file across generations of the artefact. Attach scores and repeat counts to each entry when you build it, because a lesson with no measurement attached is an anecdote. Seen in SIA, which passes only generation numbers.

Keep a cookbook of symptom-to-intervention recipes. Label it as human-curated, version it, and ablate it separately from the autonomous search, or you will eventually publish the cookbook’s performance as the loop’s. Seen in SimpleMem.

Retrieve failure traces through a graph typed by component and error. Rather than by task similarity, with success defined as passing every gate. Do not key on exact task strings and do not inject unbounded history, which are the two ways this particular design goes wrong. Seen in RD-Agent.

Sweep the transcripts for failure evidence at session end. With secret redaction and an opt-in gate. Most of what a system learns is sitting in its transcripts and is never extracted, because extraction is a job nobody schedules. Seen in self-improving-agent.

Sixteen parts down, and every single one of them has been about making the loop’s output trustworthy. Which leaves the question nobody asks, and it is the one I would build first.

Measuring the researcher itself

Here is a thing that took me embarrassingly long to notice. Every part above measures candidates. Almost nothing measures the loop.

You will have a leaderboard of results and no idea whether your search is any good. And those are different questions, because a loop that proposes brilliant ideas and picks the wrong ones looks identical, from the outside, to a loop that proposes rubbish and picks the best of it.

Print what you selected next to what perfect selection would have scored. You already have both numbers. You evaluated every candidate, so you know which one was best; you also know which one you kept. The gap between those two is the most diagnostic number in your entire system. A large gap means your proposer is fine and your selection rule is throwing away winners, so go and fix acceptance. A small gap with a low absolute score means your selection is working perfectly and there was nothing good to select, so go and fix the proposer. Without this number you are guessing, and the guess is usually wrong, because a bad picker and a bad proposer feel exactly the same from where you are sitting. Seen in OpenRSI, the only project I found that separates them.

Keep pass and fail counts for every gate. Persisted in the checkpoint, so you can answer a question every mature system eventually needs to ask: is this gate selecting anything, or is it just burning budget? A gate that has never rejected anything is not protecting you. A gate that rejects everything is not either. Seen in AdalFlow.

Test your implementation against the reference implementation. A development-only test that runs both and compares, to back any claim of algorithmic fidelity. If your code claims to implement Thompson sampling, this is what makes that a testable claim rather than a comment. Seen in reef.

Move the optimum somewhere the literature has never put it. This is my favourite mechanism in the whole catalogue. When a model-driven optimiser beats a classical baseline on a standard benchmark function, there are two possible explanations, and only one of them is “it searched well”. The other is that it has read the paper. Rastrigin’s minimum is at the origin; every model on earth knows that. So before trusting the result, re-run on shifted and rotated bounds with the variable names changed, and see whether the search still finds it. That is the cleanest de-confounding move in this entire post, and it exists because contamination in this field is a different animal from contamination anywhere else. A drug cannot have read about the trial it is in. Seen in Optim-Agent.

When the loop starts editing itself, make it name its integration point. Meta-level changes go through explore, critique, specify and generate, must state exactly where the new code hooks in, and get A/B tested old runner against new on the same seeds in a sandbox. Naming the integration point is the requirement that matters, and one project proves why: in its own artefacts, the generated mechanisms were never actually called by anything. The loop improved itself into code that did not run. Check that the integration point genuinely executes. Seen in Bilevel-Autoresearch.

And if you let a model rewrite the selection policy, keep it in a sandbox. On stagnation, a model rewrites the selection rule itself. It is the most ambitious mechanism here and the one I would be most careful with: strict sandbox only, never in the durable controller. A self-modifying selection policy that can touch your controller is a system that can lose the ability to tell you it has broken. Seen in skydiscover.

That is the last of the seventeen parts. Which leaves the only question that actually matters on a Monday morning.

Where I’d start

If you have read this far you have a very long list of things you could build and no night is long enough. So here is the honest ordering, which is not the order I have presented them in.

Build the scoring-health guard first, because it is four lines. Make “the candidate scored badly” and “the evaluator could not score anything” different outcomes, and make the second one impossible to complete silently. An evaluator that errors into a zero will reject every good idea you ever have, and it will do it cheerfully, all night, with nothing in the log.

Then print selected against oracle, because it tells you what to build third. It costs one extra column and it is the difference between engineering and guessing.

Then the lazy repeat rule, because it is the cheapest way to stop acting on noise: repeat only the close ones. And the admissible bound, because it pays for the repeats.

After that, the rule the whole thing rests on: do not send the answers. Everything else in the integrity part is a patch on the cases where that was not possible.

None of this is hard, which is the part I keep coming back to. There is no breakthrough in this post. There is a set of small, boring, mostly subtractive rules that a few hundred people each learned the expensive way, one incident at a time, and then wrote into a file in a repository nobody reads. The mechanisms are not the scarce resource. Knowing which one you needed, before the night you needed it, is.

Now you do.

Sources

Every mechanism above is linked, at the point it is described, to the repository it was found in. There is no separate bibliography because that list would be the post again: the source for each mechanism is the code, and the link on the project’s name is the citation.

Two things worth saying about those links. First, I read the code rather than the README, which is why a fair number of entries carry a correction: the mechanism as described here is the working version, not always the shipped one. Second, none of these repositories are stable objects. Something I describe may have been refactored, fixed or removed since; if you are borrowing, read the file before you trust the paragraph.