What if AI could suggest fixes to itself?

In my homelab chatbot, a “skill” is a written workflow: instructions for each stage, the tools each stage may call, and how the stages connect. When a run goes wrong, the user can flag it. A reviewer skill then reads the run and returns a diagnosis with proposed changes.

The tempting next step is to apply the suggestion. The diagnosis often sounds right. It may well be right.

That leaves a small question with a useful answer: what has to be true before an AI-suggested fix is allowed to change the AI?

A diagnosis is a proposal, not a patch

The reviewer returns structured output: what went wrong, the proposed edits, a change note, and its own confidence score. None of that touches the live skill.

The proposal is first validated against a frozen copy of the current skill. Only the wording can change: stage instructions, the synthesis prompt and the description. Stage identity, order, the dependency graph, tools, delegation and governance settings must stay exactly as they were. A proposal that tries to add a tool or reorder the workflow is rejected as invalid, whatever the diagnosis says.

That boundary matters more than the quality of the diagnosis. A reviewer that can grant a skill new permissions has become a way to escalate privilege. Limiting the edit to wording keeps the review loop inside the same permission model as everything else.

Both the frozen baseline and the edited candidate are recorded with content hashes. From here on, every step refers to those exact versions.

Decide what “better” means before the comparison

A fix is only better against something. The skill owner writes a review suite: synthetic conversations, each with expected outcomes that are checked against stored records rather than the wording of the reply.

The supported checks are deliberately plain. Did the number of transactions change by exactly one? Do the recorded amounts total the expected figure in minor currency units? Was the write tool called exactly once and confirmed? Did anything change that the case did not allow to change? Cases are split into development and held-out partitions, and a suite has to be approved before it can be used.

The comparison also declares its intended benefit in advance. Either the candidate must pass checks the baseline failed, or, when the baseline already passes everything, it must reduce a chosen cost such as tokens or tool calls by a stated percentage. Choosing the measure after seeing the results would let any change look like an improvement.

Run both, on the same cases

Each case runs twice: once with the baseline, once with the candidate. Both runs use the same suite, the same case definition and the same prepared test data, under separate run identities. Before each run, the test records are allocated exclusively and read back to confirm their starting state. Afterwards they are read again and quarantined, so a dirty fixture cannot feed the next comparison.

The outcome is one of four answers:

Candidate supported. Every mandatory check passed and the predeclared improvement was shown.
Keep baseline. Nothing broke, but no predeclared improvement appeared. Ties go to the version already running.
Reject. A mandatory check failed, or something the baseline did correctly stopped working.
Inconclusive. Evidence was missing, stale, or a tool call ended in an uncertain state.

Only the first answer can lead to publication. The last one matters most to me: an unknown result is never counted as a pass. The report also states what it cannot control: model sampling is not identical between runs, latency depends on conditions, and costs come from recorded usage rather than a reconciled bill.

Publish the exact thing that was tested

Between a passing comparison and a click on “publish”, a lot can change: someone edits the skill, a dependency moves, the suite is revised, or the evidence ages out. So publication checks everything again inside the same database transaction: the live skill still matches the frozen baseline, the candidate hash matches what ran, the suite and test environment are unchanged, and the supporting evidence is still within its validity window.

Only the skill's owner can make that decision, and only with review, publish and edit permissions together. System skills are excluded entirely; they change through reviewed database migrations. The published change is appended to the skill's version history with the reviewer's change note.

Test runs on deterministic fixtures exercise the machinery end to end, and the system labels them as simulations. Fixture evidence cannot qualify a real publication, because a scripted model shows that the plumbing works, not that the new instructions help.

What the loop is allowed to remember

After a decision, the system can propose a lesson for future reviews. The lesson records the observed effect as helped, hurt, no observed effect, or confounded when more than one edit changed at once. Its confidence is always labelled observational, and its text says plainly that the association does not establish causation.

Only a single-edit change that helped can be approved as guidance, and approval is a separate human decision from publishing the fix. A lesson applies only to the exact skill definition and dependencies it was observed on. If those change, the lesson has to be earned again.

Where this stops

This is a mechanism, not a result. The review suites currently target one service, a personal spending tracker, and the cases are synthetic conversations that I wrote. A suite only checks what its author thought to check. One matched comparison is a single observation rather than a statistical study.

By project policy, CI and the test environment make no calls to real model providers, so the automated evidence covers the mechanism with deterministic fixtures. This piece reports no before-and-after numbers from real model comparisons.

Why this belongs in an AI operating model

My background is quality and delivery, and this loop is regression testing applied to instructions: a defined baseline, a change under test, a suite agreed before the run, and a release decision that names its evidence.

AI makes the proposal cheap. It does not make the proposal trustworthy. The model can suggest how to do better. The system still has to show that it did, before anything changes.

Based on a review of the skill-review code, schemas and test definitions in my personal homelab chatbot project, merged in October 2026. The spending example is illustrative of the supported checks. No real-model comparison results are reported, and no independent audit was performed for this article.