The Routine Runs Itself

I am becoming a robot.

I work on tasks in stages, and I kept doing the same ones in the same order. Over time I could feel the fatigue of asking AI to do the same thing over and over, just for a different issue. So I built a pipeline that does the routine for me.

This post covers how the pipeline is shaped and why. I’m not going to go through each skill’s prompt. They’re in the repo if you want to read them.

What I used to do by hand

1. I’d come up with an idea and ask Claude to write a GitHub issue describing it, plus the goal (the definition of done).
2. In a new chat, I’d ask an agent to analyze the issue and find gaps. This is where the open questions show up, and they’re the ones that need a human decision.
3. I’d feed the findings back and have it write a phased plan. I wanted phases because I wanted the work broken into small units.
4. Each unit of work got its own agent, so every agent starts with a small, scoped context instead of a long chat.
5. After the implementation, I’d run tests and a code review.

Note: Phases run one after another. Maybe I’ll find a way to let an agent decide which phases can run in parallel, but I haven’t yet.

This is a common workflow in software development, and nothing about it is special, but having to type it out again and again for every issue can be very tiring.

The pipeline

Now I run one command:

pipeline run "Add a dark mode toggle"
pipeline run --issue 42
pipeline run --resume add-a-dark-mode-toggle

Every stage is a fresh agent run: a separate claude process with its own model, turn cap, and budget. It never sees another stage’s chat. What it gets is the earlier stages’ output, saved on the GitHub issue (the issue body, then one comment per stage for the analysis, decisions, plan, phase summaries, test report, review, and PR text) and handed over as input files. It also gets the code on the branch, as the earlier stages left it. Analyze, decide, plan, and review run on Opus, while issue, implement-phase, test, fix, docs, and revise run on Sonnet. Those four are the judgment calls, the ones where a stronger model earns its cost: spotting the gaps, making the decisions, shaping the plan, and critiquing the result. The rest is execution where Sonnet would be perfect for.

The test stage runs before the review, so the reviewer reads a test report along with the diff.

It still asks me

As you can see from the diagram, “one command” is only partly true. The run asks y/N after every stage, and two gates wait for me.

The question gate comes after decide. The decide stage sorts every open question from the analysis into one of three groups: (1) decided from the code, (2) decided because it’s low stakes and easy to undo, or (3) escalated to me. Anything touching a public API, a data model, security, or cost gets escalated. The run won’t continue while an escalated question has no answer. I answer in the terminal: either type an answer, r to accept the recommendation, or Enter to skip.

The sign-off gate comes after docs. I read the PR text and the changes. If I approve, the draft PR is marked ready, but merging is still on me. If I say no, the pipeline asks what should change, and that starts a revision (next section).

What if I don’t like what it built

Reading the result is usually when I find out what I actually wanted because a feature can match the issue and still not be what I had in mind.

Now when I decline at the sign-off gate, the pipeline asks “What should change?” and I answer it. I can also run it later on a finished run:

pipeline revise add-a-dark-mode-toggle "what should change"

This starts a revise stage. It’s a fresh agent like the others, and it gets the issue, plan, test report, review, PR text, and any earlier revisions. It turns my feedback into acceptance criteria, writes a failing test first, then makes the smallest change that passes. It only applies the feedback. If my feedback contradicts a criterion in the issue, the feedback wins and the summary lists that criterion as superseded. If the feedback is too vague, it stops and says what’s missing.

After the change, test and review run again, checking the revision’s criteria along with the original ones. Then docs rewrites the PR text to match the new code, and I’m back at the sign-off gate. Each revision is saved as its own comment on the issue (revision-1, revision-2, and so on), so a second round knows what the first one changed.

Sometimes my feedback isn’t only about the one thing I pointed out. The agent can recognize the pattern behind it and propose a new convention, and the pipeline asks me whether to keep it. For example, if my feedback is about UX, an accepted convention is added to the repo’s UX_CONVENTIONS.md, which some of the stages read. That way I stop pointing out the same problem, because later runs already follow the rule.

Why GitHub comments

I used to manage the markdown files by hand. Too many of them made it hard to keep track of things, and the clutter added to my cognitive load. Now with the pipeline, notes are kept as comments on the issue instead of as markdown files in the repository. It tags each comment so it knows which is which, and the pull request ends up with only the actual code changes.

There’s a caveat, though: GitHub caps a comment at 65,536 (2¹⁶) characters, and the run stops with an error if a stage goes over. I haven’t tested how comments hold up for long conversations or small context windows.

Checking the agents’ work

Fresh agents are only useful if I can trust what they hand over, so I put checks around every stage.

• Replies are checked before they’re posted. The analysis has to list its open questions, and the decisions have to answer every one. Each phase in the plan needs a check that runs on its own, and every acceptance criterion has to point to a phase. If an agent cites a spot in the code, like src/auth/login.py:42, the pipeline checks that the file and line exist. A reply that fails goes back once with the list of problems. If it fails again, the run stops.
• The reviewer can’t edit files. It can only read and report. It opens with a clear pass or fail and labels each finding Critical, Required, or Nit. A fail triggers one fix round, then test and review run again. If it still fails, the run stops and waits for me.
• The test stage breaks things on purpose. It makes one small edit that should break the new behavior, runs the test, and undoes the edit. If the test stays green, it wasn’t covering that behavior, and the stage reports a failure.

The skills also carry a list of excuses to reject. My favorite is in the review skill. The excuse is “The tests pass, so it is correct,” and the reply is “Tests cover only what they test. Check the diff.”

A real run, from start to merge

Here’s one from start to merge: issue #265. The landing page showed no answer until you signed in, so the idea was to show one saved, real answer to signed-out visitors. Every stage’s output is a comment on that issue, so you can read it in order.

• Question gate. Decide sorted 11 questions and escalated 3: which production thread to show, what happens when its saved receipt goes stale, and whose thread can be shown publicly. That last one is a privacy call, so it came to me. I took the recommendation on all three.
• The agent stopped instead of making something up. Phase 2 needed a saved copy of a real answer from production, which only I could capture. The agent didn’t make one up. It stopped and listed what I had to do. I captured it, commented on the issue, and resumed the run.
• Review failed it. Seven of the 8 citations had no highlighted passage, and a doc change skipped a required step. The fix stage repaired the doc. I accepted the citations as they were and said so in a comment, and the second review dropped that finding.
• Sign-off gate. I read the PR text, approved it, and merged the PR.

Total: $5.81 of the $15 budget, over 3 phases. The budget is checked before each stage, and the run stops if it’s over.

What I measured

Every time I change a skill, I want to know if I made it better or just made it different. So I have evals for the stages I trust least: decide, plan, and review. They run the real skill on the real model, which costs money (about $2.50 for the review set), so I don’t automate them. I run them by hand.

None of them use a model to grade. Each test case comes with the right answer written down ahead of time, and the score is a plain check against it.

For decide, the question is whether it escalates the right things. Say an issue has two open questions: what columns should a public CSV export have, and what should an internal helper file be called. The first one has to come to me, because other people’s scripts will depend on it. The second one doesn’t. If decide gets either wrong, the case fails.

Review is harder to score, because on a real change I don’t know ahead of time what bugs are in it. So I make changes where I do know. Each case is a tiny repo with an issue, a change that’s supposed to solve it, and a bug I planted on purpose. Since I put the bug there, I know exactly which line it’s on, and I can check whether the reviewer found it. I also write tests that pass even with the bug in place, because in a real run the change that reaches review usually has passing tests too. That way the tests can’t give the bug away, and the only way to find it is to read the code.

For example, one issue says customers get charged twice when they double-click Pay, so the button needs a loading state while the order goes through. The change I wrote makes the button look like it’s loading, and the test checks that it does, so it passes. But the button still works while it’s loading, so you can still pay twice. That’s the planted bug.

The reviewer only gets credit if it fails the change and points at the line with the bug. The other cases include an SQL injection, an off-by-one error (where a loop skips or repeats one item), and a one-line bug buried in a long diff. There’s also one case with no bug at all, to check that the reviewer doesn’t invent one.

What the numbers say. Less than I’d like. The current review skill catches all nine bugs. The older version, before I added checklists, scored 8 of 9, but on its one miss it did find the bug; it just wrote the answer in a format the pipeline couldn’t read. Decide and plan passed every case both before and after my changes.

So, did the changes make the reviewer better at finding bugs? I can’t say yet. The evals tell me I didn’t break anything, but they can’t tell me I improved anything, because no case trips up either version. For that I need harder cases, and the best place to get them is the next bug it misses on a real issue.

What’s next

1. I don’t want to approve every plan, implementation, and review myself. Another agent could do it.
2. I want to work without looking at my laptop. The pipeline could message me on Telegram when it needs an answer, and I’d reply from my phone.
3. It shouldn’t depend on Claude. An agent in the pipeline is a class with one method, run(prompt, model, budget), and right now only ClaudeAgent exists. I’d like to try an open-source harness with open-weight models behind it.