Why a one-shot agent annoys everyone
I’ve spent enough time babysitting AI coding agents to recognize the pattern: the first time you turn loose an automated reviewer or a triage bot, it feels like a miracle. The fifth time, you want to mute it. A prompt that nails eighty percent of a task looks smart on day one — the same lesson the agent harness comparisons keep hammering home, but that missing twenty percent shows up over and over — and costs more than people admit, as I dug into with the hidden cost of AI agents. Every wrong label, every useless comment, every email that should have been filed but wasn’t.
Warp, the AI-powered terminal and agentic development environment, hit this square on. Their internal code review agent was making comments their own engineers found unhelpful and low quality. The obvious fixes — hand-rewriting the prompt, fattening up AGENTS.md — helped a little and then stopped scaling. What finally clicked was reframing the problem. Feedback to an agent died when the session ended, so the agent never had a chance to learn from its own mistakes.
Their answer is worth stealing because it’s genuinely simple to reproduce. It’s called a self-improvement loop built on skills, and it’s the closest thing I’ve seen to an agent that quietly gets better every week.
The pattern: two skills and a human in between
Before we build anything, here’s the architecture. It’s just two files doing the talking, with a human feedback step sandwiched between them.
The inner skill holds the domain knowledge. It’s what your agent actually runs when it does its job — say, triaging an issue, reviewing a pull request, or drafting a patch. It reads like instructions to a smart colleague, not like a computer program.
The outer (improver) skill is an observer. It runs on a schedule instead of per task. Its job is to pull the accumulated human feedback, compare what the agent suggested against how people actually corrected it, and propose a small, focused edit to the inner skill. That edit becomes a file diff, so it lands like any normal code change.
Zach Lloyd, Warp’s CEO, describes the elegance in his write-up on the Claude blog: the framework is simply a domain-specific skill, plus an improver skill that refines it. Because skills are plain files, agents are unusually good at updating them — and because the update is a diff, it flows through your usual review workflow. A human approves it, it merges, and the next run of the inner skill inherits the improvement.
Step 1: write the inner skill like you’re teaching a person
Don’t open a text editor and start dumping rules. The Warp team’s first piece of advice is to write principles, not rules. “Construct the skill as though you’re instructing a smart person, not like you’re programming a computer,” Lloyd says. A line like “look for repeated code” beats an exhaustive list of variable-naming conventions.
The second tip is to explain the why, not just the what. When a skill says why a rule exists, the agent can reason about edge cases instead of mechanically applying a checklist. That’s what makes it generalize.
For my test, I built a triage skill with three buckets: ready-to-implement, duplicate, and needs-info. Nothing exotic — just the domain knowledge an issue-sorting agent needs. The exact same trick works for code review, bug fixing, or incident response.
Step 2: make feedback effortless to give
A self-improvement loop is only as good as the signal feeding it, and the signal dies if people have to go out of their way to supply it. Warp’s rule is to capture feedback where people already work — commenting directly on the PR or the issue — and to make it automatic, with no extra submission step. “Low friction is what keeps signal flowing,” Lloyd notes. Make it hard and nobody bothers.
Here’s the twist on quality: a small amount of detailed feedback from a senior engineer is worth more than a flood of thumbs-ups. Binary up or down doesn’t tell the agent anything; a comment that says “you suggested renaming this variable, but our convention for a global is this particular naming context” teaches it something actionable. Volume helps, but specific quality wins.
Step 3: run the improver to turn feedback into a diff
The full walkthrough, including the sample repo, is on Warp’s blog. This is where it gets mechanical. On a schedule — Warp runs cloud agents on a loop; you could equally use a cron job — the improver pulls every record of past runs, reads back the human corrections, and makes the smallest edit that captures them.
I tested this locally with the real mechanics. I kept a feedback log in JSONL with two corrections where a human had moved an issue from ready-to-implement to a spec bucket, explaining in both cases that the problem was real and actionable but the UI shape wasn’t defined yet. My improver script read the log, correlated the repeated signal, and proposed an edit adding that missing bucket to the base skill. The whole thing came back as a clean, reviewable diff I could eyeball before approving.
That diff-is-the-loop point matters more than it sounds. Because the improvement is a file change, it goes through the ordinary PR flow — description, review, approval, merge. Nothing changes about your process; the agent just inherits a better skill on the next run. A human’s judgment stays in control of what actually changes about the agent’s behavior.
Step 4: don’t let the agent accept feedback blindly
Assume some of the feedback will be wrong, because it will. Warp’s guidance is to give the improver context to sanity-check the signal, filter whose input counts, and keep a human in the loop at either the filtering or the final-review stage. A rogue comment shouldn’t silently rewire your review agent’s judgment.
That connects to a question I get a lot: is a skill the same thing as memory? Not really. Skills are procedural and stable — they’re “how to do X,” changed deliberately, and run-agnostic. Memory is auto-written during inference and never stops changing. Mixing the two is how agents drift. Keep the deliberate knowledge in skills and let the volatile context live in memory.
Step 5: prove the loop is actually improving
If your domain is verifiable, build the verification harness first, then let the agent tune against it: generate a reference corpus, compare output to the reference, fix, repeat. If it isn’t verifiable, lean on deterministic evals against golden outputs wherever they exist, and where you must rely on human feedback, restrict it to domain experts instead of opening the floodgates to anyone.
And whether or not you have an automated grader, track the metrics humans already happen to look at — time to merge, contributor count, cost. Feed those back into the improver. If the loop is working, those numbers trend in your favor even before anyone notices an individual agent got smarter.
Scale it with discipline
A templated base loop captures the overlap across all your agents, with domain-specific weights layered on. A handful of improver agents can each own one loop; if you’ve scaled to a hundred, they should be sharing a common one rather than each reinventing it.
Warp now runs this across its entire open-source repository, with separate spec-writing, review, and triage agents each carrying their own self-improvement loop. The whole thing stays reviewable because the improvements are just files.
I’m a sucker for systems that compound quietly, and this is one of the rare agent patterns that actually does. You don’t need a fleet of model trains or a bigger model to get a better assistant — you need a feedback path that survives the session. That’s the whole trick, and it fits in a couple of small files.