How to Vet an AI Coding Agent Before Trusting It With Your Code

Your AI coding agent is about to become the one choosing your dev tools, your CI pipeline, and maybe even the hardware your company buys. That is already happening quietly: ask an agent to “fix the build” and it installs whatever package it first read about, not necessarily the one that is safest. A moment ago I was reading Geoff Stearns’ essay Your Agent Will Pick the Hackable One, and it clicked. We kept assuming the human is the one in charge of the machine. The agent is starting to be the one pressing the buttons.

Felix at his desk editing code with an AI coding agent on screen
Image: Daudi Mukiibi via Wikimedia Commons (CC BY-SA 4.0)

I am a software developer and an ICT division manager. I also run Bleuken and I write about tech, chess, and computing. I test these tools the way I test anything: small, then bigger. And I have learned that vetting an AI coding agent is not a one-time checkbox. It is a habit. If you want to go deeper, How to Build an AI Agent with Python: A Step-by-Step Guide walks through the same workup from the builder’s side. This is a practical checklist you can run before you let an agent touch your codebase.

Why your agent will pick the hackable one

Stearns’ core point is simple and it made me rethink a few of my own purchases. When a device lets you change what it runs, your agent will lean toward recommending it, because that is the one it can fix later. The buyer never has to think about openness for it to matter.

The same is true of software. An agent that leans toward tools it can modify, fork, or replace is an agent that is easier for you to control. An agent that leans toward tools it cannot see is an agent that can quietly do things you did not ask for. “Hactivable” here does not mean “will be broken into next week.” It means the agent can reason about the code, change it, and make decisions about it. The more agency the software gives the model, the more power it has to act on its own.

The uncomfortable part is that the trait that makes an agent useful — it can act — is the trait that makes it risky. You want an agent that does what you mean, not one that does exactly what it inferred you meant.

What “open” actually buys you

You can tell real security from lock-in by asking one question: who controls the signing keys? A device can refuse unsigned firmware and still let the owner add their own key. Pixel phones already work this way. If a maker blocks code the owner chose and signed, that block is usually protecting the maker’s business, not your security.

Open firmware changes the math of attacks. When millions of devices share the same weakness, one exploit works on all of them — the Mirai botnet did exactly this in 2016. A mix of different firmwares makes that harder. The flip side: AI finds bugs cheaply now, so it also finds variants of a weakness cheaply. The bigger risk is malicious firmware dressed up as something useful. Verified installers, owner-held keys, recovery modes, and agents that check firmware before they install it all help, and none of them require locking the device.

A vetting checklist for AI coding agents

Here is the checklist I run before I hand an agent a real repo. You do not need to do every item on day one; just start.

1. Ask it to generate a known-insecure snippet

Give it a trivial task that has a well-known wrong answer — a SQL query built by string concatenation, a password comparison that ignores timing, a shell call built from user input. If it confidently hands you the broken pattern instead of flagging it, that is a data point. Note what it does when you correct it. Does it absorb the correction, or argue?

2. Check what tools it actually runs

List the tools and permissions: can it run shell commands, edit files outside the workspace, call the network, or read your environment variables? Turn off what you do not need. An agent with a exec tool is a powerful feature; it is also a wide-open door.

3. Prefer agents you can inspect and self-host

The more you can read the model’s prompts, the tool list, and the sandbox rules, the less you have to trust a black box. Local and self-hosted models — the same class I covered in my How to Run Open-Source AI Models Locally with Ollama guide — let you audit what the agent actually sees. Hosting it yourself also removes any question about where your code is being sent.

4. Run it on a throwaway repo first

Start with a mini-project you do not care about. Give it a ticket, watch what it does, and then review the diff instead of accepting it. You are testing whether it follows intent, not whether it produces a working feature. I run three sessions on the throwaway repo before I trust it on a customer repo.

5. Verify the diff before you merge

Treat every agent-generated diff like a pull request from a very fast but unfamiliar junior developer. Search the diff for secrets, for eval, for unexpected network calls, for anything that writes outside the intended directory. My rule of thumb: 5 minutes of review saves an hour of incident hunting.

6. Keep a rollback and a snapshot

Make sure your repo has a clean git history, your CI has reproducible builds, and you can revert a bad agent-generated commit in one click. If the agent is the first line of change, git is your second line of defense. Keep the second line real.

The trade-off nobody spells out

Openness is not a free security win. It is a trade. More openness means more choice, more transparency, and more control — and also more attack surface if you do not verify. The makers who win here will not be the ones who lock everything down; they will be the ones who make it easy to inspect, easy to roll back, and hard to brick. That is exactly what the open-firmware hardware makers are discovering, and it applies to your software stack too.

An agent that recommends the device it can fix pushes makers toward openness. An agent built by a company with its own devices to sell may not. Who your agent works for will decide whether this helps you or just moves the lock-in somewhere harder to see.

What to watch for in the next year

The buying conversation is moving from “which laptop?” to “which agent can change it?”. As I covered in AI Agents Are Now Shopping for You — and That Changes Everything, more buying decisions are already running through agents. Expect open firmware and agent-accessible devices to win fastest where people compare options every time — synths, TVs, routers, printers — and closed, deep-agent access to win where switching is painful, like phones. Some of this is real security and some of it is lock-in. You can usually tell which by asking who controls the signing keys.

For your own stack, the practical takeaway is boring and reliable: limit the agent’s tools, review its diffs, prefer things you can read and control, and keep the ability to undo. That is how you stay the person in charge. When United Airlines recently banned AI agents from booking flights, the story was less about the airline than about who decides what an agent should do for you in a world where agents act for you.

Bottom line: your agent is about to pick a lot of what you run. Give it less rope, verify more, and keep the off switch real.

Filed under Tech & Gadgets
Last Update: November 16, 2026 by Felix AlterEgo
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Newest
Oldest Most Voted