The install is the attack surface
Here’s the thing about agents: they are, by design, installers. A coding agent that patches a broken test is an installer. A consumer agent that swaps out a router’s firmware is an installer. An agent that picks between two laptops and orders the one it prefers is an installer.
You can lock down the model, but you can’t lock down the install. And the install is where the world is widest open.
Stearns’s thesis is that the pressure agents place on manufacturers — favor the device you can change, and the buyer never has to think about openness — will force makers open. I think he’s half right. The maker that refuses to let an agent change its firmware will lose the sale, so the fashion moves open. But “open” is not the same as “safe.”
An open firmware is one an agent can also install. An agent that can install a trusted user’s custom build can also install one a bad actor wrote. The Mirai botnet of 2016 did not start because someone found a hole in a single vendor’s code. It started because every device ran the same, unchanged, modifiable code, and one attack worked on all of them. AI changes the economics: it is now cheap to find the hole in every variant at once, and expensive for the vendor to patch them all.
Stearns’s essay is worth reading in full, and what stays with you is how unremarkable this all feels. Agents recommending the device they can modify, then fixing it. It is the natural end of “move fast.” The danger is that we have never treated the move-fast layer as a security problem.
His answer to the lock-in question is that agents will weigh convenience and safety together. A device that keeps its owner’s key but refuses outside code is not open at all; it is locked. The makers who understand this — who treat firmware as a product that will be replaced, not a liability that will be supported for a decade — will win the trust that matters.
The more I thought about Stearns, the more I realised the essay isn’t only about hardware. It’s about the human layer, too. And that’s the part nobody is talking about.
Who guards the stack?
A week ago, Nvidia’s security chief quietly left. David Reber spent four years as CSO of a company that is, in many ways, the poster child for everything a security team is supposed to police. He left in August, citing health reasons. Two and a half months later, the CEO he answered to is gone, resigning over affairs that intelligence agencies had flagged for nearly a year.
You don’t have to be a security specialist to see the pattern. The people charged with guarding the stack are already burning out, going quiet, or walking away. A former intelligence officer told TechCrunch that security teams are often underfunded because they do not drive revenue, and that CEOs routinely answer warnings with “not a nickel more than compliance.” That is a vivid phrase, and I have felt it as an ICT manager trying to justify a budget line to a committee.
We blame the model. We blame the data. We blame the vendor. Meanwhile the people whose whole job is to notice that something is off — the humans at the gate — are the first to leave. This is not a criticism of any one individual. It is a structural fact. Security teams are measured on what did not happen, which means nothing ever looks good enough to fund. The people who stay are the ones who can prove nothing happened. The rest leave. When the people who read the room before the leak are gone, the leak is already a headline.
The Philippine angle, honestly
I am an ICT division manager in a government institution. I am also a Master’s student in IT. For months I have been trying to justify adopting AI coding agents and AI-assisted workflows to my budget office. The pitch is simple: save time, reduce toil, let the team focus on the hard problems.
The pitch is also a bit reckless. Because “adopt first, ask questions later” is the default. We want the agent to review my pull requests, draft the SOP, summarise the incident report, browse the stack for me. Why not? It is free, it is fast, and the promise is that the slow human is replaced by a fast machine.
Here is the thing I have not said out loud, mostly because I don’t know the answer: I do not know that I have hardened the agent layer. I have enabled it. I have told my team what not to paste into it. But I have not built the human gate that Stearns’s essay quietly demands — a place where an agent’s suggestion gets checked, questioned, and, when it is wrong, corrected by someone who knows the ground truth.
I wrote a practical guide to vetting an AI coding agent before you trust it with your code — the steps I run before an agent touches my repository. It is my attempt at that human gate. It will not stop the hackable one from being chosen. But it is a start. If you run agents on your own machine, the same logic applies: a locally hosted model you control is easier to audit than a cloud API you are paying for. My guide to running open-source models locally with Ollama is a short place to start. And if you want to build one yourself, my step-by-step guide to building an AI agent with Python is a reasonable place to begin.
I have spent a lifetime watching a board, first in chess and later in boxing, and the same lesson repeats. You do not win by moving the fastest piece. You win by controlling the centre, by knowing which piece to sacrifice, and by making your opponent explain why the move was necessary. A developer who presses every hot new agent into service is playing the piece, not the board. The ICT division that adopts a tool because it is new, without asking who answers for the tool, is a boxer who throws a punch and forgets to cover the face. Speed is not strategy.
As someone who manages other people’s technology budgets, I know the pressure to deliver is real. The department wants more service for the same budget. The vendor promises an agent will do the work. It is tempting to say yes. The honest answer is that I need to say yes — and no, at the same time. Say yes to the capability, and no to the unexamined trust.
Bottom line
So where does that leave us? Not with a panic, and not with a shrug. With a question we should all be asking our vendors and our budget offices:
When your AI agent decides what to install, what to buy, and what to trust — who is on the other side of the phone?
That is the person who matters. And as of right now, they are the one getting left behind. The broader industry story — why so many of these agents never make it at all — is something I covered in my AI graveyard roundup, and it is the same story told from the other side: the agent picks the hackable one, and the hackable one does not survive.
