OpenAI finally pulled back the curtain on Jalapeño, its first custom inference chip, at the Hot Chips conference this week — and the benchmark results are the kind of numbers that make the whole AI industry pay attention. Tested on SemiAnalysis’ InferenceX benchmark, the chip registered both more tokens per user and more throughput per kilowatt than the currently available state-of-the-art inference processors.
That matters far beyond a spec sheet. Inference is where the real cost of AI lives, especially now that agents are chewing through longer tasks and more tokens every single day. If a chip can serve more AI work per unit of power while cutting latency, it changes the economics of everything built on top of it.
The headline numbers
OpenAI’s head of hardware Richard Ho was direct about what the results show. “The bottom line is that the results show a very, very significant performance advance over state of the art,” he said in a press call. “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly.”
Across three public models — GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 1T — the chip delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the comparison systems. For highly interactive workloads, the kind agents depend on, it posted 2.1 to 4.1 times higher performance. Notably, the comparison systems were leading Nvidia Blackwell setups, so this wasn’t a win against some hypothetical baseline.
Why inference efficiency is suddenly the whole game
For a long time the AI conversation revolved around training — who could throw the most GPUs at the biggest model. But the economics have shifted. Training happens once; inference happens billions of times. Every prompt you send, every agent step, every token generated carries a serving cost. When agents started running multi-step tasks, token spend exploded, and inference became the bottleneck that actually decides how much you pay.
I’ve written before about how agent costs became the new frontier, and this chip is the clearest sign yet that the labs agree. Jalapeño is rated at 700 watts but stayed at or below 550 watts of sustained power on the tested workloads. Doing more work while drawing less power is the direction the entire industry has to move in.
The full-stack play underneath the chip
What makes Jalapeño interesting isn’t just the silicon. OpenAI designed the chip, memory, network, software, and rack-scale system together, around real language-model workloads. The design targets the specific phases where inference usually stalls — the compute-heavy prefill stage and the communication phase where data moves between cores and chips.
“We designed Jalapeño to minimize data movement and communication delays,” the company said in its results post. “Model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase.”
That’s a hardware answer to a problem developers feel every day. When response latency compounds across an agent task, a small per-step delay becomes a large total delay. Cutting end-to-end latency 3.6 times on a big model isn’t a benchmark flex — it’s the difference between an agent that feels alive and one that feels like waiting.
AI helped design it, and the design is meant to be programmed by AI
OpenAI says AI played a direct role in the chip’s development, helping the team go from design to tapeout in nine months. Using Codex with GPT-Astra, the team brought three open-weight models that weren’t part of the original production plan up to high performance within two months. For selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than the human-expert-written versions.
There’s a loop here worth noting. OpenAI built a clear, predictable programming target — local tensors, explicit communication, predictable synchronization — so that AI can optimize how work is mapped and scheduled. The chip was designed as something AI can program, and AI is what helped program it. This is the harness and model argument upgraded to the silicon level.
What this means for the cost of running AI
The practical takeaway is about money. OpenAI said the performance is the beginning of a multigenerational platform, with Gen 2 deep in development and Gen 3 taking shape. Each generation is meant to push efficiency and speed further. That translates to a lower cost per successful result, which OpenAI says improves operating leverage — useful work and revenue can grow faster than the cost to serve it.
For anyone building with AI, cheaper inference is a double-edged sword. On one side, it lowers your own serving costs if you’re running models. On the other, the frontier labs get cheaper at serving their own models, which is power they don’t have to hand to the hardware giants. I touched on how deeply hidden agent costs bite when a single agent workstream spirals out of control — the kind of surprise that gets more painful the longer agents run.
The Nvidia question none of us can ignore
There’s an obvious elephant here. The comparison was against Nvidia Blackwell systems, and Ho acknowledged that by the time Jalapeño reaches meaningful deployment, the competition may have moved ahead. OpenAI plans to begin deploying the chip within its own compute infrastructure by the end of the year, with more significant volume coming in 2027.
Even so, OpenAI was explicit that it will keep deploying Nvidia and other partner accelerators for both training and inference. This isn’t a farewell to Nvidia — it’s a diversification bet. In-house silicon gives OpenAI leverage in pricing negotiations and a hedge against soured supply relationships. The same logic that pushed AI memory shortages into consumer prices is pushing labs to control their own stack.
Bottom line
Jalapeño is one chip generation, but it’s a signal about where AI infrastructure is heading. When the biggest lab starts turning X amount of watts into markedly more useful AI work, the math doesn’t stay contained inside one data center — it ripples into pricing, into what agents can afford to do, and into how much compute you need to get a task done.
For developers the practical lesson is to start thinking about efficiency the way the labs are. Measure tokens, watch latency, and treat inference cost as a design constraint rather than a billing surprise. The hardware race is ultimately trying to solve the same problem you are — doing more with less. When the biggest players push that boundary, everyone downstream eventually feels the benefit.