How to Add an AI Assistant to Emacs with gptel and Ollama

A few days ago I read an essay called Emacs in the Age of Magic, and it stung a little. The author’s argument: everything that made Emacs users feel faster — magit, elisp, the editing speed, the endless customization — evaporated the moment we stopped typing and started telling an AI what we wanted. “We may have felt safe bringing a gun to a knife fight,” he wrote, “but now everyone’s just sending a drone.”

He’s not wrong. But I read it differently. The answer isn’t to abandon the editor and live in a terminal running a coding agent. The answer is to drag the model into Emacs — to make the assistant sit in the same buffer as the code, the config file, the org-mode notes. That’s what gptel does, and after a morning of wiring it to my local Ollama, I’m not sure I want to work any other way.

This is a tested setup, not a theoretical one. Everything below ran on my machine: gptel from MELPA, a local Ollama server, and a couple of small models. I’ll show you the config, the commands, the version gotcha that cost me twenty minutes, and what the responses actually looked like.

Why bother bringing the AI into Emacs?

Because context-switching is a tax you pay every time you leave the editor.

The usual loop goes like this: you’re reading code, you hit something you don’t understand, you alt-tab to a browser tab, paste the function, wait, copy the answer back, and try to remember where you were. Ten seconds of work, thirty seconds of re-orientation. Do that twenty times a day and it adds up.

gptel collapses that loop. You’re in a buffer, you put the cursor somewhere or select a region, you call M-x gptel-send, and the answer appears in a side window. No tab, no paste, no leaving. The assistant sees the same text you’re looking at.

There’s a second reason, and it’s the one that matters to me: the model can be local. Point gptel at Ollama instead of a cloud API and the code you’re asking about never leaves the machine. I’ve written before about auditing what your AI coding tools actually upload — a local model is the simplest possible answer to that whole class of problem.

What you need

• Emacs — 29.1 or newer is the comfortable choice. More on the version wrinkle in a moment.
• Ollama running locally, with at least one model pulled.
• gptel from MELPA.

Check that Ollama is alive before you start:

curl http://localhost:11434/api/tags

If you get back a JSON list of models, you’re good. If not, run ollama serve first.

Step 1: Install gptel

If you already have MELPA in your package archives, this is the standard dance:

M-x package-install RET gptel RET

The gptel manual lives in the repo’s README, and it’s worth skimming — but the short version is that gptel is a chat client that talks to whatever backend you configure. Ollama is one of about a dozen it supports out of the box.

Step 2: The version gotcha

Here’s the thing nobody tells you: the current gptel release uses a handful of Emacs 29 macros — define-keymap, defvar-keymap, and a setf expander for plist-get — even though the package metadata still says it needs only Emacs 27.1. On an older Emacs without help, gptel fails to load with errors like Symbol's function definition is void: define-keymap.

I hit this myself on a scratch Emacs 27.1 install. The fix is one package: compat from GNU ELPA, which backports those macros.

M-x package-install RET compat RET

Then make sure it loads before gptel:

(require 'compat)

On Emacs 29.1 and newer you don’t need compat at all — those macros are built in. But if you’re on an LTS distro that ships an older Emacs, this one line is the difference between a working setup and an afternoon of confusion.

Step 3: Point gptel at Ollama

Add this to your init file. It’s the whole config.

(require 'gptel)
(require 'gptel-ollama)

(setq gptel-backend
      (gptel-make-ollama
       "Ollama"
       :host "localhost:11434"
       :models '(qwen3.5:0.8b llama3.2:latest)
       :stream t))

(setq gptel-model 'llama3.2:latest)
(setq gptel-temperature 0.2)

Let me translate the important parts:

• :host is where Ollama listens. The default is localhost:11434; change it if you run Ollama on another machine or port.
• :models is the list gptel will let you pick from. Use the exact names ollama list prints — the :latest tag matters.
• :stream t gets you tokens as they arrive instead of one blob at the end. It’s the default, and it’s worth keeping.

Restarting Emacs after this is optional — M-x eval-buffer on your init file is enough.

If you’d rather drive Ollama from a browser than from Emacs, I covered a full self-hosted setup with Open WebUI and Ollama a while back. gptel is the other half of that: same models, no browser.

Step 4: Start a chat

Two commands matter:

• M-x gptel — opens, or switches to, a chat buffer for the current backend.
• M-x gptel-send — sends the text up to the cursor, or the active region, to the model and prints the response in a window.

The workflow I actually use: put the cursor at the end of a function, M-x gptel-send, and type a question. The answer lands in a side buffer. If you want the assistant to see a specific chunk of code, select it first — the region goes along with your prompt.

Step 5: Give it a system prompt that sounds like you

gptel’s default system prompt is fine: “You are a large language model living in Emacs and a helpful assistant. Respond concisely.” But the whole point of Emacs is that you can make it yours.

(setq gptel-system-prompt
      "You are a senior embedded Linux engineer. Answer concisely,
prefer shell commands and short code, and never suggest a
GUI tool when a command-line one exists.")

I keep mine short and opinionated. A vague system prompt gets you a vague assistant.

Step 6: A different model per project

This is the part that made me a believer. You can set gptel-model per directory with a .dir-locals.el file:

((nil . ((gptel-model . llama3.2:latest)
         (gptel-temperature . 0.1))))

Now the firmware repo gets the small fast model and the writing repo gets something bigger. I tested this: opening a file inside a directory with that .dir-locals.el switched the active model, and the next request went out under the new model’s name. If you’re choosing between models for different tasks, my local AI daily workflow post has the breakdown I use.

What I actually observed

I didn’t just configure this and trust it. I ran prompts through it.

With llama3.2:latest behind gptel, I asked: “In one short sentence: what does the Linux command ‘chmod 600’ do?” The response came back in seconds:

chmod 600 sets the permissions of a file to read and write for the owner only, with no access for others.

Correct, concise, no fluff. Streaming worked too — a counting prompt returned 1, 2, 3, 4, 5 token by token rather than all at once.

One honest quirk: I first tested with qwen3.5:0.8b, which is a reasoning model. gptel’s non-streaming path handed me the model’s internal reasoning trace instead of the final answer. That’s a model-format issue rather than a gptel bug, but it’s worth knowing — if you use a reasoning model, expect the reasoning field to show up in the response. For predictable results, plain instruction-tuned models like llama3.2 are the safer pick.

And if you want the model to give you structured output rather than prose, that’s a different feature — I covered the underlying Ollama side in getting reliable JSON from local LLMs.

The honest limits

Let me not oversell this. A 3B model in your editor is not a frontier coding agent. It won’t refactor a whole repository, it won’t run tools, and it will confidently get things wrong in ways a bigger model wouldn’t. gptel can do tool-use and agentic loops, but you need a model that supports it and a fair bit more configuration than what’s above.

What it is good at: explaining a function, drafting a regex, writing a commit message, summarizing a diff, answering “what does this flag do” without leaving the buffer. For that slice of work, it’s genuinely faster than the browser-tab dance.

If you want to go further down the local-inference road — bigger models, quantization, squeezing a 70B onto modest hardware — my AirLLM tutorial picks up where this one leaves off.

Bottom line

The essay I started with mourned the Emacs advantages that AI made obsolete. I think it had it backwards. The editor’s real advantage was never magit or elisp — it was that Emacs is a place you can build things. gptel is just the latest thing built there: the model, sitting in the buffer, waiting for a question.

Install it, point it at Ollama, and give it a system prompt that sounds like you. You’ll be surprised how quickly “alt-tab to the browser” starts to feel like the slow way.