How to Run Open-Source AI Models Locally with Ollama

I stopped sending my code to a cloud API for everyday tasks. Not because the cloud is slow or expensive — it is neither — but because I grew tired of mailing my own work to a server I do not control. Running AI locally is now fast, cheap, and genuinely useful for a developer. Here is how to do it on your own machine, with open-source models you can inspect, change, and keep forever.

Open-source AI model running locally with Ollama on a laptop — *Image: via Wikimedia Commons (CC0)*

I am an ICT division manager and a Master’s student, and I run these models on a laptop and a small desktop. Nothing here requires a data center. You do not need a GPU. You need a few gigabytes of free disk and about twenty minutes.

Table of Contents

Toggle

What “running AI locally” actually means

When you run AI locally, the model lives on your own hardware. You type a question, the model replies, and no data ever leaves your network. You own the pipeline end to end. There is no API key to rotate, no usage limit, and no surprise bill when you go over the free tier.

The trade-off is clear: local models are slower and smaller than the best cloud models. They are not trying to replace a frontier assistant. They are for tasks where privacy, continuity, or cost matters more than raw capability.

Step 1: Install Ollama

Ollama wraps the model and your hardware into a single tool. It handles downloads, memory, and the API, so you do not need to learn CUDA or write a training script.

On Linux and macOS:

“`bash
curl -fsSL https://ollama.com/install.sh | sh
“`

On Windows, grab the installer from [ollama.com/download](https://ollama.com/download) and run it.

The installation script is maintained in the project’s official repository at github.com/ollama/ollama, and the full user manual is at ollama.com/docs.

Then confirm it is alive:

“`bash
ollama –version
“`

You should see a version number. If you get “command not found”, restart your terminal or log out and back in.

Step 2: Pull a model

A model is a file that has learned patterns from text. It is large, usually between one and ten gigabytes, but you only download it once.

The simplest command:

“`bash
ollama pull gemma3
“`

This downloads the “gemma3” model. There are other options: “llama3.2” is a solid general-purpose model, “qwen3.5” is fast and small, and “nomic-embed-text” is built for turning text into vectors for search. Pick one and pull it.

To see what you have:

“`bash
ollama list
“`

Your models stay on your disk until you remove them with `ollama rm `.

Which model should you pick?

For a first model, I recommend starting with `llama3.2` or `gemma3`. Both are reliable for everyday questions and easy to test.

Step 3: Talk to it

Once the model is downloaded, you can chat with it directly:

“`bash
ollama run gemma3 “Explain Linux permissions in one paragraph”
“`

The first run downloads the model if it is missing and then starts a chat loop. Type your question, press Enter, and get a reply. Type `exit` to leave.

I tested this myself on this machine. The model answered straight away, and the whole session stayed on the machine that ran it.

Step 4: Connect it to your own tools

The chat loop is fine for casual questions. For actual work, you want the Ollama API. Any program on your network can send it a prompt and read a reply.

Here is a minimal `curl` example:

“`bash
curl http://localhost:11434/api/generate -H “Content-Type: application/json” -d ‘{
“model”: “gemma3”,
“prompt”: “What is 2 plus 2?”,
“stream”: false
}’
“`

The reply comes back as JSON with a `response` field. On this machine, the model answered with `4` — correct, and done in under a second.

For streaming output, change `stream` to `true` and read the response line by line. Web UIs and editor plugins rely on exactly this endpoint.

Practical things I run locally

– **Summarizing a long thread or meeting note** — paste the text, ask for three bullet points and a one-sentence takeaway.
– **Explaining unfamiliar code** — paste a function, ask for a plain-English breakdown and the edge cases.
– **Drafting commits and release notes** — point the model at your diff and ask for a conventional-commit message.
– **Local search over my own notes** — pair Ollama with a small embedding model (nomic-embed-text) and a lightweight vector store to ask questions about documents that never leave the machine.

These are not demos. They pull their weight in my daily workflow, and the setup takes less time than a coffee run.

Why run AI at home instead of the cloud

Three reasons keep me coming back to local models.

**Privacy.** When source code, internal tooling, or customer data touches a cloud model, it leaves your machine. Local models never do. That matters for any ICT team handling regulated or proprietary work.

**Cost.** A local model costs only electricity and disk space once it is downloaded. No per-token billing, no overage notices.

**Control.** You can inspect the model, pin the version, and keep it running offline. Cloud APIs change or restrict access without warning. A model you hold on your own hardware cannot be pulled away.

When you still want the cloud

Local models are not always the answer. For long documents, heavy reasoning, or anything that needs the newest knowledge, a cloud model is still faster and more capable. The honest path is not “local or cloud” — it is to pick the right tool for the job and move between them as needed. I use both, without pretending one is always better.

Troubleshooting the setup

**The model takes a long time to respond.** That usually means the model is under-sized for the task. Try a larger model, or switch from a CPU-only machine to one with a GPU. If you are on a laptop, run the model at a lower quality setting (the `-q4_K_M` suffix) to trade quality for speed.

**`ollama run` hangs after “pulling manifest”.** The model is still downloading. Wait, or run `ollama pull ` in another terminal to see progress.

**You get a “no such model” error.** Re-run `ollama pull ` to fetch it. Check `ollama list` to confirm the name exactly.

**The API is unreachable.** Confirm Ollama is running (`ollama ps`) and that no firewall is blocking port 11434.

Where to go next

If you want a chat interface instead of the command line, the site’s guide on self-hosting a ChatGPT-style interface with Open WebUI and Ollama walks through the setup end to end.

If you prefer to keep AI inside your editor, I also wrote about adding an AI assistant to Emacs with gptel and Ollama, which shows how to send those same requests from a text editor without leaving your workflow.

If you use AI coding tools, remember that open-source tools have their own supply-chain risks. Read our guide on detecting compromised npm packages before trusting any dependency that talks to a model.

For a broader picture of what models can and cannot do, take a look at the site’s interview-style pieces on the economics of local AI — they explain why the “local versus cloud” debate is less important than building a workflow that fits how you actually work.

Bottom line

Local AI is no longer a hobby only. With Ollama, you can pull a model, talk to it, and wire it into your own tools in under twenty minutes. Start small: pull one model, ask it a question you would normally send to a cloud API, and see how it feels. Your code stays on your machine, and the only thing you pay is electricity.