Run a local LLM with Ollama and call it from your code

Run an open model in Ollama on your own machine and call it from curl, TypeScript, Python and an OpenAI SDK. It is for developers who want a private, local LLM API with no per-token bill.

Prerequisites#

• Linux (x86_64 or ARM64), or macOS 14 Sonoma or newer.
• At least 8 GB of RAM. With 16 GB or more you can run larger models.
• A GPU is optional. Ollama supports NVIDIA cards with compute capability 5.0 or higher and driver 550 or newer. Cards with compute capability 5.0 to 6.2 need driver 570 or newer. On Linux it supports AMD cards with ROCm v7. On Apple Silicon it uses Metal, and Intel Macs run on the CPU only.
• Node.js 24 LTS for the TypeScript example. It runs .mts files without a build step. Node.js 26 also works and becomes LTS on October 28, 2026.
• Python 3.10 or newer. The ollama package needs 3.8 or newer, but the openai package needs 3.10 or newer. Check your version with python3 --version.
• curl, plus jq if you want to filter JSON in the terminal.

Versions used here: Ollama v0.40.8, ollama 0.6.4 on npm, ollama 0.6.3 and openai 3.36.1 on PyPI.

1. Install Ollama#

Linux#

The official script installs the binary, creates an ollama system user and registers a systemd service.

curl -fsSL https://ollama.com/install.sh | sh

Check that the service is running:

sudo systemctl status ollama

To read the script before it runs, download it first:

curl -fsSL https://ollama.com/install.sh -o install.sh
less install.sh
sh install.sh

The Linux docs (opens in a new tab) also cover a manual .tar.zst install and the extra ROCm package for AMD GPUs.

macOS#

The same script works on macOS. It installs Ollama.app into /Applications, links the ollama CLI into /usr/local/bin and starts the app.

curl -fsSL https://ollama.com/install.sh | sh

You can also download the .dmg from ollama.com/download/mac (opens in a new tab) and drag the app into Applications. On first launch the app offers to link the CLI.

Confirm the server is up#

Ollama listens on 127.0.0.1:11434 by default.

ollama -v
curl http://localhost:11434/api/version

The second command returns JSON such as {"version":"0.40.8"}.

2. Pick a model that fits your hardware#

The model weights must fit in memory, with room left for the context window. On a discrete GPU that memory is VRAM. On Apple Silicon it is unified memory, shared with macOS and your other apps. When a model does not fit, Ollama splits it between GPU and CPU, and generation gets much slower.

The sizes below are from the Ollama library (opens in a new tab) in October 2026. Where the library lists a range, the download depends on the variant your platform gets.

gemma4:26b is a mixture-of-experts model with 25.2B total and 3.8B active parameters. Each token costs far less compute than in a dense 26B model, but all the weights still have to fit in memory.

Ollama picks the default context window from your VRAM: 4K tokens under 24 GiB, 32K from 24 to 48 GiB, and 256K at 48 GiB or more. To raise it, set the OLLAMA_CONTEXT_LENGTH server variable. Memory use grows with the window.

3. Pull and run a model#

This guide uses gemma4, the model the Ollama docs use in their own examples.

ollama pull gemma4

Start an interactive chat. Type /bye to leave.

ollama run gemma4

Or send a one-off prompt:

ollama run gemma4 "Explain a nonce in one sentence."

List the downloaded models, then the ones loaded in memory:

ollama ls
ollama ps

A model stays loaded for 5 minutes after its last request. To unload it right away, run:

ollama stop gemma4

4. Call the REST API#

Every client in this guide uses the same HTTP API on port 11434. The local API needs no key.

Single prompts with /api/generate#

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

With "stream": false you get one JSON object. The text is in response. The object also includes done_reason, token counts (prompt_eval_count, eval_count) and durations in nanoseconds (total_duration, eval_duration).

Print only the text:

curl -s http://localhost:11434/api/generate \
  -d '{"model": "gemma4", "prompt": "Why is the sky blue?", "stream": false}' \
  | jq -r .response

Conversations with /api/chat#

/api/chat takes a messages array with system, user, assistant and tool roles. Send the full history on every call.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [
    {"role": "system", "content": "Answer in one short paragraph."},
    {"role": "user", "content": "What is a Merkle root?"}
  ]
}'

Streamed and single responses#

The request above leaves stream unset, and both endpoints default to true. The reply arrives as newline-delimited JSON (application/x-ndjson). Each line has "done": false and a chunk of text in message.content, or in response for /api/generate. The last line has "done": true, a done_reason and the stats.

A trimmed line looks like this:

{"model":"gemma4","message":{"role":"assistant","content":"A Merkle"},"done":false}

Stream for chat UIs and long answers, so users see text as it arrives. Use "stream": false for scripts, short replies and structured output. If an error happens mid-stream, the HTTP status stays 200 and the error arrives as a line with an error field. Check each line for that field.

gemma4 and qwen3 support thinking. They can stream reasoning in message.thinking before the answer starts in message.content. Send "think": false to skip it when the model allows that.

5. Call Ollama from TypeScript#

The official ollama npm package wraps the same endpoints and ships its own types.

mkdir ollama-demo && cd ollama-demo
npm init -y
npm install ollama@0.6.4

Create chat.mts:

1import { Ollama, type Message } from "ollama";
2
3const ollama = new Ollama({
4  host: process.env.OLLAMA_HOST ?? "http://127.0.0.1:11434",
5});
6const model = process.env.OLLAMA_MODEL ?? "gemma4";
7
8const messages: Message[] = [
9  { role: "system", content: "You are a concise senior engineer." },
10  { role: "user", content: "Explain a Merkle tree in two sentences." },
11];
12
13// Wait for the full answer.
14const reply = await ollama.chat({ model, messages });
15console.log(reply.message.content);
16console.log(`tokens: ${reply.eval_count}`);
17
18// Stream the answer chunk by chunk.
19const stream = await ollama.chat({
20  model,
21  messages: [{ role: "user", content: "Name three uses for a local LLM." }],
22  stream: true,
23});
24for await (const part of stream) {
25  process.stdout.write(part.message.content);
26}
27process.stdout.write("\n");

Run it. Node 24 strips the type annotations itself.

node chat.mts

Without stream: true, chat() resolves to one ChatResponse. With it, you get an async iterator of chunks. The npm client does not read OLLAMA_HOST on its own, so the code passes it as host. In browser code, import from ollama/browser.

6. Call Ollama from Python#

Create a virtual environment and install the client:

python3 -m venv .venv
source .venv/bin/activate
pip install ollama==0.6.3

Create chat.py:

1import os
2
3from ollama import Client, ResponseError
4
5MODEL = os.getenv("OLLAMA_MODEL", "gemma4")
6client = Client()  # reads OLLAMA_HOST, defaults to http://127.0.0.1:11434
7
8messages = [
9    {"role": "system", "content": "You are a concise senior engineer."},
10    {"role": "user", "content": "Explain a Merkle tree in two sentences."},
11]
12
13try:
14    reply = client.chat(model=MODEL, messages=messages)
15    print(reply.message.content)
16
17    for chunk in client.chat(
18        model=MODEL,
19        messages=[{"role": "user", "content": "Name three uses for a local LLM."}],
20        stream=True,
21    ):
22        print(chunk.message.content, end="", flush=True)
23    print()
24except ResponseError as err:
25    if err.status_code == 404:
26        print(f"Model not found. Run: ollama pull {MODEL}")
27    else:
28        raise

Run it:

python chat.py

Responses support attribute access (reply.message.content) and dict access (reply["message"]["content"]). For asyncio code, use AsyncClient, which has the same methods.

7. Use the OpenAI-compatible endpoint#

Ollama also serves /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models and /v1/responses. Point an OpenAI SDK at http://localhost:11434/v1/. Most existing code then works after you change the base URL and the model name.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma4", "messages": [{"role": "user", "content": "Say hello"}]}'

With the official Python SDK:

pip install openai==3.36.1

1from openai import OpenAI
2
3# The SDK requires a key. Ollama ignores it for local requests.
4client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
5
6completion = client.chat.completions.create(
7    model="gemma4",
8    messages=[{"role": "user", "content": "Write a haiku about block times."}],
9)
10print(completion.choices[0].message.content)

In the JavaScript openai package the options are baseURL and apiKey. The OpenAI API has no field for context size. To change it, set OLLAMA_CONTEXT_LENGTH on the server, or build a model with PARAMETER num_ctx in a Modelfile (opens in a new tab).

8. Reach Ollama from another machine#

A common setup runs Ollama on a GPU box and calls it from a laptop. The server accepts only local connections until you change OLLAMA_HOST.

Read this before you expose the port#

The local API has no authentication. Anyone who reaches port 11434 can run prompts on your hardware, pull models and delete them. Use the narrowest option that works:

1. An SSH tunnel. The server stays on 127.0.0.1 and nothing is exposed.
2. A private LAN or VPN address, with a firewall rule for trusted IPs only.
3. A reverse proxy such as Nginx with TLS and authentication, for anything public. The FAQ (opens in a new tab) has a starting Nginx config.

Never forward port 11434 straight to the internet.

Option 1. Use an SSH tunnel#

Run this on the laptop:

ssh -N -L 11434:127.0.0.1:11434 user@gpu-box

Requests to localhost:11434 on the laptop now reach the remote server. If the laptop runs its own Ollama, forward a different local port and point clients at it:

ssh -N -L 11435:127.0.0.1:11434 user@gpu-box
export OLLAMA_HOST=127.0.0.1:11435

Option 2. Bind to a network address on Linux#

Open a systemd override for the service:

sudo systemctl edit ollama.service

Add the server's private IP:

[Service]
Environment="OLLAMA_HOST=192.168.1.50:11434"

The FAQ example uses 0.0.0.0:11434 instead. That address listens on every interface, public ones included.

Reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

After this change nothing listens on 127.0.0.1, so tools on the server itself also need OLLAMA_HOST set to that IP. Then limit the port to your subnet. With ufw:

sudo ufw allow from 192.168.1.0/24 to any port 11434 proto tcp

Option 2 on macOS#

The macOS app reads server variables from launchctl:

launchctl setenv OLLAMA_HOST "192.168.1.50:11434"

Quit and reopen the Ollama app to apply it.

Point clients at the server#

export OLLAMA_HOST=http://192.168.1.50:11434
ollama ls

The ollama CLI, the Python Client() and the TypeScript example above all read this variable. If a browser app on another origin calls the API, add that origin to OLLAMA_ORIGINS on the server. Add only origins you trust.

Verify it works#

List the models the server can see:

curl -s http://localhost:11434/api/tags | jq -r '.models[].name'

From the ollama-demo folder, run the TypeScript example, then check where the model loaded:

node chat.mts
ollama ps

You should see the Merkle tree answer, a token count and a streamed list. In ollama ps, 100% GPU under PROCESSOR means the model fits in VRAM. The CONTEXT column shows the allocated window.

Troubleshooting#

Connection refused, or ConnectionError: Failed to connect to Ollama#

The server is not running, or it listens on another address. On Linux, follow the logs:

journalctl -u ollama --no-pager --follow --pager-end

On macOS, read the server log:

cat ~/.ollama/logs/server.log

The API returns 404#

The tag in your request must match a pulled model. gemma4 and gemma4:latest are the same model, but gemma4:e2b is a different one. Compare the tag with the output of ollama ls, then pull the one you asked for:

ollama pull gemma4:e2b

Output is slow and CPU usage is high#

If ollama ps shows a split such as 48%/52% CPU/GPU, part of the model runs on the CPU. Pick a smaller tag or a shorter context. On NVIDIA, confirm the driver sees the card:

nvidia-smi

To see GPU detection details, turn on debug logs in the service itself. Starting ollama serve by hand gives you a different setup. It runs as your user and looks for models in ~/.ollama/models, while the service keeps them in /usr/share/ollama/.ollama/models. Open the override:

sudo systemctl edit ollama.service

Add this line under [Service], next to any line you added earlier:

[Service]
Environment="OLLAMA_DEBUG=1"

Restart the service and read the logs:

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -u ollama --no-pager --follow --pager-end

Remove the line and restart again when you finish.

Remote clients time out#

Set OLLAMA_HOST on the service, not only in your shell. Check the listen address on the server:

sudo ss -ltnp | grep 11434

If it still shows 127.0.0.1:11434, the override did not apply. Repeat the reload and restart commands from option 2.

Next steps#

• Let the model call your functions with tool calling (opens in a new tab).
• To expose your own tools over the Model Context Protocol, see Build an MCP server in TypeScript.
• Get typed JSON back with structured outputs (opens in a new tab) and a schema in format.
• Build search or RAG on embeddings (opens in a new tab).
• Pin a system prompt, temperature and num_ctx in a Modelfile (opens in a new tab), then build it with ollama create.
• Read the full API reference (opens in a new tab).