Ollama

Run open-weight language models locally with one command and a local HTTP API. Use it as the model backend for Open WebUI, Aider and scripts.

Project: Ollama Documentation: Ollama docs · Model library

Verification: commands, environment variables and API paths are from the official docs and CLI help from memory; Ollama was not installed here, so nothing was executed. Model tags change; check the library. Items marked (unverified) need confirming.

Basics #

Model tags look like llama3.2:3b (name, then size or variant). Pick a size that fits your VRAM; see quantisation.

Modelfile #

FROM llama3.2:3b
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM You are a terse homelab assistant.

Environment Variables #

Set them in the service environment (sudo systemctl edit ollama), not just your shell.

HTTP API #

The API has no authentication. Do not publish port 11434; put an authenticating reverse proxy in front if you must.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false
}'

curl http://localhost:11434/api/tags

Docker #

GPU use needs the NVIDIA Container Toolkit on the host. For a VM, pass the GPU through first: Proxmox GPU and PCIe Passthrough. See Docker Compose for the file format.

services:
  ollama:
    image: ollama/ollama:<TAG>
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

Troubleshooting #

See Also #

• Open WebUI and Self-Hosted AI Stacks
• llama.cpp
• Hugging Face CLI and Model Formats
• Docker
• AI