Running Local LLMs with Docker: A Practical Guide to Ollama and GPU Acceleration

Running large language models locally used to require significant infrastructure expertise and custom setup. Today, containers have made this process remarkably straightforward. Docker and Ollama together provide a reproducible way to run local LLMs that works across development machines, CI environments, and production servers.

This guide shows how to set up Ollama with Docker, configure GPU acceleration, and build a complete inference stack that you can version control and deploy anywhere.

Why Docker for Local LLMs

I have found that containerizing LLM workloads solves several recurring problems. Reproducibility is the most obvious. When your model configuration lives in a Dockerfile and environment variables, you can recreate the exact same setup on any machine. Version control becomes straightforward. Team members can pull a specific tag and get the same weights, dependencies, and runtime configuration.

Resource isolation matters too. LLMs are memory-hungry. Without containers, a model competing with other processes can cause unpredictable behavior. Docker provides clean boundaries and predictable resource limits.

Perhaps most importantly for teams, the barrier to entry drops dramatically. A new developer can pull an image and have a working LLM in minutes instead of hours of dependency installation.

Setting Up Ollama with Docker

Ollama provides official Docker images that work well for local development and production use. The basic setup is simple.

Create a docker-compose.yml file:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0

volumes:
  ollama-data:

Start the service:

docker-compose up -d

Pull a model:

docker exec -it ollama ollama pull llama3.2

Test the API:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain Docker in one sentence"
}'

That is it for the basic setup. The container exposes Ollama on port 11434 and persists downloaded models in a Docker volume.

Enabling GPU Acceleration

CPU inference works for testing, but it is painfully slow for any real use. GPU acceleration requires additional configuration.

Prerequisites

NVIDIA GPU with CUDA support
NVIDIA Docker runtime installed on the host
NVIDIA Container Toolkit

Install the toolkit on Ubuntu:

# Add the NVIDIA package repositories
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

# Install
sudo apt update
sudo apt install -y nvidia-container-toolkit

# Restart Docker
sudo systemctl restart docker

Modified docker-compose.yml with GPU Support

Update your Compose file to include GPU access:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  ollama-data:

The deploy.resources.reservations.devices section tells Docker to reserve one GPU for this container. Ollama automatically detects and uses available CUDA devices.

Verify GPU is working:

docker exec ollama nvidia-smi

You should see your GPU listed along with Ollama processes using VRAM.

Building a Complete Inference Stack

A production-ready setup typically includes more than just the model container. Here is a complete stack with a web interface, load balancer, and monitoring.

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama-data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - llm-network
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=false
    depends_on:
      - ollama
    volumes:
      - open-webui-data:/app/backend/data
    networks:
      - llm-network
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    container_name: llm-nginx
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - open-webui
    networks:
      - llm-network
    restart: unless-stopped

volumes:
  ollama-data:
  open-webui-data:

networks:
  llm-network:
    driver: bridge

This stack includes Ollama with GPU support, Open WebUI for a ChatGPT-like interface, and Nginx as a reverse proxy with SSL termination.

Architecture: How It Fits Together

flowchart TB
    subgraph Client["Client Layer"]
        Browser["Web Browser"]
        CLI["CLI / API Client"]
    end

    subgraph Proxy["Edge Layer"]
        Nginx["Nginx Reverse Proxy"]
    end

    subgraph App["Application Layer"]
        WebUI["Open WebUI"]
    end

    subgraph Model["Model Layer"]
        Ollama["Ollama Service"]
        GPU["NVIDIA GPU"]
        Storage[("Model Storage")]
    end

    Browser -->|HTTPS| Nginx
    CLI -->|HTTP| Nginx
    Nginx -->|HTTP| WebUI
    WebUI -->|/api/generate| Ollama
    Ollama -->|Inference| GPU
    Ollama -->|Read/Write| Storage

    style Client fill:#e1f5fe
    style Proxy fill:#fff3e0
    style App fill:#e8f5e9
    style Model fill:#fce4ec

The flow is straightforward. Clients connect through Nginx, which routes to Open WebUI for browser sessions or directly to Ollama for API calls. Ollama handles model loading and offloads computation to the GPU.

Persisting Models and Configuration

One challenge with containerized LLMs is managing model files. These are large, and you do not want to redownload them on every container restart.

Docker volumes solve this. The ollama-data volume persists models across container restarts. This also means you can backup your models by backing up the volume:

# Backup
docker run --rm -v ollama-data:/data -v $(pwd):/backup alpine tar czf /backup/ollama-backup.tar.gz -C /data .

# Restore
docker run --rm -v ollama-data:/data -v $(pwd):/backup alpine tar xzf /backup/ollama-backup.tar.gz -C /data

For reproducibility across environments, consider using a pre-built image that includes your preferred models:

FROM ollama/ollama:latest

RUN ollama serve & sleep 5 && ollama pull llama3.2 && ollama pull codellama

EXPOSE 11434

Build once, push to your registry, and every environment gets the same models without repeated downloads.

Security Considerations

Running any API locally has security implications.

Network binding* matters. By default, Ollama binds to localhost only. Setting OLLAMA_HOST=0.0.0.0 makes it accessible from any interface. In production, restrict this with firewall rules or bind only to internal networks.
Container isolation* provides some protection, but it is not a security boundary. Do not expose the Ollama port directly to the internet without authentication. The Open WebUI stack above includes basic authentication, but you should add your own identity layer for production.
Model provenance* is another concern. Ollama pulls models from various sources. For sensitive applications, pin specific digests or host models in your own registry:

ollama pull llama3.2@sha256:abcd1234...

Performance Tuning

GPU utilization and token throughput require experimentation.

Context length* affects memory usage linearly. A 4096 token context uses approximately twice the VRAM of 2048 tokens. Choose the smallest context that works for your use case.
Batch size* for concurrent requests can improve throughput but increases memory consumption. Ollama typically handles this automatically, but you can tune with environment variables.
Model quantization* dramatically reduces memory requirements. The default Ollama models use 4-bit quantization, trading some quality for size. For maximum quality at a memory cost, use unquantized models.

Monitor GPU utilization during load:

watch -n 1 nvidia-smi

You want high compute utilization but headroom on memory. If VRAM is saturated, the system will offload to system RAM, destroying performance.

Deployment Patterns

Several patterns work well for production.

Single node with GPU*: Run the entire stack on one machine with GPU. This is simplest and works for most local development and small production loads.
Separate GPU workers*: Run Ollama on GPU-enabled machines. Your application servers connect via the API. This lets you scale GPU and application tiers independently.
Kubernetes with GPU operator*: For larger deployments, Kubernetes with the NVIDIA GPU operator manages GPU scheduling. Ollama runs as a Deployment with GPU resource requests.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        resources:
          limits:
            nvidia.com/gpu: 1
        ports:
        - containerPort: 11434

Troubleshooting Common Issues

Container fails to start with GPU errors*: Usually means the NVIDIA Container Toolkit is not installed or Docker is not configured to use the nvidia runtime. Check docker info | grep Runtime and ensure nvidia is listed.
Out of memory errors*: Your model is too large for available VRAM. Try a smaller quantized model or reduce context length.
Slow first inference*: The model loads on first use. Subsequent requests are faster. Warm up containers with a healthcheck request at startup.
Model not found*: Ensure the model is pulled. The Ollama volume persists, but if you are using a fresh volume on a new host, you need to repull.

Conclusion

Docker has made running local LLMs accessible to developers who are not ML infrastructure specialists. With a single Compose file, you can have GPU-accelerated inference running locally. The reproducibility and version control benefits make this approach viable for teams and production workloads.

Start simple. Get Ollama running with CPU inference. Add GPU acceleration once that works. Then layer on additional services like web UIs and monitoring. The containerized approach lets you evolve your stack incrementally without breaking what already works.