Ollama Local Inference Setup Guide

Getting Started with Ollama

Ollama provides the simplest path to running LLMs locally. This guide covers everything from installation to production API configuration, based on Blue Note Logic's deployment experience.

Installation

Linux (recommended for production):

curl -fsSL https://ollama.com/install.sh | sh

macOS:

brew install ollama

Windows: Download the installer from ollama.com/download

Loading a Custom GGUF Model

To load a custom model like dobetter-norge-v2:

1. Create a Modelfile:

   FROM ./dobetter-norge-v2-q5_k_m.gguf
   PARAMETER temperature 0.3
   PARAMETER num_ctx 4096
   SYSTEM You are a Norwegian legal expert assistant.

2. Build the model:

   ollama create dobetter-norge-v2 -f Modelfile

3. Run inference:

   ollama run dobetter-norge-v2 "Hva sier arbeidsmiljoloven om overtid?"

API Configuration

Ollama exposes a REST API on port 11434 by default:

curl http://localhost:11434/api/generate -d '{"model": "dobetter-norge-v2", "prompt": "Your question here"}'

Performance Tuning

• GPU Layers: Set OLLAMA_NUM_GPU=999 to offload all layers to GPU
• Context Size: Match num_ctx to your model's training context (4096 for dobetter-norge-v2)
• Concurrent Requests: Set OLLAMA_NUM_PARALLEL=4 for multi-user deployments
• Keep Alive: Use OLLAMA_KEEP_ALIVE=24h to keep models loaded in memory