Ollama — Run LLMs Locally Complete Guide

In this tutorial, you'll learn about Ollama. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Ollama lets you download and run large language models locally on your own hardware, providing complete privacy, offline capability, and zero API costs for AI inference.

What You'll Learn

• Installing Ollama and downloading models on Linux, macOS, and Windows
• Running local chat models like Llama 3, Mistral, and Gemma
• Building Python applications with Ollama's API and LangChain integration
• Creating custom Modelfiles for tailored model behavior
• Optimizing performance for CPU and GPU inference
• Using embeddings and multi-modal models locally

Why Ollama Matters

Running LLMs locally means your data never leaves your machine. No API keys, no rate limits, no privacy concerns — just your models running on your hardware. This is critical for security-sensitive applications where sending data to third-party APIs is unacceptable. DodaTech's Durga Antivirus Pro uses Ollama for offline threat analysis in air-gapped environments, and Doda Browser uses local models for private page summarization that never sends browsing data to external servers. This guide covers everything from first install to production local deployment.

flowchart LR
    A["Install Ollama\nLinux / macOS / Windows"] --> B["Pull Models\nLlama 3, Mistral"]
    B --> C["Run & Chat\nCLI & API"]
    B --> D["Modelfiles\nCustom Config"]
    C --> E["Python SDK\nApplications"]
    C --> F["Embeddings\nLocal RAG"]
    D --> G["Custom Models\nFine-Tuned Locally"]
    style C fill:#dbeafe,stroke:#2563eb

Installing Ollama

Ollama runs on all major platforms. The install Process takes under a minute.

Linux

curl -fsSL HTTPS://ollama.com/install.sh | sh

macOS

Download the .dmg from ollama.com or use Homebrew:

brew install ollama

Windows

Download the installer from ollama.com — Windows preview supports CPU and GPU inference.

After installation, verify it's running:

ollama --version

Expected output:

ollama version 0.5.0

Ollama runs as a background service on port 11434. You can check its status with ollama serve or verify the API is responsive:

curl http://localhost:11434/api/tags

Expected output: A JSON response listing the models you've pulled (initially {"models":[]}).

Downloading and Running Models

Models are downloaded as "pulls" — similar to Docker images. Ollama manages versioning, quantization, and hardware optimization automatically.

# Pull a popular model
ollama pull llama3.1

# Run an interactive chat session
ollama run llama3.1

Expected output:

pulling manifest
pulling d6bb13e6300d... 100%
pulling 8f5a4f03dfff... 100%
verifying sha256 digest
writing manifest
removing any unused layers
success
>>> Send a message (/? for help)
>>> What is a firewall in networking?

Available Models

The ollama list Command shows all downloaded models and their sizes:

ollama list

Expected output:

NAME                    ID              SIZE    MODIFIED
llama3.1:latest         8a1d2b7d8b2a    4.7 GB  2 hours ago
mistral:latest          7e3b2c1a4f5e    4.1 GB  5 hours ago
mxbai-embed-large:latest 3a4b5c6d7e8f   669 MB  1 day ago

Building Python Applications

Ollama exposes a REST API and provides a Python library. This lets you build local AI applications without external dependencies.

import ollama

response = ollama.chat(
    model="llama3.1",
    messages=[
        {"role": "user", "content": "Explain how AES-256 encryption works in three sentences."}
    ]
)

print(response["message"]["content"])

Expected output:

AES-256 is a symmetric encryption algorithm that uses a 256-bit key to encrypt and decrypt data in 128-bit blocks. It applies 14 rounds of substitution-permutation network operations, including byte substitution, row shifting, column mixing, and key addition. The same key is required for both encryption and decryption, making secure key management essential for protecting sensitive data.

The ollama.chat() function returns a dictionary with the response content, model metadata, and token usage. You can also stream responses for real-time display:

stream = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Count from 1 to 5."}],
    stream=True
)

for chunk in stream:
    print(chunk["message"]["content"], end="")

Expected output:

1, 2, 3, 4, 5

The Ollama REST API

Ollama runs a local HTTP server that any language can call. This is useful for building web applications and Microservices.

import requests
import JSON

response = requests.post(
    "HTTP://localhost:11434/API/chat",
    JSON={
        "model": "llama3.1",
        "messages": [
            {"role": "user", "content": "What are the three pillars of cybersecurity?"}
        ],
        "stream": False
    }
)

data = response.JSON()
print(data["message"]["content"])

Expected output:

The three pillars of cybersecurity are:

1. **Confidentiality** — Ensuring data is accessible only to authorized users, typically through encryption and access controls.
2. **Integrity** — Maintaining the accuracy and completeness of data, preventing unauthorized modification.
3. **Availability** — Ensuring systems and data are accessible when needed, protecting against downtime and denial-of-service attacks.

JSON mode is supported for structured output:

response = requests.post(
    "HTTP://localhost:11434/API/chat",
    JSON={
        "model": "llama3.1",
        "messages": [{"role": "user", "content": "List 3 cybersecurity tools as JSON"}],
        "format": "JSON",
        "stream": False
    }
)

print(response.JSON()["message"]["content"])

Expected output:

{
  "tools": [
    {"name": "Wireshark", "category": "Network Analysis"},
    {"name": "Metasploit", "category": "Penetration Testing"},
    {"name": "Durga Antivirus Pro", "category": "Endpoint Protection"}
  ]
}

Custom Modelfiles

Modelfiles let you customize model behavior — adding system prompts, adjusting parameters, and modifying the BASE model. Think of them as Dockerfiles for AI models.

FROM llama3.1

# Set system prompt for a cybersecurity assistant
SYSTEM """You are SecBot, a cybersecurity assistant built by DodaTech.
You help users understand security concepts, analyze threats, and recommend protections.
Keep answers practical, actionable, and beginner-friendly."""

# Adjust parameters
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER stop "</answer>"

Save this as SecBotModelfile and create the custom model:

ollama create secbot -f SecBotModelfile

Expected output:

transferring model data
creating model layer
using existing layer sha256:4e2f5c3a1b8d...
writing manifest
success

Now run your custom model:

ollama run secbot

import ollama

response = ollama.chat(
    model="secbot",
    messages=[{"role": "user", "content": "What is the best way to protect against ransomware?"}]
)

print(response["message"]["content"])

Expected output:

Great question! Here's a practical approach to ransomware protection:

1. **Backup, Backup, Backup** — Follow the 3-2-1 rule: 3 copies, 2 different media, 1 offsite.
2. **Patch Everything** — Keep your OS, software, and firmware updated. Ransomware often exploits known vulnerabilities.
3. **Email Filtering** — Most ransomware starts with phishing. Use good email security filters.
4. **Least Privilege** — Users should only have access to what they need. Don't run as admin.
5. **Endpoint Protection** — Tools like Durga Antivirus Pro detect ransomware behavior before files are encrypted.

Start with backups — they're your last line of defense when everything else fails.

Local Embeddings for RAG

Ollama supports embedding models for building local retrieval-augmented generation (RAG) systems entirely offline.

import ollama
import numpy as np

documents = [
    "Durga Antivirus Pro detects malware using signature matching",
    "DodaZIP compresses files with AES-256 encryption",
    "Doda Browser blocks trackers and protects privacy]
]

# Generate embeddings for each document
doc_embeddings = []
for doc in documents:
    response = ollama.embed(model="mxbai-embed-large", input=doc)
    doc_embeddings.append(response["embeddings"][0])

# Embed a query
query = "What encryption does DodaZIP use?"
query_emb = ollama.embed(model="mxbai-embed-large", input=query)["embeddings"][0]

# Find most similar document
scores = [np.dot(query_emb, de) for de in doc_embeddings]
best_idx = np.argmax(scores)
print(f"Best match: {documents[best_idx]} (score: {scores[best_idx]:.4f})")

Expected output:

Best match: DodaZIP compresses files with AES-256 encryption (score: 0.8734)

All embedding computation happens locally — no data leaves your machine. This is the approach Durga Antivirus Pro uses for offline threat intelligence matching in air-gapped environments.

GPU Acceleration and Performance

Ollama automatically uses GPU if available. Check your setup:

ollama run llama3.1 --verbose

Expected output (showing performance metrics):

>>> What is 2+2?
4
total duration: 1.23s
load duration: 342ms
prompt eval count: 14 tokens
prompt eval duration: 168ms
prompt eval rate: 83.33 tokens/s
eval count: 1 tokens
eval duration: 22ms
eval rate: 45.45 tokens/s

Performance varies significantly by hardware:

Use ollama ps to see currently loaded models and their memory usage:

ollama ps

Expected output:

NAME                    ID              SIZE    PROCESSOR
llama3.1:latest         8a1d2b7d8b2a    4.7 GB  100% GPU

Multi-Modal Models

Ollama supports vision models that can analyze images — all locally.

import ollama

response = ollama.chat(
    model="llama3.2-vision",
    messages=[
        {
            "role": "user",
            "content": "Describe this image in detail.",
            "images": ["screenshot.png"]
        }
    ]
)

print(response["message"]["content"])

Expected output:

The image shows a web browser interface with a dark theme. The navigation bar at the top contains tabs labeled 'File', 'Edit', 'View', and 'Help'. Below that is an address bar displaying 'https://github.com'. The main content area shows a GitHub repository page with a file listing, including folders like 'src', 'docs', and 'tests'. The right sidebar shows the repository's statistics, including stars, forks, and recent activity.

All image processing happens on your local machine. Doda Browser uses this to provide private, offline page description for Accessibility features.

Common Errors

1. Error: pull access denied, Repository does not exist

The model name is misspelled or doesn't exist. Check the correct name with ollama list or visit ollama.com/library for available models.

2. Error: model requires more memory than available

The model is too large for your system's available RAM/VRAM. Use a smaller quantization (e.g., llama3.1:8b-q4_0 instead of llama3.1:70b) or close other memory-intensive applications.

3. Error: model not found

You tried to run a model that hasn't been pulled yet. Run ollama pull <model-name> first. Use ollama list to see downloaded models.

4. Error: context deadline exceeded

Slow inference on CPU or overloaded system. Reduce concurrent requests, use a smaller model, or enable GPU acceleration. Restart Ollama with ollama serve if it becomes unresponsive.

5. Error: connection refused

Ollama server is not running. Start it with ollama serve or launch the Ollama application. Ensure port 11434 is not blocked by a firewall.

6. Error: JSON: invalid character — Malformed Modelfile

Your Modelfile has a syntax error. Check that all PARAMETER values are correct types (numbers for temperature, strings for system prompts) and there are no stray characters.

7. Error: unimplemented — Model doesn't support vision

You're trying to use image input with a text-only model. Use llama3.2-vision or llava for image support. Text-only models will reject image inputs.

Practice Questions

1. What port does Ollama's API listen on by default?
2. How do you create a custom model with a specific system prompt?
3. What is the difference between ollama run and the Python SDK's ollama.chat()?
4. Which Ollama model is best for local embeddings?
5. How do you check if GPU acceleration is working?

Answers:

1. Port 11434 — the REST API is available at http://localhost:11434.
2. Create a Modelfile with FROM <base> and SYSTEM """your prompt""", then run ollama create <name> -f Modelfile.
3. ollama run is the interactive CLI; ollama.chat() is the Python function that communicates with the local API.
4. mxbai-embed-large (669 MB) is optimized for local embedding generation. It produces 1024-dimensional vectors suitable for RAG.
5. Run ollama run llama3.1 --verbose and check the processor field in ollama ps. If it shows 100% GPU, acceleration is active.

Challenge: DodaZIP needs a fully offline document classification system that: scans a folder of documents, generates embeddings locally using Ollama, classifies documents by type (invoice, report, code), and moves them into organized subdirectories — all without internet access.

Mini Project: Private Offline Chatbot

Build a completely private chatbot that works without internet:

import ollama
import JSON

class OfflineChatbot:
    def __init__(self, model: str = "llama3.1", system_prompt: str = ""):
        self.model = model
        self.messages = []
        if system_prompt:
            self.messages.append({"role": "system", "content": system_prompt})

    def chat(self, user_input: str) -> str:
        self.messages.append({"role": "user", "content": user_input})
        response = ollama.chat(model=self.model, messages=self.messages)
        reply = response["message"]["content"]
        self.messages.append({"role": "assistant", "content": reply})
        return reply

    def save_conversation(self, filepath: str):
        with open(filepath, "w") as f:
            JSON.dump(self.messages, f, indent=2)

    def load_conversation(self, filepath: str):
        with open(filepath) as f:
            self.messages = JSON.load(f)

bot = OfflineChatbot(
    model="llama3.1",
    system_prompt="You are a helpful security assistant. Answer questions briefly."
)

print("=== Private Offline Chatbot ===")
print("Your data never leaves this machine.")
print("Type 'exit' to quit.\n")

while True:
    user = input("You: ")
    if user.lower() == "exit":
        bot.save_conversation("chat_history.JSON")
        print("Conversation saved to chat_history.JSON")
        break
    reply = bot.chat(user)
    print(f"Bot: {reply}\n")

Expected output:

=== Private Offline Chatbot ===
Your data never leaves this machine.
Type 'exit' to quit.

You: What is a VPN?
Bot: A VPN (Virtual Private Network) encrypts your internet traffic and routes it through a remote server, hiding your IP address and protecting your data from eavesdroppers on public networks. It's essential for privacy when using public Wi-Fi.

You: exit
Conversation saved to chat_history.json

Try it: Add streaming output, conversation search, and a web UI using Flask. Deploy it as a local network service for your home or office, keeping all data completely private.

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.