Set Up Ollama on a Mini PC: Local AI Server Guide

Private Ai Server Setup Guide — How to set up ollama on a mini pc: complete local ai server guide (private ai server setup guide)

Your mini PC can serve AI models to every device in your house—here’s how to get there in under 30 minutes. Ollama is the easiest way to run LLMs locally: free, open-source, and available on Linux, macOS, and Windows. This guide walks you from installation to a network-accessible AI server with a ChatGPT-like web UI. If you haven’t chosen hardware yet, see our best mini PC for AI buyer’s guide; for mini PCs for virtualization, we have a separate pillar. To confirm your machine has enough memory, check our RAM and VRAM requirements for running AI models locally.

Prerequisites

Hardware Requirements

Minimum 16 GB RAM (8 GB only handles tiny models). What you can run at each tier:

16 GB — 7B–8B parameter models (Llama 3.2 8B, Phi-4 3.8B). Comfortable for a single chat model.
32 GB — Up to 14B models (DeepSeek-R1 14B, DeepSeek-Coder-V2 16B). Good for coding assistants.
64 GB — Up to 32B models (Qwen 2.5 32B). The sweet spot for power users who want one large model loaded.
128 GB — 70B models (Llama 3.1 70B). Requires high memory bandwidth to be practical; DDR5 or LPDDR5X strongly recommended.

Before starting, check your GPU compatibility. NVIDIA cards need CUDA drivers (535+), AMD cards need ROCm 6.0+ (Linux only), and Apple Silicon uses Metal automatically. Arsturn’s Ollama hardware guide has a full compatibility checklist if you need to double-check.

Choose Your OS

Each operating system has trade-offs for running Ollama:

Ubuntu Server 22.04+ — Best for headless operation. Strongest Docker and GPU driver support, most community guides target Linux, and systemd makes auto-start trivial. If you plan to run other services alongside Ollama, this is the safest choice.
macOS 14+ — Simplest install experience. Metal GPU acceleration works automatically with zero configuration, and unified memory means all your RAM is available as “GPU memory.” The downside: no CUDA support and fewer Docker-native workflows.
Windows 11 — GUI-friendly with full CUDA support for NVIDIA GPUs. Slightly more overhead than Linux, and some advanced features (like systemd services) require workarounds. Good if you also use the machine as a desktop.

If you run Proxmox as your hypervisor, you can host Ollama inside a VM or LXC container instead. See our how to run AI models on Proxmox VE guide for that path.

Network Plan

Decide up front whether the server stays local-only or should be reachable from other devices. A DHCP reservation (set in your router) or a static IP on the mini PC avoids the headache of the address changing after a reboot. If you plan to install Open WebUI later, you’ll need port 3000 accessible from client devices. Ollama’s API listens on port 11434 by default—other tools and scripts on your network will hit that port to send prompts.

Step 1 — Install Ollama

Linux (Native)

curl -fsSL https://ollama.com/install.sh | sh

The install script downloads the Ollama binary, creates a dedicated ollama system user, and registers a systemd service so Ollama starts automatically on boot. It also auto-detects installed GPU drivers—if NVIDIA or AMD drivers are present, Ollama will use them without additional configuration. Verify the installation with ollama --version.

Linux (Docker)

docker run -d --gpus all -p 11434:11434 --name ollama ollama/ollama

Use Docker when you want isolation—especially inside a Proxmox LXC container, in multi-tenant environments, or when you want easy rollback (just remove the container and re-pull). The --gpus all flag passes NVIDIA GPUs into the container; make sure you have the NVIDIA Container Toolkit installed on the host. For AMD ROCm GPUs, replace --gpus all with --device=/dev/kfd --device=/dev/dri to expose the GPU devices directly. For GPU passthrough and Ollama on Proxmox VMs or LXC, see our how to run AI models on Proxmox VE—we don’t repeat those steps here.

macOS

Download from ollama.com, drag to Applications, and run the first-launch wizard. The app installs a CLI tool alongside the GUI—after the first launch, ollama is available in Terminal for all commands in this guide. Note that the first run downloads roughly 1 GB of runtime components, so give it a minute on slower connections.

Windows

Download the installer from ollama.com and run it. The installer adds ollama to your system PATH so it works from any PowerShell or Command Prompt window. You may see a Windows Defender SmartScreen prompt or a firewall dialog during installation—allow both. Verify in PowerShell with ollama --version.

Step 2 — Pull and Run Your First Model

The ollama pull command downloads GGUF-format model weights from the Ollama library to your local machine. File sizes range from about 2 GB for small models to 20+ GB for large ones, so the first pull takes a few minutes depending on your connection.

Pick a model based on your RAM and use case:

Quick test (any RAM): ollama pull phi4:3.8b — Smallest practical model, fast even on CPU. Good for verifying your install works.
General chat (16 GB): ollama pull llama3.2:8b — Solid all-around model for conversation, summarization, and Q&A.
Coding assistant (32 GB): ollama pull deepseek-coder-v2:16b or ollama pull deepseek-r1:14b — Strong at code generation, debugging, and technical writing.
Power user (64 GB): ollama pull qwen2.5:32b — Noticeably smarter responses; handles complex reasoning and long documents.

Once a model is downloaded, run it interactively:

ollama run llama3.2:8b

Type a prompt and watch. The first generation may take a few seconds to start while Ollama loads the model into memory. After that, tokens appear in a stream. On a 64 GB LPDDR5X mini PC with GPU offload, expect roughly 20–25 tok/s for 8B models. On CPU-only with DDR4, expect 5–10 tok/s. Press Ctrl+D to exit the interactive session.

Step 3 — Verify GPU Acceleration

NVIDIA (CUDA)

Run nvidia-smi to confirm your driver is loaded and shows your GPU. You need driver version 535 or newer for Ollama’s CUDA support. Ollama auto-detects NVIDIA GPUs at startup—check the logs to confirm:

journalctl -u ollama | grep -i cuda

You should see a line containing “using CUDA.” If Ollama fell back to CPU, the logs will say so, and you’ll need to update your driver or reinstall Ollama after the driver update.

AMD (ROCm)

ROCm is Linux only. Check that your GPU is visible with rocm-smi. You need ROCm 6.0 or newer installed. ROCm support has improved significantly, but can still be tricky on some cards—if Ollama doesn’t detect the GPU, verify that the amdgpu kernel module is loaded with lsmod | grep amdgpu. Some models may fall back to CPU even with ROCm present; check ollama ps to confirm the backend.

Apple (Metal)

Metal acceleration is automatic on macOS with Apple Silicon—no drivers to install, no flags to set. Verify by running ollama ps while a model is loaded; the backend column should show “metal.” Because Apple Silicon uses unified memory, there’s no separate “GPU memory”—your total system RAM is your GPU memory. A 64 GB M4 Pro can fully offload a 32B model into Metal.

CPU-Only Fallback

If no supported GPU is detected, Ollama falls back to CPU inference. This is still usable for light tasks—expect roughly 5–15 tok/s for 7B–8B models depending on your CPU and memory bandwidth. You can confirm the backend with ollama ps. If you expected GPU acceleration but see CPU, revisit the driver sections above.

Step 4 — Performance Tuning

GPU Layer Count

The environment variable OLLAMA_NUM_GPU (or num_gpu in a Modelfile) controls how many model layers are offloaded to the GPU. Set it to 99 to offload everything—this is the fastest option if your GPU has enough memory. If you’re running out of VRAM, lower the number to keep some layers on the CPU. Experiment to find the sweet spot: full GPU offload gives the best tok/s, but partial offload lets you run larger models than your VRAM alone would allow.

Context Length

Ollama defaults to a 2048-token context window. You can increase it with num_ctx:

ollama run llama3.2:8b --num_ctx 4096

Each doubling of context length roughly doubles the KV cache memory requirement. Going from 2048 to 8192 can add several gigabytes of RAM usage. The trade-off: longer context lets the model reference more of the conversation, but if RAM is tight, you’ll see slower inference or out-of-memory errors. Start at 4096 and increase only if you need it.

Keep-Alive

By default, Ollama unloads a model after 5 minutes of inactivity. You can change this with OLLAMA_KEEP_ALIVE:

OLLAMA_KEEP_ALIVE=30m — Keeps the model loaded for 30 minutes between requests. Good for intermittent use.
OLLAMA_KEEP_ALIVE=-1 — Keeps the model loaded forever (until you restart Ollama or manually unload it).
OLLAMA_KEEP_ALIVE=0 — Unloads immediately after each response.

Keeping a model loaded avoids a 5–15 second reload delay on the next request. If your mini PC is a dedicated AI server, -1 is usually the right choice.

Memory Management

The OLLAMA_MAX_LOADED_MODELS variable controls how many models can be loaded simultaneously. The default is 1. If you set it to 2 or higher, Ollama will keep multiple models in RAM so you can switch between them without reload delays. Just make sure you have enough RAM for all of them—two 8B models need roughly 12–14 GB together, while an 8B and a 14B need about 16–18 GB.

Step 5 — Set Up Open WebUI (ChatGPT-Like Interface)

Open WebUI gives you a browser-based chat interface that looks and feels like ChatGPT. Docker is required.

docker run -d -p 3000:8080 
  --add-host=host.docker.internal:host-gateway 
  -v open-webui:/app/backend/data 
  --name open-webui 
  ghcr.io/open-webui/open-webui:main

The flags: -p 3000:8080 maps port 3000 on your host to the container’s internal port 8080. --add-host=host.docker.internal:host-gateway lets the container reach Ollama running on the host machine. The -v open-webui:/app/backend/data volume persists your conversations, settings, and uploaded documents across container restarts.

Open http://your-mini-pc-ip:3000 in a browser. On first visit, create an admin account. Then go to Settings → Connections and enter the Ollama URL: http://host.docker.internal:11434. Hit the check button to verify the connection, then go back to the main chat, select a model from the dropdown, and send a test message.

Open WebUI packs a lot of features beyond basic chat. You can switch between any models you’ve pulled in Ollama, browse full conversation history, set system prompts to create custom personas (e.g., “You are a Python tutor”), upload documents for retrieval-augmented generation (RAG), and manage multiple user accounts if other people in your household want access.

Ollama step-by-step tutorial: install, run models locally, and set up Open WebUI.

Step 6 — Expose Ollama on Your Network

Set OLLAMA_HOST

By default, Ollama only listens on localhost. To make it accessible from other devices on your network, set OLLAMA_HOST=0.0.0.0:

Linux: Edit /etc/systemd/system/ollama.service and add Environment="OLLAMA_HOST=0.0.0.0" under the [Service] section. Then reload and restart: sudo systemctl daemon-reload && sudo systemctl restart ollama.
macOS: Run launchctl setenv OLLAMA_HOST 0.0.0.0, then restart the Ollama app.
Windows: Add OLLAMA_HOST as a system environment variable with a value 0.0.0.0 through System Properties → Environment Variables. Restart the Ollama service.

Reverse Proxy with nginx

Putting nginx in front of Ollama gives you HTTPS, request logging, and rate limiting. A minimal config proxies requests to Ollama’s local port:

server {
    listen 443 ssl;
    server_name ai.local;
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    location / {
        proxy_pass http://127.0.0.1:11434;
    }
}

This is optional for a home network but recommended if you expose Ollama beyond your LAN.

Firewall

Open the required ports, so other devices can reach Ollama and Open WebUI:

Linux (UFW): sudo ufw allow 11434 and sudo ufw allow 3000.
Windows: Add inbound rules in Windows Defender Firewall for TCP ports 11434 and 3000.

Step 7 — Run as a System Service (Auto-Start)

Linux (systemd)

The native install script already creates a systemd unit. Enable and start it with:

sudo systemctl enable --now ollama

Check status with systemctl status ollama. Stream live logs with journalctl -u ollama -f—useful for watching model loads, GPU detection, and API requests in real time.

macOS (launchd)

The Ollama desktop app auto-starts at login by default—open System Settings → General → Login Items to verify. For a headless Mac mini (no GUI session), create a launchd plist in ~/Library/LaunchAgents/ that runs /usr/local/bin/ollama serve at boot, or simply add Ollama to Login Items in System Settings.

Windows (Task Scheduler)

Open Task Scheduler and create a new task: set the trigger to “At log on,” set the action to start ollama serve, and check “Run whether user is logged on or not” if you want it running as a background service. For more robust service behavior, use NSSM (Non-Sucking Service Manager)—it wraps any executable as a proper Windows service with automatic restart on failure.

Step 8 — Remote Access (Optional)

Tailscale

Tailscale creates a private WireGuard VPN mesh between your devices with zero firewall configuration. Install Tailscale on the mini PC and on any client device (phone, laptop, tablet). Each device gets a stable Tailscale IP in the 100.x.x.x range. Point your browser or API client at http://100.x.x.x:11434 to reach Ollama from anywhere—coffee shop, office, or another city. Tailscale is free for personal use with up to 100 devices.

Cloudflare Tunnel

Cloudflare Tunnel (cloudflared) creates a secure outbound-only tunnel from your mini PC to Cloudflare’s edge, so you don’t need to open any ports on your router. Install cloudflared, create a tunnel, and assign a subdomain (e.g., ai.yourdomain.com). Add Cloudflare Access on top for authentication—this puts a login page in front of your Ollama instance, so only authorized users can reach it. The free tier covers personal use.

Monitoring and Updating

Use ollama list to see all downloaded models with their name, size, and last-modified date. Use ollama ps to see which models are currently loaded in memory, how much RAM/VRAM they’re consuming, and which backend (CUDA, Metal, CPU) they’re running on. To free disk space, remove models you no longer need with ollama rm <model>.

Check for updates regularly—new Ollama releases bring performance improvements, new model support, and bug fixes. On Linux, re-run the install script: curl -fsSL https://ollama.com/install.sh | sh. On macOS and Windows, download the latest installer from ollama.com. On Linux, check logs anytime with journalctl -u ollama.

Troubleshooting

GPU Not Detected

Check that your driver version meets Ollama’s minimum requirements (NVIDIA 535+, ROCm 6.0+). If you’re using Docker, make sure you passed the --gpus all flag (NVIDIA) or --device=/dev/kfd --device=/dev/dri (AMD). For ROCm, verify the amdgpu kernel module is loaded with lsmod | grep amdgpu. If you installed or updated GPU drivers after installing Ollama, reinstall Ollama so it picks up the new drivers.

Out of Memory

Symptoms include a crash, “not enough memory” error, or the system becoming unresponsive. First, check the model’s actual size with ollama show <model> before pulling. If you’re already out of memory, try a smaller quantization (switch from Q4_K_M to Q3_K_S for roughly 20% less RAM), drop to a smaller model size (14B instead of 32B), or close other applications to free RAM. Reducing context length also helps.

Slow Inference

Verify that GPU offload is active by checking ollama ps—if it shows CPU backend, you’re not getting GPU acceleration. Memory bandwidth matters: DDR4 is significantly slower than DDR5 or LPDDR5X for model inference. Reduce context length if you’ve set it high. As a baseline test, try a small model like phi4:3.8b first—if that’s fast but larger models are slow, the bottleneck is memory capacity or bandwidth.

Can’t Connect from Network

Run through this checklist: OLLAMA_HOST is set to 0.0.0.0 (not localhost); you restarted the Ollama service after changing it; your firewall allows TCP on port 11434; and the client is using the correct IP address of the mini PC (check with ip addr on Linux, ipconfig on Windows, or ifconfig on macOS).

FAQ

Can I run Ollama in a Proxmox VM or LXC?

Yes. For VMs, use PCI passthrough to give the VM direct GPU access. For LXC containers, run Ollama inside Docker within the container. Full steps including GPU passthrough configuration are in our Run AI models on Proxmox VE guide (linked in Step 1).

How many models can I load at once?

By default, Ollama loads one model at a time and unloads it before loading another. You can change this with OLLAMA_MAX_LOADED_MODELS—set it to 2 or 3 if you have the RAM. Each loaded model consumes memory independently, so two 8B models need roughly 12–14 GB together.

Is Ollama free?

Yes, fully open-source and free for personal and commercial use. There are no usage limits, no API keys, and no telemetry. You download models from Ollama’s public library at no cost.

Can I use Ollama with VS Code or other IDEs?

Yes. Ollama exposes a local API at http://localhost:11434 that works with IDE extensions like Continue, Cody, and Copilot-compatible plugins. Point the extension at your Ollama endpoint and select a model—code completions and chat run entirely on your mini PC.

Does Ollama support RAG?

Not as a built-in feature. Ollama handles model inference only. For retrieval-augmented generation, use Open WebUI (which has document upload built in) or pair Ollama with a framework like LangChain or LlamaIndex that handles document chunking and embedding.

How do I switch between models?

From the command line, run ollama run <model-name> to start a different model. In Open WebUI, select a model from the dropdown at the top of the chat. Ollama automatically unloads the previous model unless you’ve configured keep-alive or increased OLLAMA_MAX_LOADED_MODELS.

What port does Ollama use?

Port 11434 by default for the API. Open WebUI, if installed, typically runs on port 3000 (mapped from its internal 8080). Both are configurable.

Can I run Ollama alongside VMs?

Yes, just make sure you leave enough RAM for both. If you’re running Proxmox, consider running Ollama inside a dedicated LXC container rather than on the host—this keeps resource allocation cleaner and lets you set memory limits per container.

Conclusion

That’s the full path: install → first model → GPU check → tuning → Open WebUI → network and service. The whole process takes about 30 minutes, and once it’s running, your mini PC serves AI to every device on your network without sending a single prompt to the cloud. For additional mini PC picks oriented toward Ollama, Mayhemcode’s 2026 Ollama mini PC roundup is worth a look. Don’t have a mini PC yet? Start with our best mini PC for AI guide (linked in the intro).