NVIDIA NIM on Unraid

Run NVIDIA NIM inference microservices locally on Unraid using Docker. This guide covers setup, common errors, and connecting OpenAI-compatible clients.

Table of Contents

• Prerequisites
• Model Selection
• NGC Registry Login
• Docker Template Setup
• Environment Variables
• First Run
• Connecting Clients
• Switching Models
• Troubleshooting

Prerequisites

• Unraid 6.12 or later
• NVIDIA GPU (Turing architecture or newer — GTX 16xx, RTX 20xx+)
• NVIDIA drivers installed in Unraid (Community Applications → NerdTools or GPU Statistics plugin)
• Free NGC account at build.nvidia.com
• NGC API key generated at your NGC account dashboard

Tested on: RTX 3060 12 GB · Unraid 6.12+ · NIM 1.10.1

Model Selection

NIM uses pre-optimized engine profiles. Consumer GPUs require smaller models and reduced context windows. Below are Examples.

For 7B+ models on a 12 GB consumer GPU, consider Ollama instead — it uses quantized weights and fits comfortably.

NGC Registry Login

⚠️ This must be done before Unraid can pull NIM images. NIM images are hosted on NVIDIA's private registry (nvcr.io), not Docker Hub.

One-time login via Unraid terminal

docker login nvcr.io
# Username: $oauthtoken       ← type this literally
# Password: YOUR_NGC_API_KEY

Persist login across reboots

Add to /boot/config/go (runs at every boot):

docker login nvcr.io -u '$oauthtoken' -p 'YOUR_NGC_API_KEY'

Note: docker login (image pull auth) and the NGC_API_KEY environment variable (runtime model weight download auth) are two separate authentications. Both are required.

Docker Template Setup

In the Unraid Docker GUI, click Add Container and fill in the following fields.

Basic Settings

Port Mapping

Volume Mapping

Before starting the container, create the cache directory with correct permissions:

mkdir -p /mnt/user/appdata/nvidia-nim/cache
chown -R 1000:1000 /mnt/user/appdata/nvidia-nim/cache
chmod 775 /mnt/user/appdata/nvidia-nim/cache

Environment Variables

First Run

On first start, NIM downloads model weights to the cache directory (~6 GB for the 3B model). This can take several minutes depending on your connection.

Watch the logs in the Unraid Docker UI or via terminal:

docker logs -f nvidia-nim

A successful startup looks like:

INFO:     Uvicorn running on http://0.0.0.0:8000

You can also verify the API is running:

curl http://localhost:8000/v1/models

Connecting Clients

NIM exposes an OpenAI-compatible API. Use these settings in any compatible client:

Compatible clients

• AnythingLLM
• Open WebUI
• LangChain
• LlamaIndex
• Cursor (custom OpenAI base URL)
• Any app with a configurable OpenAI-compatible endpoint

Quick test

curl http://[unraid-ip]:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.2-3b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Switching Models

NIM images for the template are currently model-specific — there is no in-app model browser. To switch:

1. Stop the existing container
2. Update the Repository field to the new model image (e.g. nvcr.io/nim/microsoft/phi-3-mini-4k-instruct:latest)
3. Update NIM_MODEL_NAME to match (e.g. microsoft/phi-3-mini-4k-instruct)
4. Start the container

To run multiple models simultaneously, create separate containers on different host ports (e.g. 8000, 8001). They can share the same cache folder — weights are not duplicated if the same model is used.

Troubleshooting

CUDA_VISIBLE_DEVICES must be numeric

Error:

If you get following "ValueError: invalid literal for int() with base 10: 'all'" it's probably becuase you changed value to all!

Fix: Set CUDA_VISIBLE_DEVICES=0 (not all). The --gpus all flag in Extra Parameters handles Docker-level GPU exposure separately.

Cache directory permission denied

Error:

The container will launch after creation you will probably get the following "PermissionError: [Errno 13] Permission denied: '/opt/nim/.cache/local_cache'"

Fix:

mkdir -p /mnt/user/appdata/nvidia-nim/cache
chmod 775 /mnt/user/appdata/nvidia-nim/cache

KV cache size error

Error:

If you get something like "ValueError: The model's max seq len (131072) is larger than the maximum number
of tokens that can be stored in KV cache (30320)".

Fix: Set NIM_MAX_MODEL_LEN=16384. If the error persists, try 8192. Or

Consumer GPUs cannot accommodate the full context window that data center profiles request. This variable caps it to a size that fits in available VRAM.

General error reference

XML Template

An Unraid Community Applications-compatible XML template is included in this repo as nvidia-nim.xml. You can place it in /boot/config/plugins/dockerMan/templates-user/ on your Unraid server to have it appear in the Docker template list.

Resources

• NVIDIA NIM Documentation
• NGC Model Catalog
• NIM API Reference
• Unraid Docker Documentation