llamaMan

What it is. A browser UI + API front-end for running multiple llama.cpp server instances, across one machine or a small cluster of them. Ships an Ollama-compatible proxy so it drops in for Open WebUI.

What's different. llamaMan itself has no llama.cpp code and no GPU dependency - it spawns ghcr.io/ggml-org/llama.cpp:server-* containers as siblings over the Docker socket. Nodes are heterogeneous (each has its own GPUs, presets, images), a shared queue can route the same model name to the least-loaded peer, and every launch knob (spec-decoding, MoE offload, KV quant, flash-attn, load-mode, mmproj, PDF input) is a first-class UI field. Update llama.cpp without touching llamaMan: docker pull a newer server-* image.

llama-server flag semantics are the source of truth on llama.cpp's side: server CLI reference.

Contents

• Features · Architecture · Request Flow · Design Decisions
• Quick Start · Authentication · Models · Launching Instances · Launch settings reference
• Image & PDF Input · Anti-Loop · Per-Instance Proxy · Idle Timeout · GPU Stats
• Request Recording & Stats · Model Eviction · OpenWebUI
• Storage & DB Outage Mirror · Clustering
• Environment Variables · REST API · Troubleshooting

Features

• Universal GPU image - one flow for NVIDIA, AMD (ROCm), Intel Arc, CPU; vendor auto-detected (GPU_TYPE / LLAMA_IMAGE override)
• Runs in Docker or bare-metal - auto-detected; both reach spawned containers correctly
• Model library + downloader - HuggingFace pulls with speed limits, resume, auto-retry, republish detection with atomic re-pull
• One-click launch + presets - per-model settings; live updates for fields that don't need a relaunch (idle-timeout, gates, sampling overrides)
• Speculative decoding - all five draft families (draft-simple/-mtp/-dflash/-dspark/-eagle3) with advanced knobs (n-max/n-min/p-split/p-min)
• MoE offload - --cpu-moe / --n-cpu-moe exposed as a sentinel int; shrinks big-MoE VRAM ~4-10x
• Flash Attention + KV cache quant + reasoning format + load mode - all exposed; UI enforces the quantized-V-requires-FA-on constraint
• Anti-Loop - DRY sampler plus proxy-side output loop detection that hard-kills the turn when a chunk repeats too often; both off by default, tuned per preset
• Image & PDF input - --mmproj for vision models; PDFs rasterized page-by-page (or inlined as text via the born-digital shortcut); OpenAI image_url/file and Ollama images[] both work
• Instance monitoring - per-GPU VRAM, container CPU/RAM, per-instance TTFT / throughput / latency rolled up from the request log
• Ollama + OpenAI proxy on :42069 - auto-starts models on demand; LRU-evicts once LLAMAMAN_MAX_MODELS is hit
• Auth - user accounts, API keys (bearer tokens), per-endpoint auth toggle
• Persistent state - JSON (default) or MariaDB/MySQL; MariaDB unlocks clustering + optional local DB-outage mirror
• Multi-node clustering (opt-in) - heterogeneous nodes as one logical cluster; aggregated dashboard; shared-queue load balancing
• Request recording + logging dashboard - opt-in per-request or per-conversation, with retention
• Per-model display names - friendly name API clients (OpenWebUI) see and accept instead of the raw quant filename
• Docker image management - pull any llama.cpp image by name; delete old local images from the UI

Architecture

llamaMan is a small Python app that never touches llama.cpp code directly. It spawns each llama-server as a sibling container over the Docker socket, monitors the fleet, and hands out an OpenAI/Ollama front door.

flowchart LR
  client[Client<br/>Open WebUI / any OpenAI or Ollama client]

  subgraph nodeA[Node A]
    lmA[llamaman<br/>:5000 UI · :42069 compat]
    subgraph workersA[llama-server siblings on host GPUs]
      wA1[qwen2.5-14b]
      wA2[gpt-oss-20b]
    end
  end

  subgraph nodeB[Node B]
    lmB[llamaman]
    subgraph workersB[llama-server siblings]
      wB1[qwen2.5-14b]
      wB2[deepseek-v3]
    end
  end

  db[(Shared storage<br/>presets · instance state · heartbeats)]

  client -->|prompt| lmA
  lmA -->|forward when a peer is less loaded| lmB
  lmA --> wA1
  lmA --> wA2
  lmB --> wB1
  lmB --> wB2
  lmA <-.-> db
  lmB <-.-> db

The single-node picture (default) is just the top half - one llamaman container, N llama-server siblings on the local Docker daemon, no cluster peers, JSON files instead of the shared DB. Clustering is opt-in and additive.

Containerized vs bare-metal. llamaMan runs in Docker by default (llamaman-net, reaching siblings by container name) or bare-metal on the host (reaches spawned containers via localhost on their published ports). Mode is auto-detected; force with LLAMAMAN_IN_DOCKER.

Heterogeneous clusters. Every node has its own hardware (GPU count/vendor/VRAM), its own model files on disk, its own llama.cpp image, and its own hardware-scoped preset overrides. The shared DB carries presets, users, API keys, per-node instance state, and cluster heartbeats - not the model files.

Queue-aware dispatch + cross-node work stealing. When two nodes serve the same model (or the same share_queue_group alias), an inbound request for that name gets routed to whichever peer has the fewest in-flight requests. If the picked node is saturated its RequestGate still tries to hand the request off to a peer with free capacity instead of queueing.

Model loading + LRU eviction. The :42069 compat proxy auto-launches models on demand. Once LLAMAMAN_MAX_MODELS is hit, the least-recently-used non-admin instance is stopped to make room. Sleeping instances still count against the slot; idle timeout is a separate axis.

Resilient storage. MariaDB is on the critical path of every request (auth reads settings and keys per call). A local JSON mirror in DATA_DIR/db_mirror/ write-throughs each node-scoped update; when the breaker opens, reads fall back to the mirror and writes are journalled + replayed on reconnect. Off by default, one setting to turn on.

Multimodal. Vision models load their --mmproj projector alongside the main GGUF. OpenAI image_url / file and Ollama images[] both work. PDFs are rasterized page-by-page, or fast-pathed as extracted text when the PDF is born-digital.

Failure behavior. A dead container is detected by the poller and marked, and the instance's slot is freed on the next tick. A killed llamaman process, on restart, re-adopts labelled containers that are still up (their gate, sidecar proxy, and log-tail relay are all rebuilt), so no --restart=always gymnastics needed. A peer node that misses its heartbeat window drops out of dispatch until it comes back.

Update llama.cpp without rebuilding llamaman: pull a newer server-* image from the UI (Settings → Docker Images), or hit the pull endpoint. New instances launch on the new image; existing ones keep running until you restart them.

Request Flow

A concrete OpenAI-compatible chat request against a two-node cluster:

1. Client POST /v1/chat/completions to nodeA:42069 with model qwen2.5-14b.
2. nodeA looks up the target: it's a live model on this node and on nodeB. Both belong to the same effective queue group.
3. Dispatcher polls each candidate's live load (from the heartbeat cache + short-window inflight counter). nodeB is less loaded → the request is forwarded to nodeB:42069.
4. nodeB claims a gate slot (429 if the queue is full and no peer has room). If sampling overrides are on, they're injected. If a per-instance sidecar exists (idle / max-concurrent / proxy-sampling), the request lands on the sidecar first, which wakes the container if it's sleeping.
5. Sidecar forwards to llama-server on its internal port. Stream flows back through the sidecar → nodeB's proxy → nodeA → client.
6. On stream end: the request-log worker records TTFT / throughput / final token count; the gate slot is released; the LRU timestamp is bumped so this instance won't get evicted next.

If nodeB had gone offline between the heartbeat and this call, the dispatcher would have detected the connection error and re-picked the next candidate (locally in this case).

Design Decisions & Trade-offs

• One gunicorn worker, gthread, preload_app=False. Live state (instances, gates, downloads, idle proxies) is in-memory Python dicts; multi-worker would fragment it. Concurrency comes from threads (threads=32 - sized for cluster-forwarded generations holding a thread for their whole duration). Trade-off: no horizontal scale within one host; scale by adding cluster nodes instead.
• LLAMAMAN_NODE_NAME is mandatory. It's the storage partition key. Changing it after first boot orphans that node's rows. Trade-off: one extra env var required for even single-node installs, but a shared DB stays sane across a cluster and legacy rows are auto-adopted on first boot.
• Entry node holds the connection. A forwarded request keeps a thread on both the entry node and the target for the whole generation - no client-side dispatch, so any OpenAI/Ollama client works unchanged. Trade-off: the entry node's thread pool caps concurrent cross-node inference.
• Sentinel forwarding. llama.cpp values like -2 = all for --n-gpu-layers or -1 = --cpu-moe are passed through verbatim rather than being translated on our side. Trade-off: fragile against upstream changing what those sentinels mean, but zero translation drift and every llama-server release lands the same day.
• In-memory state, DB is the mirror. Runtime truth is the module-level dicts under a lock; the DB is a durable mirror. Trade-off: a crash between save points loses seconds, not requests - every state-changing call snapshots to the backend before returning.
• ResilientBackend is off by default. With a shared MariaDB, one setting turns on a local JSON mirror + write journal so a DB outage doesn't 500 every request or fail startup. Off by default so single-DB deployments keep the simpler code path. Trade-off: mirror mode reconciles wholesale for node-scoped rows on reconnect, so it doesn't fit a cluster where two nodes concurrently mutate the same partitioned data (they don't, by design).

Requirements

• Docker with access to /var/run/docker.sock
• One of:
  • NVIDIA Container Toolkit for NVIDIA GPUs
  • ROCm-compatible setup for AMD GPUs
  • Intel Arc with /dev/dri access
  • CPU-only

Quick Start

Edit docker-compose.yml first:

• Set HOST_MODELS_DIR / HOST_LOGS_DIR to the real host paths that back your volume mounts (llamaMan passes them to the Docker daemon when spawning sibling containers).
• LLAMAMAN_NODE_NAME (default srv1) is required for every install - it's the partition key for this deployment's stored state. Pick once, keep it (changing later orphans stored state).

Pull the llama.cpp image for your GPU and start:

docker pull <LLAMA_IMAGE>          # from the table above
# For non-CUDA: also set LLAMA_IMAGE=... in docker-compose.yml
docker compose up --build

For NVIDIA native VRAM monitoring, also uncomment deploy.resources.reservations in docker-compose.yml.

• Management UI: http://localhost:5000 (create admin on /setup)
• Ollama-compatible API: http://localhost:42069
• llama-server instance ports: 8000-8020

Security note: the mounted /var/run/docker.sock gives the container full control of Docker on the host. Standard Docker-socket-access implications apply.

Running bare-metal

Useful for dev or hosts (e.g. WSL) where running the manager itself in Docker is awkward. llama-server still runs in containers.

pip install -r requirements.txt
MODELS_DIR=./models DATA_DIR=./data LOGS_DIR=./logs \
  LLAMAMAN_NODE_NAME=dev python app.py
# or under gunicorn:
MODELS_DIR=./models DATA_DIR=./data LOGS_DIR=./logs \
  LLAMAMAN_NODE_NAME=dev gunicorn -c gunicorn.conf.py app:app

A single process serves the UI/API on :5000 and the proxy on :42069. Container-vs-bare-metal is auto-detected; if detection is ever wrong for your runtime, set LLAMAMAN_IN_DOCKER=true or false explicitly.

Authentication

Two credential types:

• Session cookies for the UI (created via /setup on first launch)
• API keys for external clients (Authorization: Bearer llm-xxxxxxxxxx), managed under API Keys in the UI

The "Require authentication for all endpoints" toggle (on by default) controls whether model-serving endpoints demand a bearer token:

All three port surfaces (5000, 42069, 8000-8020) go through the same auth hook.

OpenWebUI with authentication

open-webui:
  environment:
    - OLLAMA_BASE_URL=http://llamaman:42069
    - OPENAI_API_BASE_URLS=http://llamaman:42069/v1
    - OPENAI_API_KEYS=llm-your-api-key-here

Models

Place models under the /models volume as .gguf files, HuggingFace repo directories (containing config.json), or use the Download button in the UI.

Display Name

Each model can be given a friendly Display Name (in the Launch form). When set it's the id API clients see on /api/tags and /v1/models, so OpenWebUI shows Qwen 2.5 14B instead of Qwen2.5-14B-Instruct-Q4_K_M. Must be unique and not clash with another model's filename or a cluster queue-group name.

Model updates

The Check for updates button (shown for models with a known source repo) compares the local file against the repo's published content hash:

• Up to date - recorded hash matches
• Update available - re-pulls into a staging folder and atomically swaps in only once the download completes (a failed pull leaves the current model intact). Refuses with 409 while the model is loaded.
• Verify hash - hashes the local file (background job with progress) when no hash has been recorded yet

An opt-in background scan (Settings → Download Settings) keeps these answers ready ahead of time.

Launching Instances

1. Select a model from the sidebar
2. Configure launch settings (or use the saved preset)
3. Click Launch - llamaMan spawns a llama-server container and the instance appears with a status badge
4. Optionally Save Preset to remember settings

The launch form's action row also carries three tool buttons on the right:

• Export preset - download the current form values as a JSON file (portable across nodes; does not save to storage).
• Import preset - load a preset JSON into the form. Populates fields only; you still hit Save Preset to persist.
• Copy llama.cpp command - copies the equivalent llama-server ... invocation for the current form values to your clipboard. Rendered server-side through the same flag-emission code the real launch uses, so it stays in sync as flags evolve.

With settings collapsed, a Quick Launch button starts the selected model straight from its preset.

Each instance exposes an OpenAI-compatible API on its assigned port. When a GGUF is selected, llamaMan reads its metadata to detect layer count and shows it next to GPU Layers (e.g. / 32).

Launch settings reference

Behavior notes below are terse; hover the field's info-tip in the UI for the full explanation. All flags map to llama-server unless stated - see the llama-server CLI reference for exact semantics and current defaults.

Model Settings

Container & Proxy Settings

GPU Settings

Extra Args (text field) - additional flags passed directly to llama-server (e.g. --mlock, or n-gram speculative types not surfaced in the form).

Proxy-Side Sampling Overrides (toggle) - when on, the proxy forces sampling params on every forwarded request:

Speculative Decoding (toggle)

Image & PDF Input (toggle) - see Image & PDF Input below.

Anti-Loop — two independent sub-toggles (both off by default). See Anti-Loop below.

DRY Sampler (baked in at launch — requires relaunch):

Output Loop Detection (proxy-side; live-updated from preset — next request picks up new thresholds):

Applies to Ollama / OpenAI compat routes (:42069) always, and to per-instance ports only when a sidecar proxy is already present (Idle Timeout, Max Concurrent, or Proxy Sampling Overrides also on). On detection, llamaman closes the upstream connection (which stops llama-server generating) and injects a terminator into the client stream — OpenAI clients see a finish_reason=stop chunk with a [llamaman: output loop detected, ending turn] marker; Ollama clients see done_reason: "loop_detected".

Live preset updates

Saving a preset updates already-running instances of that model in place where possible:

• Live (no relaunch): idle_timeout_min, max_concurrent, max_queue_depth, share_queue, all six proxy-sampling fields, all five loop-detection fields (next request picks them up)
• Require relaunch: everything baked into the container at launch (gpu layers, MoE CPU layers, ctx size, threads, threads-batch, cache types, flash-attn, reasoning-format, load-mode, DRY sampler, spec-decoding fields, embedding flag, mmproj/PDF fields, extra args, memory limit)

Proxy-sampling caveat: if the instance was launched with all of idle_timeout=0, max_concurrent=0, and override_enabled=false, no sidecar proxy was spawned. Live-toggling override_enabled=true then still applies overrides on requests routed through the app's compat endpoints, but direct hits to the public port bypass it. Relaunch to spawn the proxy in that case.

Concurrency & queueing

When Max Concurrent > 0, llamaMan places a FIFO gate in front of the instance. Excess requests wait up to Max Queue Depth before returning HTTP 429. Gate stats (active + queued) are visible via the API.

Parallel vs Max Concurrent: Parallel controls llama-server's internal KV slots; Max Concurrent gates how many requests llamaMan forwards at once. Use both together (e.g. Parallel=4, Max Concurrent=4) to keep the server well-fed without overflowing its slots.

Image & PDF Input

Enable the Image & PDF Input toggle to load a vision model's multimodal projector (--mmproj) alongside its main GGUF. The MMPROJ Model field takes the projector GGUF (typical filename: mmproj-<model>-<precision>.gguf).

Offload mmproj to GPU (on by default): llama.cpp keeps the projector on the GPU unless it gets --no-mmproj-offload, so on omits the flag entirely (byte-identical CLI to before this option existed) and off passes --no-mmproj-offload to run the vision encoder on CPU — useful on tight-VRAM setups, where the encoder pins a few GB.

PDFs are supported by rasterizing each page to a PNG before forwarding to llama-server (which has no native PDF support). Enable Accept PDF uploads to turn this on for an instance. Recognized in both:

• OpenAI: image_url with a data:application/pdf;base64,... URL, or the newer file block
• Ollama: base64 PDF bytes mixed into images[] - llamaMan sniffs %PDF- and expands PDFs; ordinary images keep their existing path

Text-layer shortcut (off by default): with Try text layer first on, llamaMan runs pypdf.extract_text() before rasterizing. Born-digital PDFs (Word/LaTeX/browser print) with substantive text (≥ ~50 chars/page) get inlined as one text block and skip the vision path entirely. Sparse text falls back to rasterization. Off by default because rasterization preserves layout for the vision model, and predictable behavior on every PDF beats a silent shortcut.

Per-request caps: PDF DPI (72-600, default 200) and Max PDF Pages (1-200, default 20). Over-cap PDFs return HTTP 400 rather than being truncated.

Rasterization concurrency: PDF rendering is CPU- and RAM-heavy and runs in the gunicorn worker thread before any per-instance gate. A process-wide semaphore caps concurrent rasters; tune via LLAMAMAN_PDF_MAX_CONCURRENT (default 4).

Anti-Loop

Two independent controls in the Anti-Loop section of the launch form, both off by default. They target the same problem (models getting stuck in stable output loops) at two different points in the pipeline; use them together for the best coverage.

DRY sampler (soft, sampling-time). Maps to llama-server's --dry-multiplier / --dry-base / --dry-allowed-length / --dry-penalty-last-n. Every token that would extend a recently-seen n-gram gets a probability penalty at sampling time, so the model is nudged away from repeating itself before it ever emits a duplicate span. This is the soft line of defense — usually enough on its own for prose. Baked in at container launch; a preset edit requires a relaunch. Multiplier = 0 disables entirely (llama.cpp's own default).

Output loop detection (hard, post-hoc). When DRY isn't enough — the model still gets stuck in a stable attractor — llamaman's proxy watches the assistant-visible output text (content + reasoning) as it streams to the client. Every scan cadence (default: inline every 64 chunks, plus a 10s worker fallback), the last min_chunk_chars of the buffer are checked for repetition. If they appear ≥ min_repetitions times, the turn is terminated:

• The upstream connection to llama-server is closed — llama-server's stop-on-client-disconnect halts generation, freeing the model for the next request.
• A synthetic terminator is injected into the client's stream — OpenAI clients see one final delta with finish_reason=stop and content [llamaman: output loop detected, ending turn]; Ollama clients see done_reason: "loop_detected" on a done: true line.

Where it watches:

• Compat routes on :42069 (Ollama /api/chat, /api/generate; OpenAI /v1/chat/completions, /v1/completions) — always, whenever the toggle is on for the target instance's preset.
• Per-instance ports (8000-8020) — only when a sidecar proxy is already present for that instance (i.e. Idle Timeout, Max Concurrent, or Proxy Sampling Overrides also on). Loop detection does not, by itself, spawn a sidecar proxy — enable one of the other three to also watch direct hits.

Tuning:

• min_chunk_chars (60-4096, default 200) — the smallest repeating period that will trigger. Lower values catch shorter loops but risk false positives on repetitive-but-legitimate content (numbered lists, poetry choruses, markdown tables). 60-100 is aggressive; 200 is the balanced default for prose; 400+ only triggers on paragraph-scale loops.
• min_repetitions (2-20, default 3) — how many exact copies of the chunk must appear before terminating. 2 fires on any doubled chunk (aggressive); 3-5 balances responsiveness against false positives.
• max_buffer_chars (512-65536, default 8192) — rolling window kept per active request. Must be at least min_chunk_chars × min_repetitions.
• scan_every_n_tokens (8-4096, default 64) — inline scan cadence. Lower = faster kill, more CPU.
• scan_interval_s (1-600, default 10) — how often the background worker rescans buffers that didn't hit the inline threshold.

Changes to the loop-detection thresholds take effect on the next request (in-flight streams keep the thresholds captured at attach time). DRY changes require a relaunch since the flags are baked into the container.

Detection algorithm (v1): take the last min_chunk_chars of the buffer as the target; count exact occurrences; flag when ≥ min_repetitions. Doesn't catch drifting near-repeats ("The answer is: The answer is: The answer is:" with slight punctuation drift between reps) — those get caught by DRY's sampling-time defenses. Deliberately simple and cheap.

Defensive invariant: every hook in the streaming path is wrapped in try/except Exception that logs and falls through — a detector bug can never break a real user's stream.

Per-Instance Proxy

When Idle Timeout, Max Concurrent, or Proxy Sampling Overrides are on for an instance, llamaMan inserts a WSGI proxy in front of the llama-server container: the public port (e.g. 8000) is the proxy; llama-server listens on a separate internal port. The proxy does model-name validation, wake-on-request for sleeping instances, RequestGate concurrency limiting, and sampling injection.

Model-name validation: on inference endpoints, if the request body has a "model" field it's compared against the loaded model's filename stem (lowercased, no extension). Prefix match is accepted (e.g. "qwen2.5-0.5b-instruct-q2" matches "qwen2.5-0.5b-instruct-q2_k"). Mismatch returns HTTP 404; for sleeping instances, no wake. Requests with no "model" field are forwarded unconditionally.

Idle Timeout

Set Idle Timeout min in the launch form (0 = disabled). When enabled, after N minutes with no requests the llama-server container is stopped to free VRAM; the next request wakes a new container with the same config. Clients see the same port with a cold-start delay.

For proxy-managed (OpenWebUI) instances, use LLAMAMAN_IDLE_TIMEOUT instead.

GPU Stats

Per-GPU VRAM and utilization, queried natively - no running llama-server required.

Without native access, llamaMan falls back to exec-ing nvidia-smi / rocm-smi inside a running container. Stats always reflect full host GPU state.

Request Recording & Stats

Under Settings → App Settings → Request recording: Off (default) / Per request / Per conversation (turns grouped by a content hash of the system prompt + first user message). Each record captures the bodies + envelope fields (model, endpoint, status, duration, token counts) plus accurate per-turn metrics (throughput measured over the generation window; TTFT). Records live under RECORDINGS_DIR (JSON) or the request_log table (MariaDB). Retention (days) prunes hourly (0 = keep forever).

Each instance card exposes a Stats button (request count, avg/peak throughput, avg TTFT/latency, token totals, active time span) rolled up from the request log - so it persists after the instance is stopped and shows an empty state when recording is off. The Logging link in the header opens a full-page dashboard (summary tiles, recent conversations, per-conversation drill-down) with a 24h / 7d / 30d / All time-window selector plus per-model and per-day filters over the conversation list.

Model Eviction

LLAMAMAN_MAX_MODELS caps concurrent chat models via the proxy. When the cap is full and a new chat model is requested, the LRU chat model is evicted:

Three toggles under Settings → App Settings relax these defaults:

• Enforce LLAMAMAN_MAX_MODELS for admin UI launches - silent LRU-evict before launch (Ollama-managed first). Off by default → UI prompts first.
• Allow Ollama API to evict admin-launched models - fallback when no Ollama-managed models are available. Off by default.
• Allow OpenAI API to evict admin-launched models - grants OpenAI API the same LRU eviction as Ollama. Off by default.

Other details:

• All running chat instances count (both admin UI and proxy-managed).
• Sleeping instances still count (slot claim persists across the idle pause) - but waking a sleeper for its own model is never blocked by the cap.
• Embedding models are excluded and never evicted.
• LLAMAMAN_MAX_MODELS=0 disables eviction entirely.

OpenWebUI Integration

Point OpenWebUI at the Ollama proxy (with an API key when require_auth is on):

open-webui:
  environment:
    - OLLAMA_BASE_URL=http://llamaman:42069
    - OPENAI_API_BASE_URLS=http://llamaman:42069/v1
    - OPENAI_API_KEYS=llm-your-api-key-here

How it works: OpenWebUI hits /api/tags → llamaMan lists available GGUFs; selecting a model → /api/chat arrives → llamaMan spawns a llama-server container (using saved preset or defaults), waits healthy, then proxies. When the cap is hit, the LRU Ollama-managed model is evicted (see Model Eviction).

Supported: Ollama /api/tags, /api/chat, /api/generate, /api/show, /api/version, /api/ps; OpenAI /v1/models, /v1/chat/completions.

Models are listed by GGUF filename stem by default. Set a per-model Display Name to have OpenWebUI show/accept a friendly name. In a cluster, live shared-queue group aliases are also advertised as selectable models.

Download Settings

Under Settings → Download Settings:

• Auto-retry failed downloads (off by default) + Retry count per failed download (default 3)
• Check models for updates in the background (off by default) - opt-in worker that asks each source repo whether a file has been republished, plus computes a checksum for any model that doesn't have one yet. Update check interval (hours) default 24. Hashes at most one model per pass; never runs while a download is in progress.

Docker Image Management

Settings → Docker Images: pull any llama.cpp image by name, delete old local images (disabled for the active LLAMA_IMAGE, and returns an error if Docker refuses because a container is using it), and optionally auto-update the active image on a schedule.

Model Backup and Restore

Settings → App Settings:

• Download Stored Models JSON - exports all scanned models + preset configs to a timestamped JSON
• Restore from JSON - per model: already-present → preset merged in (existing values preserved); missing but with a HuggingFace source → queued to download with preset pre-populated; missing and no known source → reported as unrestorable

Cleanup Settings

Under Settings → Cleanup Settings (all runs periodically in the background; only removes/updates records - never deletes model files):

• Auto-clean completed/failed downloads (default 24h) - active downloads never touched
• Auto-clean stopped instances (default 24h) - running instances never removed
• Auto-remove stale instance records (default 5 min interval) - re-checks starting/healthy/sleeping records against Docker; ones whose container is gone are marked stopped

Storage Backends

JSON (default)

Zero-config. Files under DATA_DIR (/data): state.json, presets.json, users.json, settings.json, api_keys.json, request_log/. Instance and download logs go to LOGS_DIR (/tmp/llama-logs).

MariaDB / MySQL

CREATE DATABASE llamaman CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'llamaman'@'%' IDENTIFIED BY 'yourpassword';
GRANT ALL PRIVILEGES ON llamaman.* TO 'llamaman'@'%';

environment:
  - DATABASE_URL=mysql+pymysql://llamaman:yourpassword@host:3306/llamaman

Tables auto-create on first connect. Request logs go to the request_log table instead of the JSON request_log/ directory (RECORDINGS_DIR has no effect with MariaDB).

Per-node model metadata (source repo + content hash) lives in a node-scoped model_files table keyed by (node_id, model_path). On upgrade, each node copies its own entries out of the legacy shared settings blob (only paths that exist on its own disk) and keeps the blob as a read fallback and rollback path. Migrations are versioned per node.

InnoDB note: (node_id, model_path) is (64 + 700) × 4 = 3056 bytes under utf8mb4, just under InnoDB's 3072-byte index limit, so the table is created with ROW_FORMAT=DYNAMIC.

Surviving a database outage (local mirror)

Optional, off by default.

With DATABASE_URL set, the database is on the critical path of every request (auth checks read settings and verify API keys on each call). If it becomes unreachable the node stops serving, and a container restart mid-outage can't boot at all.

Turn on Settings → App settings → "Keep a local mirror of the database" (or set LLAMAMAN_DB_MIRROR=1) and the node keeps a write-through copy in DATA_DIR/db_mirror/.

Offline changes go into an append-only journal and are replayed on reconnect (10s probe). Replay records the edit, not the resulting row, so a change another node made during the outage survives (a different preset field, a different settings key, a different HF token are all left intact).

Requirements and caveats:

• DATA_DIR must be a persistent volume (already is in the sample compose file).
• Each node needs its own DATA_DIR. The mirror is stamped with the owning node id; two nodes sharing one switches mirroring off rather than corrupting either view.
• Secrets land on local disk. Settings are copied verbatim; Hugging Face tokens are mirrored in plaintext. Password + API-key hashes are already hashed. If you chose MariaDB partly to keep secrets off individual hosts, this trade-off is why the feature is opt-in.
• Cross-node balancing stops during an outage. Peer liveness lives in the database; a degraded node sees only itself.
• API key changes are local until reconnect. A key created offline works only on this node; a key revoked offline stays valid on other nodes until reconnect. Plan revocations accordingly.
• Request logs are dropped, not buffered, while offline (the records carry full bodies; spooling them through a long outage could fill the volume).
• Turning the mirror off mid-outage is deferred until reconnect.
• Schema migrations never run while degraded; they run against the real database on reconnect, before any journal replay.

Clustering

Optional, off by default - single-node installs are completely unaffected.

Several llamaMan deployments act as one logical cluster: a single dashboard aggregating every node's GPUs, instances, and downloads, with cross-node launches/pulls/downloads and multi-node shared-queue load balancing. Nodes discover each other through the shared storage backend - no pairwise key exchange.

Requirements:

• Shared storage backend - every node points at the same DATABASE_URL (MariaDB/MySQL). JSON is per-host and can't be shared.
• Unique LLAMAMAN_NODE_NAME per node - each node's cluster identity (and required for every install anyway).
• The same CLUSTER_SECRET on every node - the bearer token (sent as X-Cluster-Secret) for all node-to-node HTTP.
• CLUSTER_ADVERTISE_URL per node for cross-node actions - a hostname/IP routable from the other hosts (not localhost), e.g. http://srv1:5000. A node without one appears in the dashboard but is view-only.

Set on each node (only LLAMAMAN_NODE_NAME and CLUSTER_ADVERTISE_URL differ):

environment:
  - LLAMAMAN_NODE_NAME=srv1                 # unique per node
  - DATABASE_URL=mysql+pymysql://llamaman:pass@db-host:3306/llamaman   # identical
  - CLUSTER_ENABLED=true
  - CLUSTER_SECRET=a-long-shared-random-secret   # identical
  - CLUSTER_ADVERTISE_URL=http://srv1:5000  # this node's address, routable from peers

Each node heartbeats every ~5s; a node silent past CLUSTER_NODE_ONLINE_WINDOW_S (default 45s) shows offline. Inspect and manage under Settings → Cluster.

A few settings are scoped per node because they're host-specific: tracked Docker images and the model-cap eviction toggles. Everything else is shared cluster-wide. Live shared-queue group aliases are advertised as selectable models in /api/tags and /v1/models (deduped cluster-wide), so a client can send the alias and have it routed to the least-loaded node serving it.

Group context length is cluster-wide. /api/show and /v1/models report a queue group's context as the min across all live members, whichever node the client asks. A group with members at different ctx values advertises the smaller value so a prompt sized to it is safe for any dispatch target — the extra headroom on the bigger member simply goes unused. A node with no local member of the group still answers (context_length filled from the peer's runtime ctx in shared instance state, or from the preset row in the shared DB); a group with no live members anywhere still returns 404 / no entry. The instance card on the operator UI adds ctx <min>, capped by <node/model> when this member's ctx exceeds the advertised min, so you can see which peer is dragging the group down.

Security: the cluster secret lets any peer drive actions on this node. Run node-to-node traffic over a trusted network or behind TLS.

Environment Variables

Core

Docker / GPU

Clustering

Optional. LLAMAMAN_NODE_NAME (under Core) doubles as the cluster identity and is required for all installs.

REST API

All endpoints return / accept JSON. Management endpoints need a session cookie (browser login) or Authorization: Bearer <key>. Model-serving endpoints also require a bearer when require_auth is on (default).

Auth & keys

Instances

Launch body (POST /api/instances): the launch form's fields, serialized as JSON. See the Launch settings reference above for every field and its default; enum-valued fields (flash_attn, reasoning_format, split_mode, cache_type_k/v, spec_type) accept only the values listed there and default to the reference's default. Numeric fields left as null or omitted use the reference default; unknown fields are ignored.

Boundary quirks worth pinning:

• n_gpu_layers: -1 = auto (llama.cpp estimates VRAM), -2 = all (force every layer), 0 = CPU only (no GPU attached to the container), positive N = exactly N layers.
• n_cpu_moe_layers: 0 (default) omits both MoE flags. -1 emits --cpu-moe (all routed experts on CPU). Positive N emits --n-cpu-moe N (experts of the first N layers on CPU). Inert on dense models.
• threads: when set, applies --threads N to llama-server and sets the container CPU quota to N cores.
• threads_batch: null / omitted → the flag is not emitted and llama-server falls back to --threads.
• flash_attn: legacy true / false values from pre-tri-state configs are folded on read (true → "on", false → "off"), so no storage migration is needed.
• cache_type_v quantized values require flash_attn: "on" - "auto" is not a guarantee; llama-server refuses to start otherwise.
• spec_draft_model is required when spec_enabled: true and spec_type is anything other than "draft-mtp" - a request without it is rejected 400.
• spec_draft_n_min / _p_split / _p_min: null / omitted omits the flag. 0 is a real value distinct from null and is passed through.

Downloads, Models, Presets

Download body: {"repo_id": "...", "filename": "...", "hf_token": "...", "speed_limit_mbps": 0}. Blank filename pulls the full repo.

Settings, System, Request log

Ollama & OpenAI compat

Ollama: /api/tags, /api/version, /api/show, /api/ps, /api/chat, /api/generate (chat/generate auto-start). OpenAI: /v1/models, /v1/chat/completions (chat auto-starts).

Troubleshooting

Credits

This work would not be possible without ggml-org/llama.cpp.

License

llamaMan is licensed under the Elastic License 2.0. You may use, copy, distribute, and modify the software, subject to:

• No providing the software to third parties as a hosted / managed service where the service gives users access to a substantial set of its features or functionality.
• No removing or obscuring licensing, copyright, or other notices of the licensor.

Third-party licenses

• Font Awesome Free 7.1.0 by Fonticons, Inc. - icons (CC BY 4.0), fonts (SIL OFL 1.1), and code (MIT). Full license text ships in static/fontawesome-free-7.1.0-web/LICENSE.txt.