fastcrw-aio

fastcrw-aio

Docker app from dub19's Repository

Overview

fastCRW is a Firecrawl-compatible web scraping, crawling, and search API for AI agents on Unraid. Search, scrape, crawl, map, and extract in one container. Bundled JS renderer, native /mcp, zero credits, AGPL-3.0.

The REST API listens on port 3000 (default) with the Firecrawl-compatible /v1 surface: scrape, crawl, map, extract, search, and batch scrape. Point MCP-compatible agents (Claude Code, Cursor, and more) at http://[IP]:3000/mcp for 9 native tools (crw_search, crw_scrape, and friends). Full API docs are served at http://[IP]:3000/openapi.json.

Teach your agent when and why to use CRW with the official fastCRW agent skill (https://fastcrw.com/skills): a ~100-token SKILL.md that auto-triggers on web-data tasks, installed with one command (npx crw-mcp@latest install, supports Claude Code, Cursor, Gemini CLI, Codex, OpenCode, and Windsurf). Point the skill at this container by setting CRW_API_URL to http://[IP]:3000, then MCP handles execution.

First run:

  • Scrape, Crawl, Map, and Extract work out of the box. Full JavaScript rendering is built in via the bundled LightPanda renderer.
  • Set the SearXNG URL (CRW_SEARCH__SEARXNG_URL) to an external SearXNG instance with its JSON API enabled to activate /v1/search and research queries, e.g. http://192.168.1.100:8080 (your Unraid IP and SearXNG port).
  • To use the heavy Chrome renderer tier, set Renderer Mode to chrome and supply an external CDP endpoint; Chrome is not bundled.
  • Persistent configuration lives under /config (runtime crw config), and application data under /data.

Advanced settings (hidden by default): optional OpenAlex and Semantic Scholar API keys, API key gating for the REST API, and renderer memory limits.

Readme

View on GitHub

fastcrw-aio

fastcrw

fastCRW as one Unraid container. fastCRW is a Rust-native, Firecrawl-compatible search, scrape, crawl, map, and extract backend.

fastcrw-aio runs one s6-supervised image:

  • crw-server on port 3000
  • Bundled LightPanda on internal port 9222
  • External SearXNG for Search and Research
  • No bundled LLM. Hermes is the summary and answer layer.

Point Hermes or any Firecrawl SDK at http://<lan>:3000.

What it provides

  • Scrape: /v1/scrape and /v2/scrape
  • Crawl: /v1/crawl and /v2/crawl
  • Map: /v1/map and /v2/map
  • Extract: /v1/extract and /v2/extract
  • Batch scrape: /v1/batch/scrape and /v2/batch/scrape
  • Parse: /v2/parse
  • Search: /v1/search and /v2/search
  • Health: /health
  • MCP: /mcp on the API port
  • OpenAPI docs: /openapi.json

There is no web UI. The published port serves the REST API only.

Search and research

fastCRW does not bundle SearXNG.

Set CRW_SEARCH__SEARXNG_URL to an external SearXNG instance with its JSON API enabled. Search and research need this setting. Scrape, crawl, map, extract, and parse work without it.

Recommended SearXNG engines:

  • Web search engines
  • arxiv
  • crossref
  • google scholar
  • semantic scholar
  • github

Optional advanced fields support OpenAlex, Semantic Scholar, research engines, GitHub engines, and an API-key gate.

First boot

  1. Install the fastcrw-aio Unraid template.
  2. Set CRW_SEARCH__SEARXNG_URL for Search and Research.
  3. Leave the default port and memory values unless you need custom limits.
  4. Apply the template.
  5. Wait for /health.
  6. Point Hermes at http://<lan>:3000.

The container writes runtime configuration under /config and runtime data under /data.

Image

Configuration

Core fields:

  • CRW_SEARCH__SEARXNG_URL, external SearXNG URL (e.g. http://192.168.1.100:8080)
  • CRW_RENDERER__MODE, default auto

The internal API port is pinned to 3000 in the image (CRW_SERVER__PORT is set in the Dockerfile). The published host port is the only port knob; change it in the Unraid template's API Port field. CRW_MEM_LIMIT (default 2g) is an advanced, hidden field.

Advanced fields are optional and hidden in the Unraid template.

LightPanda is the default JavaScript renderer. Chrome is not bundled. An external Chrome CDP endpoint remains optional.

Existing Compose stack

This image does not replace /docker-compose/fastcrw automatically.

Keep the existing Compose stack until a separate migration decision is approved.

Upgrades

Upgrades are review-gated and automated, not silent:

  1. The aio-fleet control plane watches us/crw for new stable tags.
  2. On a new tag it opens a PR that updates CRW_VERSION and the CRW_TARBALL_SHA256 pin together. The archive sha256 is recomputed at PR time and verified by the PR's own build (sha256sum -c fails closed on mismatch). The PR body inlines the upstream release notes.
  3. You review and merge that one PR. Merging publishes the new image and syncs the catalog. No manual Dockerfile edits.

The sha pin verifies integrity in transit, not upstream authenticity: the hash is computed from the same archive the build consumes. Authenticity comes from reviewing the upstream release notes in the PR before merging.

License

The fastcrw-aio wrapper is MIT.

The upstream fastCRW engine and MCP server are AGPL-3.0. The repository keeps the upstream source reference and pinned version.

See LICENSE and the upstream us/crw license for the applicable terms.

Install fastcrw-aio on Unraid in a few clicks.

Find fastcrw-aio in Community Apps on your Unraid server, review the template, and click Install. Unraid handles the Docker app or plugin setup from the published template.

Open the Apps tab on your Unraid server Search Community Apps for fastcrw-aio Review the template variables and paths Click Install
Explore Unraid OS

Requirements

An external SearXNG instance with the JSON API enabled is required for the search and research endpoints; scrape, crawl, map, extract, and parse work without it.

Categories

Tools > Utilities

Download Statistics

154
Total Downloads

Related apps

Explore more like this

Explore all
Utilities apps

Links

Projectgithub.comSupportgithub.comDocker Hubhub.docker.comTemplateraw.githubusercontent.com

Details

Repository
dub19/fastcrw-aio:latest
Registry
https://hub.docker.com/r/dub19/fastcrw-aio
Last Updated2026-08-24
First Seen2026-08-24

Runtime arguments

Web UI
http://[IP]:[PORT:3000]
Network
bridge
Shell
sh
Privileged
false

Template configuration

API PortPorttcp

Host (LAN) port that exposes the fastCRW REST API, /mcp, and /openapi.json. The API always serves on container port 3000 (pinned in the image); change only this host-side port.

Target
3000
Default
3000
Value
3000
SearXNG URLVariable

Required for Search and Research; Scrape, Crawl, Map, and Extract work without it. Point this at an external SearXNG instance with its JSON API enabled, e.g. http://192.168.1.100:8080 (your Unraid IP and SearXNG port).

Target
CRW_SEARCH__SEARXNG_URL
Renderer ModeVariable

Renderer to use: auto, lightpanda (bundled in-container), playwright, chrome (requires an external CDP endpoint; Chrome is not bundled), camoufox, or none.

Target
CRW_RENDERER__MODE
Default
auto
Value
auto
Memory LimitVariable

Cap on crw-server renderer and document-sandbox memory (e.g. 2g). Raise it when running the heavy Chrome tier or larger parse pools.

Target
CRW_MEM_LIMIT
Default
2g
Value
2g
OpenAlex API KeyVariable

Optional OpenAlex API key to lift rate limits on the research search engine. Unauthenticated access works without it.

Target
CRW_SEARCH__OPENALEX_API_KEY
OpenAlex MailtoVariable

Optional contact email (mailto) sent to the OpenAlex polite pool. Helps reliability; unauthenticated access works without it.

Target
CRW_SEARCH__OPENALEX_MAILTO
Semantic Scholar API KeyVariable

Optional Semantic Scholar API key that boosts research result quality. Works without it.

Target
CRW_SEARCH__S2_API_KEY
API KeysVariable

Optional comma-separated API keys that gate the crw-server REST API. Leave empty for open access.

Target
CRW_AUTH__API_KEYS