fastcrw-aio

fastcrw-aio

Docker app from dub19's Repository

Overview

fastcrw-aio bundles crw-server v0.31.0 (the crw web-content research and crawl engine) with the LightPanda JS renderer under s6 supervision. It exposes a REST API (default port 3000) for scraping, crawling, mapping, extracting, parsing, and search/research over URLs.

First run:

  • The API listens on port 3000 (Web UI Port on the LAN). Keep the API Port env var aligned with the host port you publish if you change it.
  • Set the SearXNG URL (CRW_SEARCH__SEARXNG_URL) to an external SearXNG instance with its JSON API enabled to activate the /v1/search and research endpoints. Scrape, Crawl, Map, Extract, and Parse work without it.
  • LightPanda (bundled, in-container) is the default renderer. To use the heavy Chrome tier, set the Renderer Mode to chrome and supply an external CDP endpoint; Chrome is not bundled.
  • Persistent configuration lives under /config (runtime crw config), and application data under /data.

Hidden advanced settings let you add optional OpenAlex and Semantic Scholar API keys, tune the search engines, gate the API with API keys, and cap memory limits.

Readme

View on GitHub

fastcrw-aio

fastcrw

fastCRW as one Unraid container. fastCRW is a Rust-native, Firecrawl-compatible search, scrape, crawl, map, and extract backend.

fastcrw-aio runs one s6-supervised image:

  • crw-server on port 3000
  • Bundled LightPanda on internal port 9222
  • External SearXNG for Search and Research
  • No bundled LLM. Hermes is the summary and answer layer.

Point Hermes or any Firecrawl SDK at http://<lan>:3000.

What it provides

  • Scrape: /v1/scrape and /v2/scrape
  • Crawl: /v1/crawl and /v2/crawl
  • Map: /v1/map and /v2/map
  • Extract: /v1/extract and /v2/extract
  • Batch scrape: /v1/batch/scrape and /v2/batch/scrape
  • Parse: /v2/parse
  • Search: /v1/search and /v2/search
  • Health: /health
  • MCP: /mcp on the same API port

Search and research

fastCRW does not bundle SearXNG.

Set CRW_SEARCH__SEARXNG_URL to an external SearXNG instance with its JSON API enabled. Search and research need this setting. Scrape, crawl, map, extract, and parse work without it.

Recommended SearXNG engines:

  • Web search engines
  • arxiv
  • crossref
  • google scholar
  • semantic scholar
  • github

Optional advanced fields support OpenAlex, Semantic Scholar, research engines, GitHub engines, and an API-key gate.

First boot

  1. Install the fastcrw-aio Unraid template.
  2. Set CRW_SEARCH__SEARXNG_URL for Search and Research.
  3. Leave the default port and memory values unless you need custom limits.
  4. Apply the template.
  5. Wait for /health.
  6. Point Hermes at http://<lan>:3000.

The container writes runtime configuration under /config and runtime data under /data.

Image

Configuration

Core fields:

  • CRW_SERVER__PORT, default 3000
  • CRW_SEARCH__SEARXNG_URL, external SearXNG URL
  • CRW_RENDERER__MODE, default auto
  • CRW_MEM_LIMIT, default 2g

Advanced fields are optional and hidden in the Unraid template.

LightPanda is the default JavaScript renderer. Chrome is not bundled. An external Chrome CDP endpoint remains optional.

Existing Compose stack

This image does not replace /docker-compose/fastcrw automatically.

Keep the existing Compose stack until a separate migration decision is approved.

License

The fastcrw-aio wrapper is MIT.

The upstream fastCRW engine and MCP server are AGPL-3.0. The repository keeps the upstream source reference and pinned version.

See LICENSE and the upstream us/crw license for the applicable terms.

Install fastcrw-aio on Unraid in a few clicks.

Find fastcrw-aio in Community Apps on your Unraid server, review the template, and click Install. Unraid handles the Docker app or plugin setup from the published template.

Open the Apps tab on your Unraid server Search Community Apps for fastcrw-aio Review the template variables and paths Click Install
Explore Unraid OS

Requirements

An external SearXNG instance with the JSON API enabled is required for the search and research endpoints; scrape, crawl, map, extract, and parse work without it.

Categories

Tools > Utilities

Download Statistics

154
Total Downloads

Related apps

Explore more like this

Explore all
Utilities apps

Links

Projectgithub.comSupportgithub.comDocker Hubhub.docker.comTemplateraw.githubusercontent.com

Details

Repository
dub19/fastcrw-aio:latest
Registry
https://hub.docker.com/r/dub19/fastcrw-aio
Last Updated2026-08-24
First Seen2026-08-24

Runtime arguments

Web UI
http://[IP]:[PORT:3000]
Network
bridge
Shell
sh
Privileged
false

Template configuration

Web UI PortPorttcp

Host (LAN) port that exposes the crw-server API (port 3000).

Target
3000
Default
3000
Value
3000
API PortVariable

Port crw-server listens on inside the container. Keep this aligned with the published Web UI Port.

Target
CRW_SERVER__PORT
Default
3000
Value
3000
SearXNG URLVariable

Required for Search and Research; Scrape, Crawl, Map, Extract, and Parse work without it. Point this at an external SearXNG instance with its JSON API enabled.

Target
CRW_SEARCH__SEARXNG_URL
Renderer ModeVariable

Renderer to use: auto, lightpanda (bundled in-container), playwright, chrome (requires an external CDP endpoint; Chrome is not bundled), camoufox, or none.

Target
CRW_RENDERER__MODE
Default
auto
Value
auto
Memory LimitVariable

Cap on crw-server renderer and document-sandbox memory (e.g. 2g). Raise it when running the heavy Chrome tier or larger parse pools.

Target
CRW_MEM_LIMIT
Default
2g
Value
2g
OpenAlex API KeyVariable

Optional OpenAlex API key to lift rate limits on the research search engine. Unauthenticated access works without it.

Target
CRW_SEARCH__OPENALEX_API_KEY
OpenAlex MailtoVariable

Optional contact email (mailto) sent to the OpenAlex polite pool. Helps reliability; unauthenticated access works without it.

Target
CRW_SEARCH__OPENALEX_MAILTO
Semantic Scholar API KeyVariable

Optional Semantic Scholar API key that boosts research result quality. Works without it.

Target
CRW_SEARCH__S2_API_KEY
Research EnginesVariable

Comma-separated SearXNG engines used for research queries.

Target
CRW_SEARCH__RESEARCH_ENGINES
Default
arxiv,crossref,google scholar,semantic scholar
Value
arxiv,crossref,google scholar,semantic scholar
GitHub EnginesVariable

Comma-separated SearXNG engines used for GitHub/code queries.

Target
CRW_SEARCH__GITHUB_ENGINES
Default
github
Value
github
API KeysVariable

Optional comma-separated API keys that gate the crw-server REST API. Leave empty for open access.

Target
CRW_AUTH__API_KEYS