TRAWL

Welcome to TRAWL! 👋

Self-hosted web scraping engine with best-effort JS challenge and CAPTCHA solving.
Dedicated flows for Cloudflare, Akamai Bot Manager, and Imperva/Incapsula (best effort), plus Turnstile, reCAPTCHA, hCaptcha, and GeeTest.
Much faster and more reliable FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

Features

• 2-6x faster - compared to FlareSolverr or Byparr it returns much faster with higher success rate
• 4-tier execution - plain HTTP fetch → cached browser session → fresh challenge solve → residential proxy
• Challenge-aware HTTP/HTTPS proxy - direct forwarding for normal traffic, automatic tier escalation for detected walls, plus WebSockets, binary bodies, and Range/206 support
• Multi-WAF handling - dedicated Cloudflare, Akamai Bot Manager, and Imperva/Incapsula detection and browser flows
• Native captcha solving - CF Turnstile/Interstitial, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide
• Camoufox Firefox - fingerprint-patched at the C++/Juggler level to reduce automation signals
• Session cache - solved cookies and browser identity stored in Redis; accepted sessions can avoid a fresh solve
• FlareSolverr compatible - works with Prowlarr, Jackett, Sonarr, and the full *arr ecosystem out of the box
• No paid solver API required - reCAPTCHA audio can use Google's free STT endpoint or an optional local Whisper service

Sponsors

View/Collapse All

Bright Data - The most powerful platform for Web Unlocker, SERP API and web scraping tools. Why Bright Data? • Web Unlocker - bypass any anti-bot protection • SERP API - real-time Google, Bing & more results • Scraping Browser & dedicated scrapers • Massive residential proxy network • Built for scale and reliability Get started for free with Bright Data!

NodeMaven - The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market. Why NodeMaven? • ZIP targeting • 99.9% uptime • IP filtering: all proxies have fraud score <97% • No KYC required • Unique free tools: Proxy Bandwidth Checker, Meta Tag Checker, IP Lookup and others! Special codes for TRAWL users: • TRAWL35 - 35% off to Mobile and Residential Proxies • TRAWL40 - 40% off to ISP (Static) Proxies

Quick start

# Clone and configure
git clone https://github.com/germondai/trawl
cd trawl
cp .env.example .env

# Start scraper + Redis
docker compose up -d

# Verify
curl http://localhost:8191/health

First boot takes 15–30s while the browser pool warms up. Subsequent starts are fast.

NAS app catalogs

Prefer a one-click installation? TRAWL is available from the community app catalogs for both TrueNAS and Unraid:

• TrueNAS Community Apps — open Apps → Discover Apps and search for TRAWL.
• Unraid Community Apps — open the Apps tab and search for Trawl.

Thanks to the TrueNAS and Unraid community contributors who packaged and published these integrations.

API

FlareSolverr-compatible (/v1)

curl -X POST http://localhost:8191/v1 \
  -H 'Content-Type: application/json' \
  -d '{"cmd":"request.get","url":"https://nowsecure.nl","maxTimeout":60000}'

Native API (/scrape)

Returns richer metadata: tier, timings, sessionCached, full cookie list.

curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nowsecure.nl","maxTimeout":60000}'

MCP tools (/mcp)

Set MCP_ENABLED=true to expose TRAWL's client-independent Streamable HTTP tools for readable content, HTML, screenshots and browser diagnostics to any MCP-compatible AI application or agent. They load known public URLs; TRAWL does not provide web search or ranking. See the MCP integration guide.

Connect Prowlarr / Jackett

Set the FlareSolverr URL to:

http://localhost:8191        # running on the same host
http://trawl:8191            # running via Docker Compose on the same network

Challenge-bypassing HTTP/HTTPS proxy

Some sites bind their Cloudflare clearance to the solving browser's full connection fingerprint. The /v1 flow can't help there: Prowlarr keeps only the cookie + user-agent and re-fetches the page with its own HTTP client, which Cloudflare re-challenges — the cookie isn't portable. For those indexers, enable TRAWL's forward proxy and add it to Prowlarr as an HTTP proxy:

MITM_ENABLED=true
MITM_PORT=8192
MITM_CA_DIR=/data/proxy-ca   # persist the CA (mount a volume)
MITM_MAX_TIER=4              # cap escalation (e.g. 3 to stay off residential)
MITM_ALWAYS_SCRAPE=false     # opt in to bypass the proxy's direct Tier 0 probe

By default the listener binds 0.0.0.0 so clients on a Docker bridge network can reach it; set MITM_HOST=127.0.0.1 to restrict it to loopback on a bare-metal host.

1. Install the proxy's CA into the client's trust store so it accepts the per-host certs: curl http://<trawl-host>:8191/proxy-ca.crt → add to the Prowlarr container's CA store (e.g. a linuxserver /custom-cont-init.d script that copies it to /usr/local/share/ca-certificates/ and runs update-ca-certificates).
2. Prowlarr → Settings → Indexer Proxies → HTTP, host <trawl-host>, port 8192. Give it a tag if only selected indexers should use it.

Ordinary requests use a direct HTTP/TLS path. Small HTML, JSON, and text responses are buffered for challenge detection; detected challenges escalate through the same tier pipeline as POST /scrape. Videos and large binary responses stream directly. Range requests are forwarded end to end and can escalate when their response is a detected challenge; WebSocket upgrades use a direct relay without browser escalation.

See the complete proxy documentation for routing details, supported traffic, limitations, CA installation, and client examples.

⚠️ A MITM proxy can impersonate any host to a client that trusts its CA. Only expose it on a private interface (localhost / a private Docker network), never publicly.

Installing the proxy CA certificate

The proxy self-generates a root CA on first run. Its certificate and private key are persisted under MITM_CA_DIR (default /data/proxy-ca). Per-host certificates are minted and cached in memory while TRAWL runs; they do not need separate installation because they are signed by the persistent root. Every client that uses the proxy must trust that root. Without it, HTTPS fails with ERR_CERT_AUTHORITY_INVALID (browsers) or PKIX path building failed (Java).

Download the CA once per client:

curl http://<trawl-host>:8191/proxy-ca.crt -o trawl-ca.crt
# or in a Docker setup where the API isn't reachable from outside:
docker cp trawl:/data/proxy-ca/ca.crt ./trawl-ca.crt

macOS (system keychain — affects most apps including Safari, curl, wget)

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ./trawl-ca.crt
# Verify
security find-certificate -c "TRAWL MITM Proxy CA"
# Remove later
sudo security delete-certificate -c "TRAWL MITM Proxy CA" \
  /Library/Keychains/System.keychain

Linux (Debian/Ubuntu — system-wide for curl, wget, apt, etc.)

sudo cp trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
sudo update-ca-certificates
# Verify
awk '/BEGIN/{c++} c==2' /etc/ssl/certs/ca-certificates.crt | grep -c "TRAWL MITM"

Linux (RHEL/Fedora/Amazon)

sudo cp trawl-ca.crt /etc/pki/ca-trust/source/anchors/trawl-ca.crt
sudo update-ca-trust

Firefox and NSS trust stores

Firefox installations that do not use operating-system roots need a per-profile NSS import:

# Firefox 115+ uses a file-backed NSS DB; older versions use the legacy libnssdb format.
# The certutil command is the same either way.
certutil -A -n "TRAWL MITM" -t "CT,C,C" -i trawl-ca.crt \
  -d sql:$HOME/.mozilla/firefox/<profile-dir>
# Or via Firefox UI: Settings → Privacy & Security → Certificates → View Certificates →
# Authorities → Import… → check "Trust this CA to identify websites".
# Profile dir location: about:profiles in Firefox.

Chrome / Chromium (Linux: separate from system trust)

Chrome uses the system trust store on macOS and Windows but has its own on Linux:

# Option A: launch Chrome with --user-data-dir + NSS DB update (same as Firefox).
# Option B: use Chrome's --ignore-certificate-errors-spki-list=<hash> (per-session, less safe).
# Option C: add the cert to the system store (above) — Chrome picks it up automatically on
# most Linux distros via the nss-tool lookup.

Java (including JDownloader)

# Find the JRE cacerts file for your client.
#   JDownloader:    <install>/jre/lib/security/cacerts
keytool -importcert -alias trawl -file trawl-ca.crt \
  -keystore "<path-to-cacerts>" -storepass changeit
# If `keytool` reports "Certificate already exists in keystore", use -delete first:
#   keytool -delete -alias trawl -keystore "<path-to-cacerts>" -storepass changeit

Prowlarr, Sonarr, and Radarr are .NET applications, not Java applications. For their Docker-based installations, add the CA to the container's Linux system trust store. A common LinuxServer pattern is a /custom-cont-init.d script:

# In the client's Compose service:
volumes:
  - ./trawl-ca.crt:/config/trawl-ca.crt:ro
  - ./install-trawl-ca.sh:/custom-cont-init.d/50-install-trawl-ca:ro

#!/usr/bin/with-contenv bash
cp /config/trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
update-ca-certificates

LinuxServer runs scripts in /custom-cont-init.d/ when the container starts. Java clients such as JDownloader require the separate keytool import described above.

JDownloader 2 (Windows / macOS / Linux — manual install)

JDownloader bundles its own JRE; the CA must be imported into it.

1. Find the JRE: Settings → Advanced → Java Path (in JDownloader) or look in the install dir:
   • Windows: C:\Program Files\JDownloader 2\jre\lib\security\cacerts
   • macOS: /Applications/JDownloader 2.app/Contents/app/jre/lib/security/cacerts
   • Linux: <install>/jre/lib/security/cacerts
2. Run the keytool -importcert command above against that file.
3. Restart JDownloader.

Windows (system trust store)

# Run PowerShell as Administrator.
Import-Certificate -FilePath .\trawl-ca.crt `
  -CertStoreLocation Cert:\LocalMachine\Root
# Remove later
Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like "*TRAWL MITM*" } | Remove-Item

Removing the CA (cleanup)

Every installation method has a symmetric removal path. Search your trust store for TRAWL MITM Proxy CA (the CA's CN) and delete that entry. The CA certificate and key also live at <MITM_CA_DIR>/ca.crt and ca.key on the TRAWL host. Deleting either causes TRAWL to generate a new root on its next start, so existing clients must install the new certificate.

Tiers

Request
  │
  ▼
Tier 1: Plain HTTP fetch ────── success ──→ return (< 100ms)
  │ blocked
  ▼
Tier 2: Cached session ─────── success ──→ return (~500ms)
  │ cache miss / expired
  ▼
Tier 3: Fresh challenge solve ─ success ──→ cache + return
  │ IP flagged
  ▼
Tier 4: Residential proxy ──── success ──→ cache + return (15–45s)
  │ failed
  ▼
  error

Docker Compose files

Docker images (one GHCR package, two tags)

Both tags live on the same ghcr.io/germondai/trawl package — they share the registry but use different Dockerfile sources. Pick whichever tag fits your hardware:

# Modern hardware (most users)
image: ghcr.io/germondai/trawl:latest

# Older CPUs without AVX2 / Synology / older kernels
image: ghcr.io/germondai/trawl:baseline

Synology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship kernel 4.4.x, which Bun's modern runtime can't fully handle. Standard Bun requires kernel 5.1+ (5.6+ recommended); the baseline build degrades gracefully down to kernel 3.10. The :baseline tag is published for that case — confirmed working on a Synology DS920+ (Celeron J4125, DSM 7.3.2, kernel 4.4.302): the container starts cleanly, /health reports healthy, and it solves live Cloudflare challenges via /v1 (see #1). Published by independent GitHub Actions workflows: pushing v1.5.0 creates :1.5.0, :latest, :1.5.0-baseline, and :baseline; pushing main creates :nightly and :nightly-<sha>.

Releases & versioning

TRAWL follows Semantic Versioning. Pushing a v-prefixed git tag (e.g. v1.0.0) triggers publish.yml, which builds and pushes the matching un-prefixed Docker tag (ghcr.io/germondai/trawl:1.0.0) alongside a major-only tag (:1). :latest always tracks the tip of main; :sha-<shortsha> images are pushed on every main commit regardless of tags. See the Releases page for the full version history and CHANGELOG.md for what changed in each one.

To publish a specific past commit that had a major fix without waiting for the next tip-of-main release, tag that exact SHA and push it — publish.yml builds off the tag ref, not off main's current tip:

git tag -a v1.0.1 <sha> -m "..."
git push origin v1.0.1

Configuration

TRAWL supports HTTP proxies, authenticated HTTP proxies, and SOCKS5 proxies. The standard Compose files read proxy settings from the local .env file:

# Optional Tier 3 datacenter proxy
PROXY_URL=http://user:pass@datacenter.example.com:8080

# Optional Tier 4 residential proxy
RESIDENTIAL_PROXY_URL=socks5://user:pass@residential.example.com:1080

docker compose up -d

Leave either value empty to disable that proxy tier. Multiple endpoints can be separated with commas; larger pools can use the corresponding *_LIST_FILE variable. See Configuration → Proxies for pool and mounted-file examples.

Upgrading from an earlier release requires renaming several environment variables. See the configuration migration guide for the complete old-to-new mapping and Redis opt-in behavior.

Stack

Built on a modern, fast-by-default stack: Bun + Elysia for the API, Redis for caching, Camoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy Node/Express baggage.

License

AGPL-3.0

Made with ❤️ by @germondai