TRAWL

Welcome to TRAWL! 👋

Self-hosted web scraping engine that bypasses any JS challenge & captcha.
Support for: Cloudflare, Turnstile, Interstitial, reCAPTCHA, hCaptcha, GeeTest, Imperva (experimental).
Much faster and more reliable FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

Features

• 2-6x faster - compared to FlareSolverr or Byparr it returns much faster with higher success rate
• 4-tier execution - plain HTTP fetch → cached browser session → fresh CF solve → residential proxy
• Native captcha solving - CF Turnstile/Interstitial, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide
• Camoufox Firefox - fingerprint-patched at the C++/Juggler level; indistinguishable from a real browser
• Session cache - bypass cookies stored in Redis; repeat requests to the same domain return in ~500ms
• FlareSolverr compatible - works with Prowlarr, Jackett, Sonarr, and the full *arr ecosystem out of the box
• No external APIs required - reCAPTCHA audio transcription uses Google's free STT endpoint by default

Sponsors

View/Collapse All

NodeMaven - The most reliable proxy provider with the Highest Quality IP on the market. Best solution for automation, web scraping, SEO research, and social media management. Why NodeMaven? • 99.9% uptime • Sticky sessions up to 7 days • IP filtering: all proxies have fraud score <97% • No KYC required • Cashback on traffic - burn GB and earn up to 10% back Special codes for TRAWL users: • TRAWL35 - 35% off to Mobile and Residential Proxies • TRAWL40 - 40% off to ISP (Static) Proxies

Swiftproxy - A global residential proxy provider offering 80M+ real residential IPs across 190+ countries and regions. Built for web scraping, browser automation, AI workflows, SEO monitoring, and multi-account management. Why Swiftproxy? • Non-expiring residential traffic • Rotating & sticky sessions • HTTP, HTTPS & SOCKS5 support • Country, state & city-level targeting • 24/7 technical support Exclusive offer for TRAWL users: • PROXY90 - Get 10% OFF all proxy plans • Free Trial Available

Quick start

# Clone and configure
git clone https://github.com/germondai/trawl
cd trawl
cp .env.example .env

# Start scraper + Redis
docker compose up -d

# Verify
curl http://localhost:8191/health

First boot takes 15–30s while the browser pool warms up. Subsequent starts are fast.

API

FlareSolverr-compatible (/v1)

curl -X POST http://localhost:8191/v1 \
  -H 'Content-Type: application/json' \
  -d '{"cmd":"request.get","url":"https://nowsecure.nl","maxTimeout":60000}'

Native API (/scrape)

Returns richer metadata: tier, timings, sessionCached, full cookie list.

curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nowsecure.nl","maxTimeout":60000}'

Connect Prowlarr / Jackett

Set the FlareSolverr URL to:

http://localhost:8191        # running on the same host
http://trawl:8191            # running via Docker Compose on the same network

MITM forward-proxy mode (fingerprint-bound Cloudflare, e.g. 1337x)

Some sites bind their Cloudflare clearance to the solving browser's full connection fingerprint. The /v1 flow can't help there: Prowlarr keeps only the cookie + user-agent and re-fetches the page with its own HTTP client, which Cloudflare re-challenges — the cookie isn't portable. For those indexers, enable the browser-backed forward proxy and add it to Prowlarr as an HTTP proxy (tag it onto just the affected indexers):

MITM_PROXY_ENABLED=true
MITM_PROXY_PORT=8192
MITM_PROXY_CA_DIR=/data/proxy-ca   # persist the CA (mount a volume)
MITM_PROXY_MAX_TIER=4              # cap escalation (e.g. 3 to stay off residential)

By default the listener binds 0.0.0.0 so clients on a Docker bridge network can reach it; set MITM_PROXY_HOST=127.0.0.1 to restrict it to loopback on a bare-metal host.

1. Install the proxy's CA into the client's trust store so it accepts the per-host certs: curl http://<trawl-host>:8191/proxy-ca.crt → add to the Prowlarr container's CA store (e.g. a linuxserver /custom-cont-init.d script that copies it to /usr/local/share/ca-certificates/ and runs update-ca-certificates).
2. Prowlarr → Settings → Indexer Proxies → HTTP, host <trawl-host>, port 8192, give it a tag, and add that tag to the CF-fingerprint-bound indexer.

Every request that indexer makes (search and the .torrent/magnet grab) is then re-issued through the browser pool and returns the exact bytes Cloudflare served the browser.

⚠️ A MITM proxy can impersonate any host to a client that trusts its CA. Only expose it on a private interface (localhost / a private Docker network), never publicly.

Tiers

Request
  │
  ▼
Tier 1: Plain HTTP fetch ────── success ──→ return (< 100ms)
  │ blocked
  ▼
Tier 2: Cached session ─────── success ──→ return (~500ms)
  │ cache miss / expired
  ▼
Tier 3: Fresh CF solve ─────── success ──→ cache + return (4–15s)
  │ IP flagged
  ▼
Tier 4: Residential proxy ──── success ──→ cache + return (15–45s)
  │ failed
  ▼
  error

Docker Compose files

Docker images (one GHCR package, two tags)

Both tags live on the same ghcr.io/germondai/trawl package — they share the registry but use different Dockerfile sources. Pick whichever tag fits your hardware:

# Modern hardware (most users)
image: ghcr.io/germondai/trawl:latest

# Older CPUs without AVX2 / Synology / older kernels
image: ghcr.io/germondai/trawl:baseline

Synology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship kernel 4.4.x, which Bun's modern runtime can't fully handle. Standard Bun requires kernel 5.1+ (5.6+ recommended); the baseline build degrades gracefully down to kernel 3.10. The :baseline tag is published for that case — confirmed working on a Synology DS920+ (Celeron J4125, DSM 7.3.2, kernel 4.4.302): the container starts cleanly, /health reports healthy, and it solves live Cloudflare challenges via /v1 (see #1). Published by independent GitHub Actions workflows (.github/workflows/publish.yml, publish-baseline.yml); tag-triggered releases push matching git tags (e.g. v1.0.0 → 1.0.0, 1.0.0-baseline → 1.0.0-baseline) and manual workflow_dispatch from main updates the rolling tag (latest and baseline respectively).

Releases & versioning

TRAWL follows Semantic Versioning. Pushing a v-prefixed git tag (e.g. v1.0.0) triggers publish.yml, which builds and pushes the matching un-prefixed Docker tag (ghcr.io/germondai/trawl:1.0.0) alongside a major-only tag (:1). :latest always tracks the tip of main; :sha-<shortsha> images are pushed on every main commit regardless of tags. See the Releases page for the full version history and CHANGELOG.md for what changed in each one.

To publish a specific past commit that had a major fix without waiting for the next tip-of-main release, tag that exact SHA and push it — publish.yml builds off the tag ref, not off main's current tip:

git tag -a v1.0.1 <sha> -m "..."
git push origin v1.0.1

Configuration

Stack

Built on a modern, fast-by-default stack: Bun + Elysia for the API, Redis for caching, Camoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy Node/Express baggage.

License

AGPL-3.0

Made with ❤️ by @germondai