shopping-hub
Docker app from allornothing's Repository
Cross-marketplace product aggregation and price comparison.
Requirements
Overview
Cross-marketplace product aggregation and price comparison.
Crawls AliExpress, Alibaba, DHgate, Banggood, Made-in-China, Geekbuying, Chinavasion
and eBay, works out which listings are the same physical product, and publishes one
item page per product showing every site's price, minimum order quantity and
disclosed shipping - translated to English, with forwarding-agent links for the sites
that will not ship to the US.
Products are matched on five signals - perceptual image hash, normalised title,
brand, manufacturer part number and extracted specs - rather than titles alone, which
are close to useless on these sites.
Comparison is done on landed cost (unit price x minimum order quantity + freight +
duty), because the cheapest unit price often belongs to the most expensive order. A
US retail baseline from eBay is included, so the tool can tell you when importing is
not actually cheaper.
Also included: reverse image search, bulk/BOM costing, break-even analysis, price and
restock alerts, supplier trust scoring, category browsing and full-text search.
TWO SETTINGS THAT MATTER:
Set SOURCEHUB_ADMIN_TOKEN below. The admin pages are disabled without it, by
design - container traffic never looks like loopback to the app, so leaving it unset
would otherwise expose an unauthenticated admin panel to your whole LAN. Sign in with
the token as the password (any username). Browsing, search and crawling work either
way.Leave Extra Parameters set to --shm-size=1g. Docker's default 64MB /dev/shm makes
Chromium crash on heavy pages, and it fails silently - the browser-rendered sites just
return zero listings.
Everything lives under the single /config mount: database, images, browser profile and
the editable YAML config. Back that up and you have backed up the app.
A slim image (allornothing/shopping-hub:slim, ~0.31GB vs ~1.17GB) is available if you
don't need the six JavaScript-rendered adapters or the eBay/break-even baseline --
change the Repository field above to use it.
Readme
View on GitHubShopping Hub
Shopping Hub compares the same product across marketplaces like AliExpress, Alibaba, 1688, DHgate, LCSC, eBay and Best Buy. Each product gets one page with every site's price, minimum order quantity and shipping, translated to English, plus a US retail price so you can tell whether importing is actually cheaper.
What it does well:
- Matches products across sites, so you see one item instead of fifteen listings.
- Shows the real cost, including minimums, fees and a US baseline.
- Handles forwarding agents for sites that won't ship to the US.
Image: allornothing/shopping-hub · Source: aon082910/shopping-hub
Install
Unraid: search shopping-hub in Apps, set an admin token, and keep --shm-size=1g in Extra Parameters. The full walkthrough is in UNRAID.md.
Docker:
docker run -d --name shopping-hub -p 8000:8000 --shm-size=1g \
-v ./appdata:/config \
-e SOURCEHUB_ADMIN_TOKEN=change-me \
allornothing/shopping-hub:latest
Open http://127.0.0.1:8000. There's a docker-compose.yml in the repo too.
| Setting | Default | What it's for |
|---|---|---|
| Port | 8000 |
Web UI |
/config |
your appdata folder | Database and settings |
SOURCEHUB_ADMIN_TOKEN |
none | Admin access |
SOURCEHUB_PROXY |
none | Optional residential proxy for heavy crawling |
Use the slim tag if you don't want the bundled browser (about 0.3 GB instead of 1.2 GB). It skips eBay, Banggood, 1688, Taobao and Tmall.
Read before crawling
Taobao, Tmall and 1688 need a logged-in session, and most marketplaces rate-limit scrapers and may forbid it in their terms. The defaults are deliberately slow. Check each site's terms and crawl politely.
Help
Open an issue. MIT licensed.
Media gallery
1 / 4Categories
Download Statistics
Related apps
Explore more like this
Explore allLinks
Details
allornothing/shopping-hub:latestRuntime arguments
- Web UI
http://[IP]:[PORT:8000]/- Network
bridge- Shell
bash- Privileged
- false
- Extra Params
--shm-size=1g
Template configuration
Port for the web interface.
- Target
- 8000
- Default
- 8000
- Value
- 8000
Database, downloaded images, browser login profile and the editable YAML config all live here. Back this up.
- Target
- /config
- Default
- /mnt/user/appdata/shopping-hub
- Value
- /mnt/user/appdata/shopping-hub
REQUIRED to use the admin pages in a container. Any strong random string. Sign in with it as the password (any username). Leave blank and admin returns 403 - browsing and search still work.
- Target
- SOURCEHUB_ADMIN_TOKEN
serve = web UI plus scheduled crawling (recommended). web = UI only, no crawling. schedule = crawler only, no UI.
- Target
- SOURCEHUB_MODE
- Default
- serve
- Value
- serve
Timezone. The crawl schedule in config.yaml is interpreted in this zone.
- Default
- America/New_York
- Value
- America/New_York
claude, deepl, google_free or none. Without translation, Chinese listings cannot be title-matched and mostly stay on their own product pages.
- Target
- TRANSLATE_PROVIDER
- Default
- none
- Value
- none
Used when Translation Provider is set to claude. Translations are cached by content hash, so a repeat crawl costs nothing.
- Target
- ANTHROPIC_API_KEY
User ID. 99 (nobody) is the Unraid default and keeps appdata editable from the host.
- Default
- 99
- Value
- 99
Group ID. 100 (users) is the Unraid default.
- Default
- 100
- Value
- 100
Free from developer.ebay.com. Without it eBay falls back to browser scraping, which works but is slower.
- Target
- EBAY_CLIENT_ID
Free from developer.ebay.com.
- Target
- EBAY_CLIENT_SECRET
From an AliExpress Affiliate app at portals.aliexpress.com (join the affiliate program, then create an app under its API section - the Dropshipping API is a different product and will not work here). Without this, AliExpress falls back to scraping its public search page. With it, the app also gets full category-tree crawling instead of only keyword search.
- Target
- ALIEXPRESS_APP_KEY
From the same AliExpress Affiliate app as App Key.
- Target
- ALIEXPRESS_APP_SECRET
Also from the Affiliate portal - required by the API on every call to attribute commission. Any tracking ID you create there works.
- Target
- ALIEXPRESS_TRACKING_ID
Optional. One http://user:pass@host:port, or several comma-separated for round-robin. Strongly recommended for sustained crawling.
- Target
- SOURCEHUB_PROXY
Optional API key for Taobao/Tmall/1688 data. Without it those three need a one-time browser login, which is awkward in a container.
- Target
- CN_PROVIDER_KEY
Set true ONLY if SWAG, Nginx Proxy Manager, Traefik or similar sits in front of this container. Searching for something uncrawled starts a background crawl, and that is rate limited per caller by source address -- behind a proxy every request looks like it came from the proxy, so one visitor's limit is shared by everyone. Turning this on reads X-Forwarded-For instead. Leave it false when the port is exposed directly: the header is otherwise whatever the client typed, and trusting it lets one caller bypass the limit entirely.
- Target
- SOURCEHUB_TRUST_PROXY
- Default
- false