fclones-scheduler
Docker app from cplieger's Repository
Runs the fclones duplicate finder on a schedule and gets back the disk space that duplicate files take.
Requirements
Hardlinks cannot span disks. For the link action, scan a single disk path such as /mnt/disk1/Media rather than /mnt/user/Media, or duplicates that live on two disks fail the run.
The container reports unhealthy until its first scan completes, and on a large share that can take hours. Later scans reuse the hash cache in /cache and are much faster.
Overview
Readme
View on GitHubdocker-fclones-scheduler
Image Size Platforms base: Distroless Mutation SBOM
docker-fclones-scheduler runs the fclones duplicate finder on a schedule, in a rootless container, and replaces the copies it finds with hardlinks or removes them. It has no web interface and writes its results to its log.
What it does
docker-fclones-scheduler gets back the disk space that duplicate files take, with no cron job to write:
- Scans your folders on the interval you set, 3 hours by default, or when your scheduler asks.
- Replaces each extra copy with a hardlink or a copy-on-write clone, deletes it, or only reports it.
- Logs each duplicate it found, up to 500 pairs per scan, and the space it freed.
- Shows in its Docker health status when a scan or its action fails, until the next good scan.
Who it is for
docker-fclones-scheduler is built for a media library or a file share on an always-on Linux machine with Docker, where the same file can land in several folders. It checks every setting before the first scan and, by default, refuses fclones options that run commands. You need a folder the container's user can write to, and hardlinks need the copies on one filesystem, in one volume.
Two other projects suit a different setup:
- Consider fclones-gui if you want to pick in a desktop window which copies to remove, with the fclones author's interactive frontend.
- Consider Krokiet if you also want to find similar images, similar videos or music duplicates, in a desktop app.
docker-fclones-scheduler is free software under the Apache-2.0 license.
Quick start
The image is on GitHub Container Registry and on Docker Hub as cplieger/docker-fclones-scheduler, for amd64 and arm64. This is the compose.yaml in this repository.
services:
fclones:
image: ghcr.io/cplieger/docker-fclones-scheduler:latest
container_name: fclones
restart: unless-stopped
# Run "sudo mkdir -p /opt/appdata/fclones && sudo chown 1000:1000 /opt/appdata/fclones"
# before the first start, or the container exits. If .env sets PUID and PGID, use those numbers.
user: "${PUID:-1000}:${PGID:-1000}"
environment:
SCAN_INTERVAL: "1h" # or 30m, 12h. "off" waits for an outside trigger and "0" scans once
FCLONES_SCAN_PATHS: "/scandir" # must match a volume target below
FCLONES_ARGS: "--rf-over 1" # report files that have more than one copy
FCLONES_ACTION: "link" # group only reports, link replaces copies with hardlinks, remove deletes them, dedupe uses reflinks
FCLONES_ACTION_ARGS: "--priority bottom" # keep the first file of each group, the one the log names keeper, and replace the others
volumes:
- "/path/to/media:/scandir" # link, remove and dedupe change files here, so this user must be able to write to it
- "/opt/appdata/fclones:/cache" # the container exits at start if this user cannot write here
- Create the cache folder and give it to user 1000 with
sudo mkdir -p /opt/appdata/fclones && sudo chown 1000:1000 /opt/appdata/fclones. If.envsetsPUIDandPGID, use those numbers. - Replace
/path/to/mediawith the folder to deduplicate. Forlink,removeanddedupe, that same user must be able to write to it. - For a first run that changes nothing, set
FCLONES_ACTIONto"group". The log then lists the duplicates and the space you would get back. - Run
docker compose up -d.
Run docker logs fclones. You should see a scan complete line with groups= and duplicate_files= counts. If you see cache directory verification failed, the cache folder does not belong to the container's user, so repeat step 1.
On Unraid, open the Apps tab, search for fclones-scheduler and click Install.
Configuration reference
Settings are environment variables, read once at start, so recreate the container after a change. Configuration covers the three scheduling modes, extra fclones options and a slow first scan.
| Variable | Description | Default |
|---|---|---|
SCAN_INTERVAL |
Time between scans, such as 30m, 1h or 12h. off waits for an outside trigger, and 0 scans once and exits |
3h |
FCLONES_SCAN_PATHS |
Folders inside the container to scan, separated by spaces. Each must be a volume or a folder inside one | /scandir |
FCLONES_ARGS |
Extra fclones options for the scan, as flags and their values. --cache, -f and --format are refused |
(unset) |
FCLONES_ACTION |
group only reports, link makes hardlinks, remove deletes, dedupe makes copy-on-write clones |
group |
FCLONES_ACTION_ARGS |
Extra fclones options for the action, as flags and their values | (unset) |
ALLOW_UNSAFE_ARGS |
true allows --transform, --in-place and --no-copy, which are refused otherwise |
false |
SCAN_TIMEOUT |
Longest time the scan and the action may each run before the run is stopped and fails. 0 means no limit |
12h |
LOG_LEVEL |
debug, info, warn or error. Any other value means info |
info |
METRICS_LISTEN_ADDR |
Address for a Prometheus metrics page and a health check, for example :9469. Empty opens no port |
(unset) |
| Mount | Description |
|---|---|
/scandir |
The folder to scan, matching FCLONES_SCAN_PATHS. link, remove and dedupe change files here, so the container's user must be able to write to it |
/cache |
The fclones hash cache and the record of the last scan. The container exits at start when its user cannot write here |
The image publishes no port. With METRICS_LISTEN_ADDR set, publish the port it names, such as 9469:9469.
Security
The image opens no port unless you set METRICS_LISTEN_ADDR, which serves only /metrics and /healthz, with no login, so keep that port on your network. Another program can start a scan with docker exec fclones /app/wrapper scan. That command talks to the app through a socket file inside the container, which only the container's own user can open. The container runs as the user that user: names, on a distroless base with no shell. Without a user: line it runs as UID 65532.
FCLONES_ACTION must be one of the four actions. The fclones options that run a command or change a file in place are refused unless ALLOW_UNSAFE_ARGS is true. Leave it false unless you need one of them, such as --transform. Options reach fclones as a list, with no shell to expand them. Security has a hardened compose example and what the image contains.
Troubleshooting
The container is unhealthy when the last scan failed, when its action failed or changed none of the duplicates the scan found, or when fclones' report was unreadable. It is also unhealthy when it could not write to /cache at start. It turns healthy again after the next good scan, with no restart.
With the built-in schedule, the container also turns unhealthy after 30 hours without a good scan, counted from its last good scan or its start, whichever is later. The 30 hours are twice SCAN_INTERVAL plus twice SCAN_TIMEOUT at the defaults, and How it works has the full rules. An unhealthy container notifies nobody. Knowing the app is alive says what does.
- The container restarts in a loop with
cache directory verification failed. The cache folder does not belong to the container's user. Repeat step 1 of the quick start. - A run fails with
action reclaimed nothing. With the shipped alert rules loaded,FclonesRunFailedfires too. The container's user cannot write to the scanned folder, or, fordedupe, the filesystem has no reflink support. - The container exits with
positional argument not allowed. Give each pattern its own flag, as in--name '*.mp4' --name '*.mkv'. - The container shows unhealthy during a long first scan. Raise
start_period, as Configuration shows.
Monitoring
docker-fclones-scheduler logs one run finished record per run and can serve a Prometheus metrics page when METRICS_LISTEN_ADDR is set. The same alerts ship for Prometheus in alerts/promql.yaml and for Loki in alerts/logql.yaml. Load one of the two files. A Grafana dashboard of each run and the space it freed ships as grafana-dashboard.json and needs Grafana 13.2 or newer. The image has a Docker healthcheck, and Knowing the app is alive says what it proves and who is told.
Documentation
- Configuration covers the scheduling modes, extra fclones options and a slow first scan.
- How it works explains the schedule, the two phases of a run and the health rules.
- Monitoring and alerts lists the metrics, the
run finishedrecord, both rule files and the dashboard, and says who tells you the schedule stopped. - Security has the hardened compose example and what the image contains.
Credits
This project packages fclones (MIT) into a container image. All credit for finding and removing duplicate files goes to the fclones maintainers.
Contributing
See CONTRIBUTING.md.
Disclaimer
This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.
This project was built with AI assistance. The human maintainer defines architecture, supervises implementation, and makes all final decisions.
License
Apache-2.0. See LICENSE. The image carries the license text of every bundled component under /usr/share/licenses/.
The image packages fclones (MIT) at the version pinned by FCLONES_REF in the Dockerfile: amd64 downloads and unpacks the upstream release tarball https://github.com/pkolaczk/fclones/releases/download/<tag>/fclones-<version>-linux-musl-x86_64.tar.gz, arm64 builds that same tag from source at a pinned commit. The build applies no patches, so this repository's Dockerfile plus the upstream sources it names is the complete recipe for the fclones binary the image ships. The license texts of the Rust crates compiled into the amd64 binary are kept under licenses/crates/ in this repository, regenerated by scripts/vendor-crate-licenses.sh.
Categories
Download Statistics
Total Downloads Over Time
Related apps
Explore more like this
Explore allLinks
Details
cplieger/docker-fclones-scheduler:latestRuntime arguments
- Network
bridge- Privileged
- false
- Extra Params
--user 99:100 --health-start-period=6h
Template configuration
The share or folder to scan for duplicates, for example /mnt/user/Media. link, remove and dedupe need it read-write
- Target
- /scandir
The fclones hash cache and the last-scan record, which keep rescans fast and the schedule stable across restarts
- Target
- /cache
- Default
- /mnt/user/appdata/fclones-scheduler
- Value
- /mnt/user/appdata/fclones-scheduler
What to do with duplicates: group reports only, link replaces them with hardlinks, remove deletes them and dedupe makes copy-on-write clones. Anything but group modifies your files
- Target
- FCLONES_ACTION
- Default
- group
- Value
- group
Time between scans as a Go duration such as 3h, 12h or 24h. off turns the schedule off so you trigger scans yourself, and 0 runs one scan and exits
- Target
- SCAN_INTERVAL
- Default
- 3h
- Value
- 3h
Container paths to scan, separated by spaces. Each needs its own path mapping. Leave at /scandir unless you add more mappings
- Target
- FCLONES_SCAN_PATHS
- Default
- /scandir
- Value
- /scandir
Extra flags for the fclones group phase, the scan, such as --name '*.mkv' --name '*.mp4' or --rf-over 1. Flags and their values only
- Target
- FCLONES_ARGS
Extra flags for the action phase, such as --priority bottom. --dry-run previews the action without changing files
- Target
- FCLONES_ACTION_ARGS
Timeout for each phase as a Go duration. Raise it when a first scan of a large share exceeds 12h. 0 turns it off
- Target
- SCAN_TIMEOUT
- Default
- 12h
- Value
- 12h
Minimum log level: debug, info, warn or error
- Target
- LOG_LEVEL
- Default
- info
- Value
- info
Leave empty to open no port. Set to :9469 to serve Prometheus metrics at /metrics and a health check at /healthz, both with no login, then add a Port mapping for 9469 so your scraper or uptime prober can reach it at the server's IP
- Target
- METRICS_LISTEN_ADDR
Set to true to allow the --transform, --in-place and --no-copy fclones flags, which are refused by default
- Target
- ALLOW_UNSAFE_ARGS
- Default
- false
- Value
- false