docker-fclones-scheduler

docker-fclones-scheduler runs the fclones duplicate finder on a schedule, in a rootless container, and replaces the copies it finds with hardlinks or removes them. It has no web page and writes its results to its log.

What it does

docker-fclones-scheduler gets back the disk space that duplicate files take, with no cron job to write:

• Scans your folders on the interval you set, 3 hours by default, or when your scheduler asks.
• Replaces each extra copy with a hardlink or a copy-on-write clone, deletes it, or only reports it.
• Logs each duplicate it found, up to 500 pairs per scan, and the space it freed.
• Marks itself unhealthy when a scan fails or an action changes no duplicate, until the next good scan.

Who it is for

docker-fclones-scheduler is built for a media library or a file share on an always-on Linux machine with Docker, where the same file can land in several folders. It checks every setting before the first scan and, by default, refuses fclones options that run commands. You need a folder the container's user can write to, and hardlinks need the copies on one filesystem.

Two other projects suit a different setup:

• Consider fclones-gui if you want to pick by hand, in a desktop window, which copies to remove. It is the fclones author's interactive frontend.
• Consider Krokiet if you also want to find similar images, similar videos or music duplicates, in a desktop app.

docker-fclones-scheduler is free software under the Apache-2.0 license.

Quick start

The image is on GitHub Container Registry and Docker Hub, for amd64 and arm64. This is the compose.yaml in this repository.

services:
  fclones:
    image: ghcr.io/cplieger/docker-fclones-scheduler:latest
    container_name: fclones
    restart: unless-stopped
    # Run "sudo mkdir -p /opt/appdata/fclones && sudo chown 1000:1000 /opt/appdata/fclones"
    # before the first start, or the container exits. If .env sets PUID and PGID, use those numbers.
    user: "${PUID:-1000}:${PGID:-1000}"

    environment:
      SCAN_INTERVAL: "1h"  # or 30m, 12h. "off" waits for an outside trigger and "0" scans once
      FCLONES_SCAN_PATHS: "/scandir"  # must match a volume target below
      FCLONES_ARGS: "--rf-over 1"  # report files that have more than one copy
      FCLONES_ACTION: "link"  # group (report only), link (hardlink), remove (delete) or dedupe (reflink)
      FCLONES_ACTION_ARGS: "--priority bottom"  # keep the first copy fclones lists, replace the others

    volumes:
      - "/path/to/media:/scandir"  # link, remove and dedupe change files here, so this user must be able to write to it
      - "/opt/appdata/fclones:/cache"  # the container exits at start if this user cannot write here

1. Create the cache folder and give it to user 1000 with sudo mkdir -p /opt/appdata/fclones && sudo chown 1000:1000 /opt/appdata/fclones. If .env sets PUID and PGID, use those numbers.
2. Replace /path/to/media with the folder to deduplicate. For link, remove and dedupe, that same user must be able to write to it.
3. For a first run that changes nothing, set FCLONES_ACTION to "group". The log then lists the duplicates and the space you would get back.
4. Run docker compose up -d.

Run docker logs fclones. You should see a scan complete line with groups= and duplicate_files= counts. If you see cache directory verification failed, the cache folder does not belong to the container's user, so repeat step 1.

On Unraid, open the Apps tab, search for fclones-scheduler and click Install.

Configuration reference

Settings are environment variables, read once at start, so recreate the container after a change. Configuration covers the three scheduling modes, extra fclones options and a slow first scan.

Security

The image opens no ports. An outside scheduler starts a scan with docker exec fclones /app/wrapper scan, through a local socket that only the container's own user can open. The container runs as the user that user: names, on a distroless base with no shell. Without a user: line it runs as UID 65532. FCLONES_ACTION must be one of the four actions.

The fclones options that run a command or change a file in place are refused unless ALLOW_UNSAFE_ARGS is true. Leave it false unless you need one of them, such as --transform. Options reach fclones as a list, with no shell to expand them. Security has a hardened compose example and what the image contains.

Troubleshooting

The healthcheck reads a file the container updates after each scan. Unhealthy means the last scan or action failed, or its report could not be read. It also means an action changed none of the duplicates the scan found, or the check of /cache at start failed. It turns healthy again after the next good scan, with no restart. With the built-in schedule, a container that has not scanned for twice SCAN_INTERVAL plus twice SCAN_TIMEOUT is unhealthy too. With the defaults that is 30 hours. How it works has the full rules.

• The container restarts in a loop with cache directory verification failed. The cache folder does not belong to the container's user. Repeat step 1 of the quick start.
• A run fails with action reclaimed nothing. The container's user cannot write to the scanned folder, or, for dedupe, the filesystem has no reflink support.
• The container exits with positional argument not allowed. Give each pattern its own flag, as in --name '*.mp4' --name '*.mkv'.
• The container shows unhealthy during a long first scan. Raise start_period, as Configuration shows.

Monitoring

docker-fclones-scheduler writes logfmt lines with UTC times to its container log and has no metrics endpoint. Four Loki alert rules ship in alerts/logql.yaml. Monitoring and alerts lists the log lines and the rules and shows how to load them.

Documentation

• Configuration covers the scheduling modes, extra fclones options and a slow first scan.
• How it works explains the schedule, the two phases of a run and the health rules.
• Monitoring and alerts lists the log lines and the alert rules.
• Security has the hardened compose example and what the image contains.

Credits

This project packages fclones (MIT) into a container image. All credit for finding and removing duplicate files goes to the fclones maintainers.

Contributing

See CONTRIBUTING.md.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

Apache-2.0. See LICENSE. The image carries the license text of every bundled component under /usr/share/licenses/.

The image packages fclones (MIT) at the version pinned by FCLONES_VERSION in the Dockerfile: amd64 downloads and unpacks the upstream release tarball https://github.com/pkolaczk/fclones/releases/download/<tag>/fclones-<version>-linux-musl-x86_64.tar.gz, arm64 builds that same tag from source at a pinned commit. The build applies no patches, so this repository's Dockerfile plus the upstream sources it names is the complete recipe for the fclones binary the image ships. The license texts of the Rust crates compiled into the amd64 binary are kept under licenses/crates/ in this repository, regenerated by scripts/vendor-crate-licenses.sh.