docker-fclones-scheduler

Find and deduplicate files on a schedule, reclaiming wasted disk space automatically.

What it does

Wraps the fclones duplicate file finder in a Go scheduler daemon with interval-based scheduling and a CLI health probe. Supports the group, link, remove, and dedupe actions with configurable arguments. Reports scan statistics including duplicates found, space reclaimable, and files processed. All output goes to stdout/stderr for collection by log aggregators (Alloy, Promtail, etc.) and alerting via Grafana or similar.

• Mount your media directory and schedule periodic scans; fclones finds duplicates and can replace them with hardlinks or remove them entirely
• Built-in scheduler, or hand scheduling to an external scheduler (cron, Ofelia, etc.) via the scan subcommand
• Built-in Docker healthcheck with automatic recovery

Why this design

• Scheduler your way: ships with a self-contained interval scheduler, so no external cron, systemd timer, or orchestrator-level scheduling is needed. If you already run a central scheduler (Ofelia, cron), set SCAN_INTERVAL=off and trigger scans with docker exec fclones /app/wrapper scan instead
• One owner for every run: the daemon executes every scan, serialized in one queue, so every run's logs land on the container's own log stream in both scheduling modes and the same alert rules work everywhere
• Machine-readable report contract: the scan consumes fclones' JSON report with a strict decoder, so an upstream output-format change fails the run loudly instead of silently zeroing the duplicate stats your alerting reads
• Distroless and rootless: runs as nonroot (UID 65532) on gcr.io/distroless/static-debian13 with no shell or package manager
• Dangerous flags blocked by default: --transform, --in-place, and --no-copy are rejected unless you explicitly opt in with ALLOW_UNSAFE_ARGS=true, preventing command injection via environment variables
• Structured logs: logfmt with UTC timestamps, so log lines are zone-stable regardless of the container's TZ and alerting needs no custom exporter

Quick start

The image is published to both GHCR (ghcr.io/cplieger/docker-fclones-scheduler) and Docker Hub (cplieger/docker-fclones-scheduler); identical contents, use whichever you prefer.

services:
  fclones:
    image: ghcr.io/cplieger/docker-fclones-scheduler:latest
    container_name: fclones
    restart: unless-stopped
    # Override with PUID/PGID in .env; defaults to 1000:1000.
    user: "${PUID:-1000}:${PGID:-1000}"  # match your host user

    environment:
      SCAN_INTERVAL: "1h"  # Go duration (e.g. 1h, 30m, 12h)
      FCLONES_SCAN_PATHS: "/scandir"
      FCLONES_ARGS: "--rf-over 1"
      FCLONES_ACTION: "link"  # group (report), link (hardlink), remove (delete), or dedupe (reflink/copy-on-write)
      FCLONES_ACTION_ARGS: "--priority bottom"

    volumes:
      - "/path/to/media:/scandir"
      - "/opt/appdata/fclones:/cache"

Scheduling modes

The container runs in one of three modes, selected by SCAN_INTERVAL.

Built-in scheduler (default)

Set SCAN_INTERVAL to a positive Go duration (1h, 30m, 12h, …). The container runs a scan at startup and then every interval. This is the zero-dependency default; nothing else is required. An unset, unparseable, or negative value falls back to the 3h default cadence in this mode (a negative value is treated as a typo and logged as a warning).

Each scheduled scan records its completion time and outcome in a file on /cache. At startup the container reads that record and skips the startup scan when the last scan completed less than one SCAN_INTERVAL ago, so a container recreate minutes after a completed scan does not repeat hours of hashing. The record also carries the schedule's phase: the next scan lands one SCAN_INTERVAL after the previous one, not one interval after boot, so a restart neither adds a scan nor delays the cadence. A failed scan also holds its slot: a restart does not repeat it, and the next interval tick is the retry, because a failed multi-hour scan is as expensive to repeat as a successful one. Only the daemon's own scheduled scans update the record; triggered scans (wrapper scan) and run-once mode do not. Without a persistent /cache volume the record does not survive a recreate and every start runs the startup scan.

External scheduler

Set SCAN_INTERVAL=off (alias: disabled). The container stays running but idle, and you trigger each scan out-of-band by exec'ing the scan subcommand:

docker exec fclones /app/wrapper scan

The scan subcommand submits one run request to the daemon, blocks until the scan finishes, and exits non-zero on failure. The run executes inside the daemon, so its full output lands on the container's log stream; the trigger sees only lifecycle lines and the result. The daemon updates the same health marker the healthcheck reports. If you interrupt the exec (SIGINT or SIGTERM), the client exits non-zero, and the daemon completes the scan it accepted. A trigger that kills the exec on its own timeout therefore records a failed job while the scan continues. Example with Ofelia labels:

services:
  fclones:
    image: ghcr.io/cplieger/docker-fclones-scheduler:latest
    container_name: fclones
    restart: unless-stopped
    user: "${PUID:-1000}:${PGID:-1000}"
    environment:
      SCAN_INTERVAL: "off"   # disable built-in loop; Ofelia drives it
      FCLONES_SCAN_PATHS: "/scandir"
      FCLONES_ACTION: "link"
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.fclones-scan.schedule: "@every 6h"
      ofelia.job-exec.fclones-scan.command: "/app/wrapper scan"
      ofelia.job-exec.fclones-scan.no-overlap: "true"
    volumes:
      - "/path/to/media:/scandir"
      - "/opt/appdata/fclones:/cache"

Runs never overlap: the daemon executes scheduled ticks and triggered scans strictly in order from one queue, and a trigger that arrives mid-run queues behind it. An advisory file lock on /cache/.fclones.lock additionally guards against a scan from a different container or a manual docker run sharing the same /cache volume; such a run skips instead of corrupting the shared fclones cache. Ofelia's no-overlap is still recommended to avoid queuing redundant triggers. The exec must run as the container's own user (the trigger socket is owner-only); a mismatched exec user fails loudly at connect.

Run once

Set SCAN_INTERVAL=0 (or 0s). The container runs exactly one scan and dedup action, then exits, and the exit code is the job result: non-zero if the scan failed, timed out, was interrupted (SIGTERM/SIGINT) before it finished, or was skipped because another process held the /cache scan lock (logged with outcome=skipped). This suits a batch or one-shot context (a Kubernetes Job, a CI step, or a manual docker run --rm) where an external system decides when to run again: a run that never completed a scan surfaces as a failure, so the orchestrator retries rather than recording success. In the long-running modes a SIGTERM is a clean shutdown and a lock conflict a benign no-op, both exiting 0.

Configuration reference

Environment variables

Passing extra fclones arguments

FCLONES_ARGS and FCLONES_ACTION_ARGS carry flags and their values only. Scan paths belong in FCLONES_SCAN_PATHS, so the wrapper rejects a bare (non-flag) token at startup and names it in the error.

Every repeatable fclones filter (--name, --path, --exclude, --keep-name, --keep-path) takes one value per flag. fclones reads a second bare pattern as another input path, resolves it against the container working directory, and fails the scan with Can't access '/app/<pattern>':

# Works, one flag per pattern
FCLONES_ARGS: "--name '*.mp4' --name '*.mkv'"

# Works, one glob for both extensions
FCLONES_ARGS: "--name '*.{mp4,mkv}'"

# Rejected at startup, '*.mkv' is a bare token that fclones reads as a path
FCLONES_ARGS: "--name '*.mp4' '*.mkv'"

The multi-value example in the fclones README has the same defect, so a config copied from there needs the repeated flag too.

Volumes

Alerting

docker-fclones-scheduler has no metrics endpoint; its operational state is in its logs. Ship the container's logs to Loki (Grafana Alloy's Docker log discovery does this with no configuration) and evaluate these with Loki's ruler; firing alerts deliver through your Alertmanager exactly like Prometheus metric alerts.

groups:
  - name: docker-fclones-scheduler
    rules:
      # The deadman. Both rules below fire on a record the wrapper LOGGED, so a
      # wedged scheduler that logs nothing at all trips neither. This one fires
      # on silence instead.
      #
      # 28h, and the size comes from the RUNTIME budget rather than the
      # interval. SCAN_TIMEOUT defaults to 12h PER PHASE, and `scan complete` is
      # logged after the scan phase but before the action phase, so the gap
      # between two heartbeats legitimately spans up to 12h of action, then the
      # interval, then up to 12h of the next scan. The schedule keeps its phase
      # across restarts (the last-scan record on /cache carries it), so a
      # restart adds nothing to that budget. Recompute this if you change
      # SCAN_TIMEOUT or SCAN_INTERVAL: 2x the phase timeout plus the interval,
      # plus margin. With
      # SCAN_TIMEOUT=0 the phases are unbounded and no finite window is sound,
      # the same caveat the healthcheck's freshness deadline already carries.
      #
      # In external mode (SCAN_INTERVAL=off) there is no startup scan and no
      # record: size the window as 2x the phase timeout plus your external
      # scheduler's cadence. In run-once mode (SCAN_INTERVAL=0) drop this rule:
      # a container that exits on purpose is silent by design.
      - alert: FclonesScanStalled
        expr: |
          absent_over_time({container="fclones"} |= "scan complete" [28h])
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "no fclones scan-completion heartbeat in 28h"
          description: >
            The wrapper logs a `scan complete` line at the end of every scan
            phase, including one that found no duplicates, and none has arrived
            in 28h. The usual cause is a wedged daemon, a scan stuck past its
            phase timeout, or nothing triggering runs in external mode. An
            absence rule cannot tell that apart from a container that never
            started or was renamed, or a log pipeline that stopped shipping this
            stream, so rule those out first. The file-marker healthcheck arms
            its own freshness deadline in built-in mode and covers the same
            failure, but only where something acts on an unhealthy container.
      - alert: FclonesLinkEstablished
        expr: |
          sum by (files_deduped, reclaimed_human) (
            count_over_time(
              {container="fclones"} |= "action complete" | logfmt | action="link" | files_deduped > 0 [15m]
            )
          ) > 0
        for: 0m
        labels:
          severity: info
        annotations:
          summary: "fclones linked {{ $labels.files_deduped }} duplicate files (reclaimed {{ $labels.reclaimed_human }})"
          description: >
            fclones established hardlinks for {{ $labels.files_deduped }} duplicate
            files, reclaiming {{ $labels.reclaimed_human }}. The individual linked
            paths are in the same run's `duplicate file` log lines (capped at 500
            pairs / 64 KB), viewable in Loki: {container="fclones"} |= "duplicate
            file" (filter by the run's scan_id). Success notification, no action
            required.
      - alert: FclonesActionReclaimedNothing
        expr: |
          sum(count_over_time({container="fclones"} |= "action reclaimed nothing" [2h])) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "fclones deduplicated none of the duplicates it found"
          description: >
            A `link`, `remove` or `dedupe` run found duplicate files and
            processed zero of them, so no space was reclaimed. fclones opens
            each duplicate for write before acting on it and only warns when
            that fails, then exits 0. The usual causes are a scan tree that is
            not writable by the container's UID, or, for `dedupe`, a filesystem
            with no reflink support. The wrapper fails the run and the container
            turns unhealthy, recovering on the next run that reclaims
            something. A `--dry-run` in FCLONES_ACTION_ARGS does not trigger
            this rule; fclones reports "Would process" for it, which reads as
            format drift instead.
      - alert: FclonesFormatDrift
        expr: |
          sum(count_over_time({container="fclones"} |= "possible fclones format drift" [2h])) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "fclones output-format drift detected (action stats may be unreliable)"
          description: >
            The wrapper logged "possible fclones format drift": the action
            phase's summary line was not recognized, so the files_deduped and
            bytes_reclaimed stats on the "action complete" line may read zero
            even though work happened. The action itself still runs and the
            run still exits 0, so a job-failure or container-restart alert
            would not catch this. (Scan-report drift is covered separately:
            the JSON report is decoded strictly, and an unreadable report
            fails the run with outcome=decode_error.) Check the fclones
            version against the wrapper's summary parser.

Thresholds and the severity labels are starting points. Adjust the container selector (or job / service, depending on your log collector) to your deployment; if you run remove or dedupe instead of link, change action="link" in FclonesLinkEstablished to match your FCLONES_ACTION. Route by whatever labels your Alertmanager uses.

These rules work in both scheduling modes: every run executes in the daemon, so its logs always land under the app's own container name. Under SCAN_INTERVAL=off the exec'd scan client emits only its own lifecycle lines (queued / started / result) to the trigger's log (an Ofelia job log, for example) and exits with the run's result, so you can additionally alert on your scheduler's own job outcome for extra failure coverage.

Healthcheck

The built-in healthcheck (/app/wrapper health) checks a marker file the daemon maintains after each run. The container becomes unhealthy when fclones exits non-zero (e.g. scan path missing, permission denied, corrupted cache), the action phase fails (e.g. hardlink across filesystems), a link/remove/dedupe action finds duplicates and deduplicates none of them, the report cannot be decoded, or startup verification fails (e.g. /cache is full or read-only). It recovers automatically on the next successful scan; no restart is required.

In built-in mode the boot state follows the last-scan record on /cache: with a successful scan younger than SCAN_INTERVAL the container starts healthy and skips the startup scan; with no record or a stale one it starts unhealthy, runs the startup scan, and turns healthy when that scan succeeds. After a failed scan and a restart the container also skips the startup scan but starts unhealthy, and it stays unhealthy until the next scheduled scan succeeds (the first tick is at most one SCAN_INTERVAL after boot); repeating an expensive failed scan at every restart would redo the same hours of work, so the ticker owns the retry. Built-in mode also arms a freshness deadline of 2 x SCAN_INTERVAL + 2 x SCAN_TIMEOUT: a marker that old means the interval loop is wedged, so the probe reports unhealthy and Docker restarts the container. The deadline is disabled automatically when SCAN_TIMEOUT=0, since the worst-case run duration is then unbounded. In external mode the container starts healthy (idle, nothing has failed), each triggered run updates the marker, and no deadline applies.

The image bakes a 15s start_period, which suits small libraries and external mode. If your first built-in scan takes minutes, raise start_period in your own compose file so the container isn't reported unhealthy, and doesn't fire spurious alerts, during that initial scan:

services:
  fclones:
    healthcheck:
      start_period: 10m # size to your library's first-scan duration

Security

No network listener, no HTTP server, no exposed ports: the trigger socket is a local unix socket (/tmp/fclones-wrapper.sock), owner-only and reachable only from inside the container. The container runs as nonroot on a distroless base image with no shell.

The FCLONES_ACTION env var is validated against an allowlist, and the dangerous flags (--transform, --in-place, --no-copy) are blocked by default to prevent command injection via env vars; set ALLOW_UNSAFE_ARGS=true only if you need --transform for content-aware deduplication. Wrapper-owned fclones flags (--cache, -f/--format) are rejected in FCLONES_ARGS at startup so user args cannot break the report contract.

Arguments reach fclones as explicit argument lists with no shell expansion. Captured subprocess output is size-capped, and the scan report is decoded with a strict streaming JSON decoder: memory stays bounded regardless of report size, and a malformed report fails the run. Runs are serialized by the daemon's single queue; an advisory file lock on /cache additionally guards the shared cache against scans from other containers or manual docker run invocations.

Accepted scanner findings, all false positives: semgrep flags the missing USER directive (the distroless base bakes in UID 65532) and the /tmp health-marker path (a fixed marker file, not sensitive data), and hadolint DL3008 fires in the Rust builder stage, which is discarded from the final image. Live scan results are on the repository's Security tab.

Dependencies

Updated automatically via Renovate; base images are pinned by digest and the upstream fclones artifact is integrity-pinned (tarball sha256 on amd64, commit on arm64). Builds carry signed SBOMs and provenance attestations verifiable with gh attestation verify.

Credits

This project packages fclones (MIT) into a container image. All credit for the core functionality goes to the upstream maintainers.

Contributing

Issues and pull requests are welcome. Please open an issue first for larger changes so the approach can be discussed before implementation.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

Apache-2.0. See LICENSE.