google-cloud-storage-backup
Docker app from vinid223's Repository
A simple way to backup important files to Google Cloud Storage.
Overview
Readme
View on GitHubgcloud-storage-docker
Note : This repository is maintained on a private Forgejo server (CI/CD via Forgejo Actions — see .forgejo/workflows/README.md) and is replicated in realtime on github.
Github repository : https://github.com/vinid223/GcloudStorage-docker Docker Hub : https://hub.docker.com/r/vinid223/gcloud-storage-backup
gcloud-storage-docker is a Docker container which backs up one or more folders to Google Cloud Storage using the gcloud storage tool.
Upgrading from 1.x? Version 2.0 replaces gsutil with
gcloud storageand no longer accepts HMAC keys.ACCESS_KEY,SECRET_KEYandBOTO_FILEare gone; you now mount a directory holding a service account JSON key at/secretsinstead. See Migrating from 1.x. The old configuration is kept for reference under Deprecated: 1.x configuration.
Configuration (2.x)
Mounts
| Container path | Description |
|---|---|
/data |
The folders to back up, each mounted under it (ex: -v /home/user/Documents:/data/documents). |
/secrets |
A host directory containing your service account JSON key, named sa.json (ex: -v /path/on/host/secrets:/secrets:ro). The key is always read from /secrets/sa.json; read-only is recommended. See How to get your service account key file. |
Required
gcloud-storage-docker is configured by setting the following environment variables during the launch of the container.
| Variable | Description |
|---|---|
GCSPATH |
Your GCS bucket and path (ex: gs://personal-backup-bucket/). |
Optional
| Variable | Default | Description |
|---|---|---|
GCSOPTIONS |
(none) | Extra parameters for gcloud storage rsync (ex: -d to make the bucket an exact copy of the local files, deleting objects that no longer exist locally). See Backup options. |
PROJECT_ID |
(none) | GCP project, if your bucket needs one to be set. |
CRON_SCHEDULE |
0 * * * * |
When to run the backup, in standard CRON notation (every hour by default). |
TZ |
UTC |
Zoneinfo name (ex: America/Montreal) used for both the schedule and the log timestamps. An unknown name falls back to UTC and logs a warning at startup. |
RUN_ON_STARTUP |
false |
Set to true to run one backup as soon as the container starts, instead of waiting up to a whole schedule interval to find out whether the configuration works. |
LOG_MAX_BYTES |
10485760 (10 MB) |
The backup log (/var/log/cron.log) is trimmed to its most recent half once it grows past this, so an hourly backup cannot fill the container's disk over months. Set 0 to never trim. |
CLOUDSDK_* |
(none) | Any Google Cloud CLI setting (endpoint overrides, proxy, ...) set on the container is carried over to the scheduled backups. |
A CRON_SCHEDULE that is neither five fields nor an @macro (@daily, @hourly, ...) is rejected
at startup rather than accepted and never fired.
Removed in 2.0
ACCESS_KEY, SECRET_KEY and BOTO_FILE are no longer supported. If any of them is still set, the
container stops at startup with migration instructions. See Migrating from 1.x.
The settings menu
Every container prints a short summary of its settings when it starts — before it
validates anything, so a container that refuses to start still shows what it was
looking for. It is one line per setting, showing the value that container actually
resolved, and is the first thing in docker logs.
The full reference — every variable with its accepted values and default, the
GCSOPTIONS values and the commands — is printed only on request.
To read the full reference without starting anything:
docker run --rm vinid223/gcloud-storage-backup help
Scheduled backups do not repeat it, so it does not bury the transfer output in the log.
Checking on it
The container's stdout is the backup log, so docker logs -f <container> follows it live.
The image also declares a healthcheck that reports the outcome of the last completed backup, so
a backup that keeps failing shows up in docker ps as unhealthy instead of only in the log:
docker inspect --format '{{.State.Health.Status}}: {{(index .State.Health.Log 0).Output}}' <container>
A container that has not completed a backup yet is healthy. Use docker run --no-healthcheck to
turn it off.
To run a backup immediately, outside the schedule:
docker exec <container> sh /opt/run.sh backup
It exits 0 on success and non-zero if the sync failed, so it can be used from a wrapper script or a
monitoring check. Add -e GCSOPTIONS=-n to the docker exec for a dry run without changing the
container's own configuration.
Note:
gcloud auth listinside the container reports no credentialed accounts even when everything is working. The CLI is pointed straight at the mounted key file rather than at a stored login, so an emptyauth listis expected and is not a sign that authentication is broken. The startup log prints the service account actually in use.
Example invocation
To backup the Documents and the Photos directories in your home folder, and
running the backup at 03:00 every day, you could use something like this:
docker run -d -v /home/user/Documents:/data/documents \
-v /home/user/Photos:/data/photos \
-v /path/on/host/secrets:/secrets:ro \
-e "GCSPATH=gs://yourgsbucket/" \
-e "GCSOPTIONS=-d" \
-e "CRON_SCHEDULE=0 3 * * *" \
-e "TZ=America/Montreal" \
vinid223/gcloud-storage-backup
How to get your service account key file
- Go to https://console.cloud.google.com/iam-admin/serviceaccounts and create a service account (or pick an existing one).
- Grant it access to your bucket:
Storage Object Adminon the bucket covers everything rsync needs, including the deletions-dperforms. Without-d,Storage Object Useris enough. - Open the service account, go to the
Keystab, thenADD KEY->Create new key->JSON, and download the file. - Save it as
sa.jsonin a directory of its own on the host, and mount that directory at/secrets, as in the example above. Mounting it read-only (:ro) is recommended.
The key file is read at backup time to mint an access token; nothing is written to the container. The startup log prints which service account is in use so you can confirm the identity.
Backup options
GCSOPTIONS is passed to gcloud storage rsync. The command already runs recursively and in
parallel, so the useful additions are:
| Option | Effect |
|---|---|
--delete-unmatched-destination-objects (or -d) |
delete objects in the bucket that no longer exist locally |
--dry-run (or -n) |
print what would be copied without copying it |
--continue-on-error (or -C) |
keep going after an individual failure |
--exclude REGEX (or -x REGEX) |
skip paths matching the regex |
--ignore-symlinks (or -e) |
do not follow symlinks |
The short forms are the gsutil flags used by 1.x: they are still accepted and translated, so an
existing GCSOPTIONS does not need to be rewritten. -r, -R and -m are ignored because
gcloud storage rsync is already recursive and parallel. Any other value is passed through
untouched, so any gcloud storage rsync flag can be used directly.
--exclude takes a regular expression, and it reaches gcloud exactly as written - it is not expanded
against the files in the container. A pattern containing a space has to be quoted inside the
variable:
-e "GCSOPTIONS=-x 'my notes'" # one pattern, with a space in it
-e "GCSOPTIONS=-x .*\.log" # backslashes are kept as-is
An unbalanced quote is reported in the log and the backup exits without touching the bucket, rather than running a half-parsed command.
Symlinks
gcloud storage rsync ignores symlinks by default, where gsutil followed them, so the image passes
--no-ignore-symlinks to keep the 1.x behaviour. Pass -e (or --ignore-symlinks) in GCSOPTIONS
to skip them instead; the default is then left out rather than being contradicted.
One difference worth knowing if you have symlinks under /data: gsutil never followed directory
symlinks, regardless of -e, while gcloud storage may descend into them. If /data contains a
symlink to a directory, check the first run (GCSOPTIONS=-n shows what would be copied without
copying it) rather than assuming the contents are identical to 1.x.
Migrating from 1.x
Version 2.0 backs up with gcloud storage, the tool Google recommends, because gsutil is removed
from the Google Cloud CLI after March 2027. The catch is authentication: gcloud storage uses the
JSON API, and HMAC keys can only sign XML API requests. There is no way to keep using HMAC keys with
it, so the credentials have to change.
| 1.x | 2.0 |
|---|---|
ACCESS_KEY + SECRET_KEY (HMAC) |
A JSON key at sa.json in a directory mounted at /secrets |
BOTO_FILE (HMAC in a boto file) |
A JSON key at sa.json in a directory mounted at /secrets |
gsutil -m rsync -r |
gcloud storage rsync --recursive |
Everything else is unchanged: /data, GCSPATH, GCSOPTIONS, CRON_SCHEDULE, TZ and the
setup / backup commands all behave the same way.
Steps:
- Create a service account and download a JSON key, as described above. The HMAC keys you use
today already belong to a service account, which you can reuse - the console shows which one
under
Service account HMACin the interoperability settings. - Replace
ACCESS_KEY/SECRET_KEY(orBOTO_FILE) with a mount of the directory holdingsa.jsonat/secrets:-v /path/on/host/secrets:/secrets:ro. - Check the result before the first scheduled run, either with
RUN_ON_STARTUP=trueon the container, or withdocker exec <container> sh /opt/run.sh backupfollowed bydocker logs <container>. AddingGCSOPTIONS=-nfor that first run makes it a dry run.
If any of the removed variables are still set, the container stops at startup with these instructions rather than running a backup that cannot authenticate.
Staying on 1.x is also an option: vinid223/gcloud-storage-backup:V1.3 keeps working with HMAC keys,
and gsutil itself keeps working until Google removes it in March 2027. Pin that tag explicitly:
latest now points at 2.x.
Deprecated: 1.x configuration
Deprecated. This section only applies to
vinid223/gcloud-storage-backup:V1.3and earlier, which back up with gsutil. It is no longer maintained and stops working when Google removes gsutil from the Google Cloud CLI after March 2027. A 2.x container given these variables refuses to start. Use it only until you have migrated.
1.x was configured with these environment variables:
ACCESS_KEY- your GCP HMAC access key for a service account. The string should not contain the character|.SECRET_KEY- your GCP HMAC secret key for a service account. The string should not contain the character|.BOTO_FILE- path to a mounted boto configuration file containing the access key, the secret key and your own settings. When set,ACCESS_KEYandSECRET_KEYare ignored.GCSPATH,GCSOPTIONS(passed togsutil -m rsync -r) andCRON_SCHEDULE, as in 2.x.
To get the HMAC keys:
- Go to the GCS Settings page, on the Interoperability tab.
- Under
Service account HMAC, click on a service account if one exists or create a new one by clicking onCREATE A KEY FOR ANOTHER SERVICE ACCOUNT. - Click on
CREATE A KEYto get the Access Key and Secret Key.
Example invocation with keys in environment variables:
docker run -d -v /home/user/Documents:/data/documents \
-v /home/user/Photos:/data/photos \
-e "ACCESS_KEY=YOURACCESSKEY" \
-e "SECRET_KEY=YOURSECRETKEY" \
-e "GCSPATH=gs://yourgsbucket/" \
-e "GCSOPTIONS=-d" \
-e "CRON_SCHEDULE=0 3 * * *" \
vinid223/gcloud-storage-backup:V1.3
Or with a boto file, mounted into the container and pointed at by BOTO_FILE:
[Credentials]
gs_access_key_id=YOURACCESSKEY
gs_secret_access_key=YOURSECRETKEY
[Boto]
[GoogleCompute]
[GSUtil]
content_language = en
default_api_version = 2
[OAuth2]
docker run -d -v /home/user/Documents:/data/documents \
-v /home/user/Photos:/data/photos \
-v /path/to/local/boto/file:/tmp/boto.tmp \
-e "BOTO_FILE=/tmp/boto.tmp" \
-e "GCSPATH=gs://yourgsbucket/" \
-e "GCSOPTIONS=-d" \
-e "CRON_SCHEDULE=0 3 * * *" \
vinid223/gcloud-storage-backup:V1.3
Development
docker build -t gcloud-storage-backup .
tests/ holds two suites - a fast one that asserts the exact command the image builds, and an
end-to-end one that runs real transfers against a local Google Cloud Storage stand-in. See
tests/README.md. CI runs the fast one on every push, plus shellcheck and
hadolint; the end-to-end suite is meant to be run locally before a release.
AI disclaimer
AI tools are used to help maintain this project, including writing code, tests and documentation. Every change is reviewed by the maintainer before it is committed and released.
Inspiration
Categories
Download Statistics
Total Downloads Over Time
Related apps
Explore more like this
Explore allLinks
Details
vinid223/gcloud-storage-backupRuntime arguments
- Privileged
- false