cloud-billing-exporter

One Prometheus endpoint for every cloud bill: Azure, Microsoft 365 / Defender licensing, AWS, GCP, Google Workspace and Hetzner.

Runs on tower (Unraid, root@10.0.0.24) as the container cloud-billing-exporter on :9825. Prometheus job cloud_billing. Dashboard Cloud — Billing & Spend (/d/cloud-billing), alert group Cloud in the Alerting folder. Dashboard and alerts are built by scripts/build_cloud_billing.py in the grafana-homelab repo.

The policy it enforces

Target spend is £0/month everywhere, with exactly one legitimate exception: Microsoft Defender for Endpoint licences. Google Workspace and one Hetzner storage box are standing subscriptions with known prices. Anything else nonzero, anywhere, is an incident — not something to discover on a statement.

Each provider has a baseline in config.json, in that provider's own billing currency. Month-to-date spend above the baseline alerts. When a charge turns out to be legitimate and permanent, raise the baseline — do not silence the rule.

Deploy

scp -r Docker/cloud-billing-exporter root@10.0.0.24:/mnt/user/appdata/
ssh root@10.0.0.24 'bash /mnt/user/appdata/cloud-billing-exporter/deploy.sh'

Idempotent. It deploys and passes with zero credentials in place — every provider then reports cloud_provider_configured 0. That is a real state, not a failure, and it is deliberate: the plumbing gets proven before anyone goes round four cloud consoles. Add credentials one at a time and re-run.

Rollback:

docker rm -f cloud-billing-exporter
IMAGE=cloud-billing-exporter:rollback bash /mnt/user/appdata/cloud-billing-exporter/deploy.sh --no-build

Tests: python3 test_exporter.py (stdlib unittest, no network, 30 cases).

Credentials

The easy path — run the guided installer from your Mac:

bash Docker/cloud-billing-exporter/setup-credentials.sh

It takes one provider at a time: prints where to click and the exact minimum scope, takes the values (hidden input), checks them against the real API where it can, and installs them on tower with the right owner and mode. It is re-runnable and resumable, so you can do Hetzner tonight and Google at the weekend. At the end it applies, shows which providers came up, and helps set the baselines from the figures actually observed rather than from guesswork.

Secrets typed into it are never echoed, never written to a file on the Mac, and never passed as command-line arguments — argv is world-readable via ps, so the live checks feed curl through --config - and python through stdin, and values reach tower by piping into ssh ... 'cat > file'.

The rest of this section is the reference for doing it by hand.

All read-only. All created by a human. Nothing here creates or rotates them.

Each file lives in /mnt/user/appdata/cloud-billing-exporter/config/, mode 600, owned by uid 9825. The directory itself is mode 700 owned by 9825 — a root-owned 700 directory is not traversable by the exporter and the container dies at startup on the config file with a permission error whose file modes all look correct.

config.json holds the non-secret settings — baselines, unit prices, poll intervals, FX rates. It is seeded from config.example.json on first deploy and is safe to commit patterns of, but the config directory is never in git.

Two provider caveats worth knowing before you go looking

Hetzner needs one token per project, and covers more than storage. Hetzner scopes an API token to a single project, so if you run more than one Hetzner project (e.g. a storage-box project and a servers project) you need one token file per project, named hetzner-token.<label> — the label comes from the filename because the API exposes no project-name endpoint. A bare hetzner-token still works and is labelled default.

A project with no token is completely invisible, including its servers. The collector enumerates every billable kind — storage boxes, servers, volumes, load balancers, floating IPs and primary IPs — because a project holding a server and no storage box would otherwise have reported a cost of zero, which is precisely the silent blind spot this exporter exists to prevent.

Prices are resolved per resource (embedded prices where the API provides them, /v1/pricing for volumes and IPs). Anything that cannot be priced is counted as unpriced and alerted on rather than treated as free — an undercounted total would read as a price drop. cloud_hetzner_resources is the count, and it is the stronger tripwire: whether or not a price resolves, a resource appearing in a project that should hold none is the event that matters.

Hetzner Storage Boxes are not on the Robot API any more. Robot Web Service support for Storage Boxes ended 2025-07-29; they moved to the unified API at api.hetzner.com. A Robot webservice user will not work here. (Verifiable: https://api.hetzner.com/v1/storage_boxes returns 401 for a bad token, while https://api.hetzner.cloud/v1/storage_boxes returns api route not found.)

One key per Google ACCOUNT. A second Google identity — a personal gmail paying for the Nest / Smart Device Management API, say — has its own billing account and its own spend, and is completely invisible to the first account's credential. Add it as gcp-sa.<label>.json; the label comes from the filename. The Cloud Billing API must be enabled separately in each account.

Because several keys can be present, workspace.service_account names which one carries the domain-wide delegation. Guessing would silently point Workspace at the wrong tenant and report the wrong seat count.

gcp.expected_billable_projects is GCP's baseline. GCP has no spend API, so the tripwire is how many projects are billing-enabled — a project that can charge you. Zero is not everyone's correct answer: an account that legitimately pays for something has a billable project by design, and alerting on its mere existence forever would train you to ignore the alert. Declare the expected number; anything above it is the incident.

No GCP billing account at all? Set gcp.enabled: false. GCP and Workspace share gcp-sa.json, so with the key installed the GCP collector will try to poll even if you only wanted Workspace — and if the Cloud Billing API is off for that project it fails every cycle and raises a critical Cloud | Billing poll stale. The flag makes it report configured 0 instead.

Be clear about what that costs: a disabled provider will not notice a GCP billing account appearing later. It is an honest "not covered", not a healthy zero. The alternative is free — leave it enabled and switch on the Cloud Billing API for the service account's project; it then reports zero billing accounts and alerts if that ever changes. Disabling GCP does not affect Workspace.

Google publishes no direct-customer billing API. Workspace subscription cost is only reachable through the Admin console or the reseller Admin SDK. So the Workspace half reports billable seat count, and models cost as seats × the workspace.seat_price you declare. A seat change is what actually moves the bill and is what gets alerted. Likewise, the GCP Cloud Billing API returns account metadata and price lists, not spend — so the GCP tripwire is structural: with zero billing-enabled projects the account cannot accrue charges, and a project appearing is the alert. Real per-day GCP cost would need a BigQuery billing export.

What it costs to run

AWS Cost Explorer bills USD 0.01 per GetCostAndUsage request. The exporter polls AWS once a day — one request, covering the whole month-to-date at daily granularity — so it accrues about USD 0.31/month. cloud_aws_cost_explorer_requests_total exposes the running count so the figure is auditable rather than assumed.

Every other provider's API is free. They are polled every 6h; billing data lags 24–48h so there is nothing to gain from going faster.

Polling is on a timer, never on scrape. /metrics only ever renders a cache. If a scrape triggered a poll, Prometheus at 5m would spend roughly USD 87/month finding out we spend nothing.

Design notes that are load-bearing

A dead exporter must not look like £0. This is the whole trap. Three separate mechanisms:

1. Cost alerts are gated on cloud_provider_up == 1, so a failing poll cannot raise or suppress a spend alert on its own.
2. Every provider exposes cloud_provider_last_success_timestamp, and a separate staleness alert fires when it stops advancing. A provider with no credential has a timestamp of 0, so the staleness rule is gated on cloud_provider_configured == 1 — otherwise every unconfigured provider would alert as stale forever.
3. Coverage is asserted with count(cloud_provider_up), not absent(). A provider that vanishes produces no cost series at all, and an absent series cannot fire a threshold rule — nor an absent() rule you never thought to write for it.

On a failed poll the cache keeps the last good cost values and freezes the timestamp, rather than publishing zeros. A transient API blip must not read as "spend went to zero".

No host label, ever. Every scrape job in this estate attaches host: tower as a target label. A metric carrying its own host is not merged and does not error — Prometheus renames it to exported_host and keeps the target's value, so every alert would name tower instead of the provider. Provider identity lives in provider. deploy.sh fails the deploy if any metric carries a host label.

Alerts compare native currency against native baselines. cloud_cost_mtd carries a currency label; cloud_cost_mtd_gbp is a converted convenience for the dashboard's single total. An FX rate must never be able to manufacture or mask an incident.

Because of that currency label, the join in the spend alert needs ignoring(currency). A plain - between cloud_cost_mtd and cloud_cost_baseline matches nothing and yields an alert that can never fire and looks exactly like "nothing is wrong". The expressions are therefore unit-tested with promtool test rules, not eyeballed:

docker exec prometheus promtool test rules /etc/prometheus/tests/cloud-billing_test.yml

Sources for those tests live in grafana-homelab/prometheus/tests/.

Stdlib only, so there is nothing to pip install and no third-party code in the image. The one external binary is openssl, which RS256-signs the Google service-account JWT — there is no RSA in the standard library. python:3.13-slim ships libssl but not the openssl CLI, so the Dockerfile installs it; without it the Google collectors fail on first poll with a FileNotFoundError that says nothing about the cause.

Metrics

Gross, not net, for Hetzner — the invoice is gross and reporting net would look like a price cut. cloud_cost_mtd is the sum across every project, so one baseline still covers the lot, with per-project detail alongside it.

Alerts (group Cloud, folder Alerting)

severity drives routing: critical goes to Discord + Matrix, warning to Discord.

Known gap: the two offset 1d seat-change rules cannot fire until 24h of history exists for that series — so they are silent for the first day after deploy, and after any gap longer than the series' retention. This is documented in the promtool tests rather than left to be rediscovered.