Cerbos PDPs keep serving stale policies when Hub is unreachable

TL;DR

When a policy change does not take effect, check the cerbosdevhub_connected Prometheus gauge first: 0 means the PDP is disconnected and serving the cached bundle, 1 means connected.

Steps

1. When a policy change does not take effect, check the cerbosdevhub_connected Prometheus gauge first: 0 means the PDP is disconnected and serving the cached bundle, 1 means connected. For resilience, set a cacheDir on a persistent volume so restarts do not re-download, and as a last resort switch the storage driver to git to read policies straight from the repo. Note PDPs need outbound HTTPS to api.cerbos.cloud and cdn.cerbos.cloud on 443 with TLS 1.3, and they honor the standard proxy env vars.

When to use

You are seeing this: Note PDPs need outbound HTTPS to api.cerbos.cloud and cdn.cerbos.cloud on 443 with TLS 1.3, and they honor the standard proxy env vars. Use this skill when you run into "Cerbos PDPs keep serving stale policies when Hub is unreachable".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note PDPs need outbound HTTPS to api.cerbos.cloud and cdn.cerbos.cloud on 443 with TLS 1.3, and they honor the standard proxy env vars.

Source: Official docs (Cerbos Hub reliability): documents a gotcha that trips agents debugging policy updates. When Cerbos Hub is unavailable, running PDPs keep serving authorization requests from the last downloaded bundle while reconnecting in the background, so a policy change that never arrives looks exactly like a policy that deployed fine. A local cacheDir avoids re-downloading on restart but does not enable an offline start: the PDP still contacts Hub at boot and fails to start if it cannot be reached.

Verified resolution: When a policy change does not take effect, check the cerbosdevhub_connected Prometheus gauge first: 0 means the PDP is disconnected and serving the cached bundle, 1 means connected. For resilience, set a cacheDir on a persistent volume so restarts do not re-download, and as a last resort switch the storage driver to git to read policies straight from the repo. Note PDPs need outbound HTTPS to api.cerbos.cloud and cdn.cerbos.cloud on 443 with TLS 1.3, and they honor the standard proxy env vars.

Source URL: https://docs.cerbos.dev/cerbos-hub/reliability