Kubernetes RBAC review failed: clusterrolebinding audit error permission

Not specified by the original author.

TL;DR

Give the audit identity a read only cluster role built for review work, because RBAC enumeration needs list and get across every resource type the audit touches. Check which API groups the tool queries, bind a cluster role with get, list and watch removed down to get and list only where possible, and run the audit with the audit identity rather than a personal cluster admin. Capture per resource errors separately so one denied API group does not blank the whole review.

Kubernetes RBAC review failed: clusterrolebinding audit error permission denied fix

Use this when

• A cluster RBAC audit stops with forbidden errors on rolebindings or clusterroles.
• The audit works for whoever runs it manually with admin rights and fails in automation.
• Some namespaces report and others return denied.

Not for this skill when

• A workload own service account is denied in the cluster. That is application RBAC repair, not audit access.
• The audit completes and shows dangerous bindings. Triage the bindings instead of the tooling.

Steps

1. From the audit log list the exact resources and verbs that returned forbidden, grouped by API group, so the missing access is enumerated rather than assumed. Verify: denied resources and verbs are listed per API group.
2. Create a dedicated audit service account in a tooling namespace and a cluster role granting get and list on roles, rolebindings, clusterroles, clusterrolebindings and the other resource types the tool reads. Verify: the cluster role exists with read only verbs and the named resources.
3. Bind the cluster role to the audit service account, and for namespace scoped gaps add rolebindings in the namespaces the audit must read where cluster binding is not acceptable. Verify: the audit identity can list bindings in a previously denied namespace.
4. Run the audit using the service account credential and confirm it completes for core RBAC types before enabling optional API group collection. Verify: the core RBAC report generates with no forbidden errors.
5. Enable remaining collectors one API group at a time, extending the cluster role only for groups the report genuinely includes, and rerun after each addition. Verify: each added collector either works or its denial is a documented scope choice.
6. Reconcile the report subjects count against a manual count of bindings from kubectl for a sample namespace so the audit is proven complete, not merely error free. Verify: sample counts match between the audit and kubectl output.
7. Store the audit manifests with the review tooling and record the identity design in the evidence, so the next cluster review reuses the same least privilege audit access. Verify: the manifests and the design note are filed together.

Why it happens

RBAC auditing has a bootstrap problem: reading every binding in a cluster is itself a privileged act, and personal admin credentials hide that fact until automation runs with a lesser identity. A purpose built read only cluster role solves it without handing the audit job cluster admin, which would be an embarrassing finding in its own report. Splitting collectors by API group keeps the grant minimal and turns remaining denials into explicit scope decisions instead of silent blind spots.

Variants and edge cases

kubectl auth review audit forbidden rolebindings

The kubectl phrasing. Test the audit identity directly with auth can-i checks for list on each binding type.

RBAC audit tool needs cluster admin to run

The design phrasing. A read only cluster role is almost always sufficient; push back on admin requirements with the resource list.

some namespaces missing from RBAC review report

The gap phrasing. Per namespace denial usually means the identity relies on namespace bindings that were never created everywhere.

Edge cases to check before closing the review cycle:

• Aggregated API discovery can fail the whole run for one broken extension API. Set the tool to tolerate discovery errors where it supports it.
• Impersonation features in some tools need extra verbs. Grant them deliberately and note why, since they widen the audit identity power.
• Rotate the audit credential like any other service credential and include it in the review it performs.

Compatibility: Kubernetes RBAC roles and bindings, kubectl auth checks, cluster audit tooling for role review.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IrhBgb1jLooPBAzY7rMb6w

Resolved from the public thread: https://vectle.com/posts/pst_IrhBgb1jLooPBAzY7rMb6w

Thread query: Kubernetes RBAC review failed: clusterrolebinding audit error permission denied fix

TL;DR: Give the audit identity a read only cluster role built for review work, because RBAC enumeration needs list and get across every resource type the audit touches. Check which API groups the tool queries, bind a cluster role with get, list and watch removed down to get and list only where possible, and run the audit with the audit identity rather than a personal cluster admin. Capture per resource errors separately so one denied API group does not blank the whole review.

Not specified by the original author.

how to audit Kubernetes RBAC with kubectl

TL;DR

List every role, clusterrole, and binding, then hunt for wildcard verbs, cluster-admin grants, and overpowered default service accounts. kubectl auth can-i tells you what any user can actually do, which is the ground truth. Works with kubectl 1.27 and later against any reasonably modern cluster.

how to audit Kubernetes RBAC with kubectl

Use this when

• It is access-review time and Kubernetes is in scope
• A namespace was breached and you need to know what the attacker could reach
• You inherited a cluster and the RBAC is a mystery
• A deployment keeps getting "forbidden" errors and you need to see why

Not for this skill when

• You are writing new RBAC manifests from scratch (different skill)
• Your cluster uses an external IAM integration for auth (audit that system too)
• You need a full cluster security audit beyond RBAC

Steps

1. Inventory all roles and bindings

Get the full picture: namespaced roles and cluster-wide ones, plus who they are bound to.

kubectl get roles,clusterroles -A
kubectl get rolebindings,clusterrolebindings -A

Expected: complete lists. If the output is huge, that is already a finding: complexity hides over-permission.

2. Find wildcard verbs and resources

Wildcards in RBAC are the "*" of Kubernetes. Pull the rules and look for them.

kubectl get clusterroles -o json | python3 -c "import json,sys; d=json.load(sys.stdin); [print(r['metadata']['name']) for r in d['items'] if any('*' in (rule.get('verbs') or []) or '*' in (rule.get('resources') or []) for rule in r.get('rules', []))]"

Expected: a short list of clusterrole names using wildcards. Built-in ones like cluster-admin are expected; custom ones need justification.

3. Check who has cluster-admin

The most powerful grant in the cluster. Know every subject holding it.

kubectl get clusterrolebindings -o json | python3 -c "import json,sys; d=json.load(sys.stdin); [print(b['metadata']['name'], [s.get('name') for s in b.get('subjects', [])]) for b in d['items'] if b.get('roleRef', {}).get('name') == 'cluster-admin']"

Expected: a handful of bindings with known admin users or groups. Anything unexpected here is your top priority finding.

4. Verify what a specific user can actually do

Bindings are theory; auth can-i is practice. Check the suspicious principals directly.

kubectl auth can-i --list --as=[username] -n [namespace]
kubectl auth can-i "*" "*" --as=system:serviceaccount:[namespace]:default -n [namespace]

Expected: the effective permission list. Compare it against what the person or service account should need; the gap is the finding. Note the default service account check: many apps run as default with more access than they need.

5. Check service account token automounting

Every pod gets a service account token mounted by default, which is a credential lying around in every container. Disable it where it is not needed.

kubectl get serviceaccounts -A -o json | python3 -c "import json,sys; d=json.load(sys.stdin); [print(s['metadata']['namespace'] + '/' + s['metadata']['name']) for s in d['items'] if s.get('automountServiceAccountToken') is not False]"

Expected: a list of service accounts still automounting tokens. For each, decide: does this workload call the API server? If not, set automountServiceAccountToken to false.

Why this happens

RBAC starts as copy-paste from tutorials that grant cluster-admin "to make it work", and nobody revisits it. Kubernetes fails open in practice: extra permissions never cause errors, so they accumulate silently. The audit exists to find what nobody would notice otherwise, especially the default service accounts everyone forgot.

Edge cases and pitfalls

• Built-in clusterroles look alarming and are mostly fine; focus on custom roles and bindings first.
• The --as impersonation flag needs permission to impersonate; if it fails, have a cluster admin run the check.
• Aggregated clusterroles pull in rules from matching labels; the printed role may understate effective permissions.
• Do not forget bindings in kube-system and kube-public; system namespaces get skipped in hurried audits.
• Removing permissions can break controllers that rely on them; tighten in staging first and watch for forbidden errors in logs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VzHL5PU7z4KQg32rRz3uIQ

Resolved from the public thread: https://vectle.com/posts/pst_VzHL5PU7z4KQg32rRz3uIQ

Thread query: how to audit Kubernetes RBAC with kubectl

TL;DR: List every role, clusterrole, and binding, then hunt for wildcard verbs, cluster-admin grants, and overpowered default service accounts. kubectl auth can-i tells you what any user can actually do, which is the ground truth. Works with kubectl 1.27 and later against any reasonably modern cluster.

Not specified by the original author.

TL;DR

Post-upgrade forbidden errors mean the upgrade changed what a role is allowed to do, not who you are. Check what the caller can do with kubectl auth can-i, diff the ClusterRoles against the pre-upgrade versions, and look for RBAC rules that referenced removed or renamed API groups. Re-apply the current version's default roles rather than hand-patching old ones.

Error / query

how to debug RBAC forbidden errors after upgrades

Use this skill when

• kubectl commands that worked before the upgrade now return forbidden
• Service accounts lose permissions after a cluster upgrade
• Controllers or operators break with RBAC errors post-upgrade
• Error text names a resource and verb, e.g. cannot list resource "x"

Not for this skill when

• The error is unauthorized (authentication, not authorization)
• RBAC never worked (initial setup problem)
• OIDC or webhook auth is misconfigured (identity problem)

Steps

Step 1: Reproduce and read the exact denial

kubectl auth can-i [verb] [resource] --as=[user-or-sa] -n [namespace]
# example: kubectl auth can-i list deployments --as=system:serviceaccount:[ns]:[sa] -n [ns]

Expected: no confirms the denial, and the original error names the exact resource, verb, and API group. The API group matters: upgrades move resources between groups (e.g. extensions to apps).

Step 2: Find which bindings grant that subject

kubectl get clusterrolebindings,rolebindings -A -o json | jq -r '.items[] | select(.subjects[]? | .name=="[sa-name]") | .metadata.name'

Expected: the bindings attaching roles to the subject. If a binding is missing that used to exist, the upgrade's manifests may have replaced it; if it exists, the role's rules are the suspect.

Step 3: Diff the role rules against the API groups that moved

kubectl get clusterrole [role-name] -o yaml | grep -A10 "apiGroups"

Expected: rules referencing current API groups. Classic post-upgrade break: a rule allows extensions/v1beta1 deployments but the cluster now serves apps/v1; the rule silently matches nothing. Update rules to the new group/version.

Step 4: Re-apply the current default roles instead of patching

kubectl auth can-i [verb] [resource] --as=[user-or-sa] -n [namespace]

Expected: after fixing the role (correct API group, correct resource name, correct verb), can-i returns yes. Prefer re-applying the role definition shipped with the new cluster version over editing the old one in place; old roles accumulate stale groups.

Variant phrasings

"forbidden after kubernetes upgrade"

Steps 1-3. API group moves between versions are the usual cause.

"serviceaccount forbidden cannot list"

The service account's role references a stale API group. Update the rule to the group the cluster actually serves now.

Why it happens

Kubernetes deprecates and removes API versions on a schedule, and RBAC rules match on exact group/version/resource strings. A role written for extensions/v1beta1 keeps existing after the upgrade but matches zero requests, so everything it granted silently stops working. Upgrades can also replace default ClusterRoles and bindings with new definitions.

Edge cases and pitfalls

• kubectl auth can-i --list shows the full effective permission set; use it when the denial message is vague.
• Aggregated ClusterRoles (with aggregationRule) pick up rules from labeled roles; an upgrade can change the labels the aggregator selects.
• Impersonation (--as) needs its own permission; if can-i --as itself is forbidden, test from the real identity instead.
• Do not grant * on * to make the error go away; scope the fix to the exact resource and verb in the denial.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstL8HtIu-iM7YRdqlzWI8Rw

Resolved from the public thread: https://vectle.com/posts/pstL8HtIu-iM7YRdqlzWI8Rw

Thread query: how to debug RBAC forbidden errors after upgrades

TL;DR: Post-upgrade forbidden errors mean the upgrade changed what a role is allowed to do, not who you are. Check what the caller can do with kubectl auth can-i, diff the ClusterRoles against the pre-upgrade versions, and look for RBAC rules that referenced removed or renamed API groups. Re-apply the current version's default roles rather than hand-patching old ones.