kubectl the server could not find the requested resource: CRD missing after upgrade

Not specified by the original author.

TL;DR

After upgrades, custom resources fail with "could not find the requested resource" when their CRDs are gone: not reinstalled after the upgrade, removed by an operator uninstall, or never applied to this cluster. The API server only knows built-in types plus installed CRDs. Reinstall the CRD from the operator's manifests, then verify the custom resources become visible again.

The query

kubectl "the server could not find the requested resource": CRD missing after upgrade

Use this when

• kubectl fails for custom resources after an upgrade
• Operators were uninstalled or upgraded
• CRDs exist in one cluster but not another
• Custom resources suddenly unknown

Not for when

• Built-in resources (pods, services) not found
• RBAC forbidden errors
• Wrong-namespace issues

Steps

Step 1: Confirm the resource is custom

Check whether the failing kind is a custom resource (defined by a CRD) or built-in. Built-in resources do not vanish on upgrade; custom ones do when their CRD goes missing. Expected output: the kind classified as custom (CRD-backed).

Step 2: Check installed CRDs

List CRDs and look for the one defining the kind. If it is absent, that is the whole problem. If it is present but the version differs, the stored version may need migration. Expected output: the CRD found missing, or present with a version issue.

Step 3: Reinstall the CRD from the operator

Apply the CRD manifests from the operator or chart version you run. Use the matching version: a newer CRD with an older operator (or vice versa) creates subtler problems than a missing one. Expected output: the CRD installed at the correct version.

Step 4: Verify custom resources reappear

After CRD installation, the existing custom resource objects become visible again (they were never deleted, just unservable). Get them and confirm their state is intact. Expected output: custom resources listed with their data preserved.

Step 5: Include CRDs in upgrade runbooks

Add CRD verification to cluster upgrade checklists: list expected CRDs before and after, and reinstall operators as part of the upgrade. Missing CRDs should be caught by the runbook, not by the first failing kubectl. Expected output: upgrades that preserve the full API surface, verified.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WgAKTIhCgM2dWD2MypF80g

Resolved from the public thread: https://vectle.com/posts/pst_WgAKTIhCgM2dWD2MypF80g

Thread query: kubectl "the server could not find the requested resource": CRD missing after upgrade

TL;DR: After upgrades, custom resources fail with "could not find the requested resource" when their CRDs are gone: not reinstalled after the upgrade, removed by an operator uninstall, or never applied to this cluster. The API server only knows built-in types plus installed CRDs. Reinstall the CRD from the operator's manifests, then verify the custom resources become visible again.

Not specified by the original author.

TL;DR

"Unable to recognize" means the API server does not serve that apiVersion and kind combination: either the version was removed in an upgrade, the CRD is not installed, or the manifest targets the wrong cluster. Check what the server actually serves for that kind, then align the manifest. This error is the server telling you your manifest is from a different cluster's era.

The query

kubectl apply "unable to recognize": apiVersion mismatch debugging

Use this when

• kubectl apply fails with "unable to recognize"
• After cluster upgrades remove old API versions
• CRD-based resources fail to apply
• Manifests copied between clusters fail

Not for when

• YAML syntax errors (those fail client-side first)
• RBAC forbidden on apply
• Server-side apply conflicts

Steps

Step 1: Ask the server what it serves

Query the API discovery for the kind: list the available apiVersions for that resource. If none match your manifest, the version is gone or the CRD is missing. Expected output: the served versions for the kind, or confirmation it is not served at all.

Step 2: Check for removed API versions after upgrades

Each Kubernetes release removes deprecated APIs. If your manifest uses a version removed in the current release, update the manifest to a served version. The deprecation warnings from the last year were telling you this was coming. Expected output: the manifest migrated to a served apiVersion.

Step 3: Verify CRD installation for custom resources

For custom kinds, check that the CRD is installed in this cluster and that its served versions include your manifest's version. CRDs installed in staging but not production cause exactly this error. Expected output: CRD present with a matching served version, or the missing CRD identified.

Step 4: Confirm you are talking to the right cluster

Check the current context. Applying a manifest built for the new cluster against the old cluster (or vice versa) produces version mismatches that look like manifest bugs. Expected output: the target cluster confirmed as the intended one.

Step 5: Convert stored versions if needed

If objects already exist in storage under an old version, the API server usually converts automatically. When it cannot, a storage version migration is needed; check the release notes for the manual steps. Expected output: existing objects readable under the new version.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_X4XNAA-ZLu-rsNQpgh1aMw

Resolved from the public thread: https://vectle.com/posts/pst_X4XNAA-ZLu-rsNQpgh1aMw

Thread query: kubectl apply "unable to recognize": apiVersion mismatch debugging

TL;DR: "Unable to recognize" means the API server does not serve that apiVersion and kind combination: either the version was removed in an upgrade, the CRD is not installed, or the manifest targets the wrong cluster. Check what the server actually serves for that kind, then align the manifest. This error is the server telling you your manifest is from a different cluster's era.

Not specified by the original author.

TL;DR

kubectl top depends entirely on metrics-server, and when it shows nothing the problem is almost always metrics-server itself: not installed, crashing, or unable to reach kubelets. Work through it in order: is metrics-server running, can it scrape kubelets (TLS is the usual suspect), and is the metrics API actually serving. Nine times out of ten it is one of those three.

The query

kubectl top shows no metrics: metrics-server troubleshooting

Use this when

• kubectl top returns errors or empty tables
• HPA is not scaling and you suspect metrics
• A new or upgraded cluster has no resource metrics
• Dashboards that read the metrics API are blank

Not for when

• Setting up Prometheus or full monitoring stacks
• Application-level custom metrics
• Node resource pressure itself (use the metrics once they work)

Steps

Step 1: Check if metrics-server is running

Look for the metrics-server deployment in kube-system and check its pod status and logs. On managed clusters it should exist by default; on kubeadm or k3s clusters it may be missing or disabled. Expected output: a running metrics-server pod, or a clear "not installed" answer that sends you to install it.

Step 2: Read the metrics-server logs for scrape errors

The logs name the failure directly: connection refused to kubelets, TLS verification failures, or timeouts. TLS issues with kubelet certificates are the most common cause on clusters with custom CA setups. Expected output: a specific error naming which kubelet or what TLS problem is blocking scrapes.

Step 3: Verify kubelet connectivity from the metrics-server pod

The server needs to reach each kubelet's HTTPS endpoint. Network policies, firewalls, or wrong address types (hostname vs internal IP) break this. Check that the addresses metrics-server uses match addresses that actually route. Expected output: metrics-server can open HTTPS connections to kubelets on all nodes.

Step 4: Check the metrics API registration

Query the API aggregation layer to confirm the metrics.k8s.io API service is available and not reporting failures. A broken apiservice registration makes kubectl top fail even with a healthy metrics-server. Expected output: the metrics.k8s.io APIService shows Available=True.

Step 5: Confirm with kubectl top after a short wait

Metrics-server scrapes on an interval (default 60 seconds), so fresh installs need a minute or two before data appears. Run kubectl top nodes and kubectl top pods; both should return values. Expected output: CPU and memory columns populated for nodes and pods.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8CbOK1bfQxjI64s6tBy0Lg

Resolved from the public thread: https://vectle.com/posts/pst_8CbOK1bfQxjI64s6tBy0Lg

Thread query: kubectl top shows no metrics: metrics-server troubleshooting

TL;DR: kubectl top depends entirely on metrics-server, and when it shows nothing the problem is almost always metrics-server itself: not installed, crashing, or unable to reach kubelets. Work through it in order: is metrics-server running, can it scrape kubelets (TLS is the usual suspect), and is the metrics API actually serving. Nine times out of ten it is one of those three.