the agent's kubectl is 1.29 but the cluster is 1.27 - the migration applied a manifest the API server rejected

TL;DR: kubectl 1.29 against a 1.27 cluster is two minors of skew - outside the supported plus-or-minus-one policy. Install a kubectl matching the cluster (1.27 or 1.28), re-validate with a server-side dry-run, and apply again. Pin the kubectl version per cluster in the runbook.

the agent's kubectl is 1.29 but the cluster is 1.27 - the migration applied a manifest the API server rejected

1. Confirm the skew on both sides.

kubectl version --output=yaml
# compare client version against server version

Expected: client reports 1.29, server reports 1.27 - two minors apart, outside the supported skew.

2. Install a kubectl within one minor of the server (1.28 or 1.27), using the project's version manager, a pinned download, or the cloud provider's CLI plugin.

Expected: the new kubectl version reports 1.27 or 1.28.

3. Validate the manifest against the real API server before applying.

kubectl apply --dry-run=server -f MANIFEST_FILE

Expected: server-side dry-run passes, meaning the API server itself accepts the manifest - client-side validation alone is not enough.

4. Apply for real and verify the resources.

kubectl apply -f MANIFEST_FILE
kubectl get -f MANIFEST_FILE

Expected: apply succeeds and the resources show the intended state.

5. Pin the kubectl version per cluster in the migration runbook and CI image so the next run cannot drift to latest.

Expected: a fresh run uses the pinned kubectl automatically.

Use this when

• kubectl apply fails against an older cluster with API or field errors
• The agent's kubectl is two or more minors newer than the API server
• A manifest validates locally but the server rejects it
• You are migrating workloads across clusters with different versions

Not for this skill when

• Client and server are within one minor and apply still fails - that is a manifest or RBAC problem, not skew
• The error is authentication or authorization - fix credentials and roles, not the CLI version
• The API was removed server-side (like old beta APIs) - then rewrite the manifest for a supported API version regardless of kubectl
• You manage kubectl via a wrapper that already pins versions - check the wrapper's config first

Variant phrasings

• kubectl version skew cluster rejected manifest
• kubectl 1.29 vs kubernetes 1.27 apply fails
• api server rejected manifest newer kubectl
• kubectl too new for cluster, how to downgrade

Why it happens

Kubernetes supports kubectl skew of one minor version in either direction. Beyond that, the newer client can send fields, defaults, or API versions the older API server does not know, and client-side validation passes because it validates against the client's newer schemas. The server is the authority on what it accepts, which is why the failure only appears at apply time - and why server-side dry-run is the honest pre-check.

Edge cases

• Server-side dry-run needs the same RBAC permissions as a real apply. A dry-run auth failure is a permissions problem, not a manifest problem.
• Deprecated APIs removed server-side fail no matter which kubectl you use. Check the API version in the manifest against the cluster's supported list.
• Multiple clusters at different versions need multiple kubectl versions. Direnv-style per-directory pins or versioned aliases prevent cross-talk.
• Some managed providers bundle their own kubectl plugin versions. If the provider manages the binary, pin through the provider's tooling rather than a manual download.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wa3U2EvoIAvN7u3a0ZotQA