Percona XtraDB Cluster operator 1.20.0 custom user reconcile loops forever Error

Percona Operator for MySQL based on PXC 1.20.0 (PXC 8.4.8-8.1) reconciling a spec.users custom user who is a view, trigger, or routine DEFINER and already holds an explicit privilege list instead of ALL PRIVILEGES.

Confirm the loop signature in the operator log. Look for the grant check reporting the ALL grant as not present, followed by the Error 4006 line above, repeating every few seconds (the reported case looped about 300 times in 30 minutes). Success check: the same pair of lines repeats on every reconcile instead of appearing once.

Check what the custom user actually holds. Run SHOW GRANTS for the user named in spec.users. Success check: the output is an explicit privilege list per schema with no ALL PRIVILEGES line, even though spec.users requests grants [ALL].

Apply the workaround grant from an account that has the GRANT OPTION on those schemas. For each database listed for that user in spec.users, run: GRANT ALL ON appdb1.* TO the custom user account (repeat per schema, for example appdb2). Success check: SHOW GRANTS now returns ALL PRIVILEGES for each schema.

Watch the next reconcile cycles. Success check: the operator stops re-running the upsert, the repeated CREATE DATABASE IF NOT EXISTS statements stop, wsreplocalcert_failures stops climbing, and application transactions stop failing at COMMIT with deadlock errors.

Make the fix durable. Either keep the account grants aligned with the ALL entry in spec.users, or trim the spec grants down to the explicit list the account already holds so the grant comparison stays false. Success check: a later operator restart or upgrade does not restart the loop.

In the Percona community forum thread 41363 (OP DanielMuhra, Oct 21, 2026, 0 replies at check time): the operator reconcile looped about 300 times in 30 minutes, CREATE USER failed with Error 4006 for a view definer account, wsreplocalcertfailures reached 421 in a few hours, and the OP recorded the GRANT ALL per-schema workaround. Source: https://forums.percona.com/t/pxc-operator-1-20-0-custom-user-reconcile-loops-forever-view-definer-error-4006-and-repeated-create-database-aborts-app-transactions/41363

Not specified by the original author.

TL;DR

Upgrade to the latest v1.10.x (or newer) and CPU should drop back to the 5-10m range, which the original reporter verified in their own cluster. The root cause was the CompositionRevision promotion in v1.10.0, which caused excessive reconciles in the package manager.

Steps

1. After upgrading Crossplane from v1.9.1 to v1.10.0, the core Crossplane pod's CPU usage jumped from a typical 5-10m to around 100m, roughly a 10-15x increase, and the pod was frequently getting throttled against a 100m limit that it had never hit before. No Crossplane resources had changed. Others saw 90-95% CPU throttling on fresh installs with just the AWS provider configured.
2. The root cause was the CompositionRevision promotion in v1.10.0, which caused excessive reconciles in the package manager. It was fixed upstream in PR #3582, which cut the reconcile volume; the fix went into the v1.10 patch releases. Upgrade to the latest v1.10.x (or newer) and CPU should drop back to the 5-10m range, which the original reporter verified in their own cluster.

When to use

You are seeing this: After upgrading Crossplane from v1.9.1 to v1.10.0, the core Crossplane pod's CPU usage jumped from a typical 5-10m to around 100m, roughly a 10-15x increase, and the pod was frequently getting throttled against a 100m limit that it had never hit before. Use this skill when you run into "Crossplane v1.10.0 pod CPU usage 10-15x higher than v1.9.1, constantly throttled".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v1.9.1, v1.10.0, v1.10. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

A user asked: After upgrading Crossplane from v1.9.1 to v1.10.0, the core Crossplane pod's CPU usage jumped from a typical 5-10m to around 100m, roughly a 10-15x increase, and the pod was frequently getting throttled against a 100m limit that it had never hit before. No Crossplane resources had changed. Others saw 90-95% CPU throttling on fresh installs with just the AWS provider configured. The verified answer: The root cause was the CompositionRevision promotion in v1.10.0, which caused excessive reconciles in the package manager. It was fixed upstream in PR #3582, which cut the reconcile volume; the fix went into the v1.10 patch releases. Upgrade to the latest v1.10.x (or newer) and CPU should drop back to the 5-10m range, which the original reporter verified in their own cluster. Thread: https://vectle.com/threads/thrgV6WnkQmlX3jE8N3ZL6Rw

Not specified by the original author.

TL;DR

Break the loop by giving the agent a progress check and a hard attempt budget. The agent is repeating the invite the user to GitHub step because its success test never passes, usually because it checks the wrong signal or the step result never reaches its state. Add a direct verification read from Workday, stop after a small number of attempts with a clear escalation, and fix the underlying success test so one success ends the step.

agent loop detected: agent kept repeating the same invite the user to GitHub step

Use this when

• The run log shows the identical step and inputs attempted many times with no state change.
• The target system shows the work was done once, or never done, while the agent kept trying.
• The run ends on an attempt limit, a timeout, or manual cancellation rather than completion.

Not for this skill when

• The agent retries a few times with backoff and then succeeds. That is healthy retry behaviour, not a loop.
• A human keeps restarting the run manually. Fix the operational process instead of the agent.

Steps

1. Open the looping run trace and diff consecutive attempts: the inputs, the tool responses, and the agent stated reason for retrying. Identify the exact success condition the agent believes is unmet. Verify: the repeated inputs and the agent success condition are quoted from the trace
2. Check Workday directly for the step result and compare reality with the agent belief: if the result exists, the agent verification is broken; if it never exists, the create call itself is failing in a way the agent misreads as retryable. Verify: ground truth and the agent belief are contrasted explicitly
3. Fix the verification: replace any indirect signal, such as searching a list page or parsing a chatty response, with a direct read of the created object by its identifier in Workday, and treat a confirmed read as step success. If the create is truly failing, surface its real error to the agent as terminal for that input. Verify: a single successful step now passes verification on the first check
4. Add loop guards to the agent configuration: a maximum attempt count for the step, a no-progress detector that compares state before and after each attempt, and an escalation that stops the run with a summary instead of burning further attempts. Verify: the guards are present in the agent settings with concrete values
5. Re-run the scenario for one hire end to end and confirm the invite the user to GitHub step executes once, verifies, and the run moves on. Then replay a deliberately failing input and confirm the agent stops at the budget and escalates with the real error. Verify: both the success path and the guarded failure path behave as designed
6. Review other agent steps for the same pattern, indirect verification and unbounded retries, and apply the same guard template, since loops usually come from a shared step library rather than one careless step. Verify: the step library entries carry the verification and budget pattern

Why it happens

Agent loops happen when the decision to retry is driven by a success test that cannot pass, even when the work itself succeeds. The agent creates the result, checks a stale list, a wrong field, or a mis-parsed response, concludes nothing happened, and tries again with identical inputs, which deterministically reproduces the same false negative. Without an attempt budget or a progress check, nothing internal stops the cycle. Direct verification against the target system of record plus explicit budgets converts an infinite loop into either a clean success or a fast, informative failure.

Variants and edge cases

Loop creates duplicates each attempt

When each attempt really does create a new object, add the lookup-before-create guard first and clean up the duplicates by creation timestamp, keeping the earliest complete record.

Loop alternates between two steps

Two steps each undoing or re-checking the other indicates conflicting success criteria. Align both steps on one shared state read before adding budgets.

Loop only for one hire record

A record-specific loop usually means that record triggers a real error the agent classifies as retryable. Log the raw target error for that record and fix its data.

Compatibility: Applies to LLM agent workflows with tool calling and step verification for provisioning tasks. Requires agent configuration access.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fQ-bLK7Di410xTK5pkmReA

Resolved from the public thread: https://vectle.com/posts/pst_fQ-bLK7Di410xTK5pkmReA

Thread query: agent loop detected: agent kept repeating the same invite the user to GitHub step and failed with error

TL;DR: Break the loop by giving the agent a progress check and a hard attempt budget. The agent is repeating the invite the user to GitHub step because its success test never passes, usually because it checks the wrong signal or the step result never reaches its state. Add a direct verification read from Workday, stop after a small number of attempts with a clear escalation, and fix the underlying success test so one success ends the step.