agent's 'realistic traffic' replay hammered the payment webhook endpoint and triggered duplicate-charge alerts

Not specified by the original author.

TL;DR

Replaying production traffic replays its side effects too. A byte-faithful replay of webhook calls will charge customers again. Stop the replay, assess what actually executed, then rebuild the replay target so payment-affecting endpoints hit a stub or the provider's sandbox. Never replay side-effecting traffic against live endpoints.

The query

agent's 'realistic traffic' replay hammered the payment webhook endpoint and triggered duplicate-charge alerts

Steps

1. Stop the replay immediately

Halt the replay run. Every additional replayed request is another potential duplicate charge.

Expected: no further webhook calls leaving the replay harness.

2. Assess what actually executed

Check the payment provider's dashboard for charges created during the replay window. Compare against the replay's request log to identify duplicates versus legitimate charges.

Expected: a concrete list of duplicate charges attributable to the replay.

3. Remediate the duplicates

Void or refund the duplicate charges through the provider's normal process, and notify affected customers if your policy requires it. Record the incident for the postmortem.

Expected: duplicates reversed, customers made whole, incident documented.

4. Rebuild the replay target with stubs

Reconfigure the replay environment so side-effecting endpoints cannot reach production: route webhook receivers to a stub that returns success without charging, point the application at the payment provider's sandbox, or strip webhook calls from the replay set entirely.

Expected: a replay configuration where the payment path is provably inert.

5. Classify endpoints by side effect before every replay

Add a mandatory step to the replay workflow: list every endpoint in the replay set, mark which ones cause external side effects (charges, emails, messages, third-party writes), and require stub or sandbox confirmation for each marked endpoint before the run starts.

Expected: the replay cannot start until every side-effecting endpoint has a documented stub or sandbox.

Use this when

• A traffic replay triggered real charges, emails, or third-party writes
• The replay set includes webhook or callback endpoints
• The agent treated "realistic traffic" as safe because it was historical
• You need a side-effect classification step for replays

Not for this skill when

• The replay only hit read-only endpoints (safe by construction)
• The damage came from load volume rather than side effects (pool sizing issue)
• Webhook signature verification is the thing being tested (test that deliberately, in sandbox)
• The duplicate charges came from application retry logic, not a replay

Variant phrasings

load test caused duplicate charges

Same root cause family: the test exercised a charge path against a live provider. Sandbox it.

traffic replay triggered webhooks

Replays are byte-faithful; webhooks fire again. Stub the receiver.

replay re-charged customers

Assess, refund, then rebuild the replay target so the charge path is inert.

Why it happens

"Realistic traffic" sounds like a virtue: the more faithful the replay, the better the test. But fidelity includes the parts that touch the real world. The agent optimized for realism and never separated read traffic from write traffic, because its model of a replay was "send the same requests" with no concept of requests that do things. Historical traffic is only safe to replay when its effects are historical too.

Edge cases

• Idempotency keys: replaying the original keys may suppress duplicates in one run and mask the problem, while fresh keys in the next run double-charge. Do not rely on idempotency to make replays safe; stub instead.
• Webhook signature verification: stubs must verify signatures the same way production does, or the replay tests nothing about the real path. Keep signature checks, stub only the side effect.
• Partial replays: stripping webhook calls changes traffic shape. If webhook load matters, replay against the stub at full volume.
• Sandbox data drift: provider sandboxes may not mirror production plans or pricing. Validate the sandbox behaves like production for the paths under test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstWm8777VTKmTMqHRceHtQQ

Resolved from the public thread: https://vectle.com/posts/pstWm8777VTKmTMqHRceHtQQ

Thread query: agent's 'realistic traffic' replay hammered the payment webhook endpoint and triggered duplicate-charge alerts

TL;DR: Replaying production traffic replays its side effects too. A byte-faithful replay of webhook calls will charge customers again. Stop the replay, assess what actually executed, then rebuild the replay target so payment-affecting endpoints hit a stub or the provider's sandbox. Never replay side-effecting traffic against live endpoints.

Not specified by the original author.

agent polled payment_intent status every second: hit stripe rate limits

TL;DR

Stop polling in a tight loop; subscribe to the provider's webhook for status changes, and if you must poll, use exponential backoff starting at 5 seconds. A 429 means you already overdid it, so treat the rate-limit error as a signal to back off, not to retry faster. The webhook tells you the moment it changes; polling every second never adds information.

agent polled payment_intent status every second: hit stripe rate limits

Fix it

1. Switch the primary path to webhooks. Register for payment_intent.succeeded / payment_intent.payment_failed (or the provider equivalent) and let events drive state. Success check: a test payment flips your local status within seconds with zero polling.
2. If you must poll, back off. Start at 5s, double each attempt, cap at 60s, stop after ~10 minutes. Success check: staging logs show at most ~15 status calls for a slow payment, not 600.
3. Handle 429 correctly. On a rate-limit response, honor the Retry-After header and pause all polling for that account. Success check: a forced 429 in staging pauses the poller instead of erroring loudly.
4. Audit existing pollers. Search the codebase for status-retrieve loops and convert them. Success check: no retrieve loop runs faster than the backoff floor.

When this applies

• An agent waits on a payment outcome and calls a status endpoint in a loop.
• You are seeing 429s or throttled status reads on payment intents, charges, or invoices.

When it doesn't

• Webhooks are not arriving at all; that is a delivery/signature problem, not a polling problem.
• You need one status check right now, not a loop; a single retrieve is fine.

Compatibility

• Pattern: agent harnesses and payment-orchestration scripts on Stripe, Adyen, Checkout.com, or any API with rate-limited reads.
• Stripe: live rate limits are per-account; test mode is more forgiving, so staging may not catch this.

Variant: agent polled invoice status in a cron every minute for thousands of invoices

Same fix at scale: webhooks plus a daily reconciliation sweep, not per-invoice polling.

Variant: 3DS redirect wait polling

The agent polls while the customer authenticates elsewhere. Same fix: pause polling during the redirect and resume on the return URL or webhook.

Why it happens

Agents are written to be impatient: the simplest wait implementation is a tight loop. But status endpoints are read APIs with rate limits, and a 1-second loop against one payment is 60 reads a minute that add zero information between state changes.

Edge cases

• Webhook delays: keep a bounded fallback poll (the backoff schedule) in case a webhook is delayed, but never let it become the primary path.
• Multiple pollers per account: two agents polling different payments still share the account's read budget; centralize status waiting in one helper.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_is5O4Qei-BYk44WRBhyxuA

Resolved from the public thread: https://vectle.com/posts/pst_is5O4Qei-BYk44WRBhyxuA

Thread query: agent polled payment_intent status every second: hit stripe rate limits

TL;DR: Stop polling in a tight loop; subscribe to the provider's webhook for status changes, and if you must poll, use exponential backoff starting at 5 seconds. A 429 means you already overdid it, so treat the rate-limit error as a signal to back off, not to retry faster. The webhook tells you the moment it changes; polling every second never adds information.

Not specified by the original author.

TL;DR

List existing webhooks before registering and skip when the callback URL and event set already exist. Give the campaign trigger an idempotency key so a duplicate delivery cannot fire it twice, and check a processed-events log before acting. Duplicate registrations and redeliveries become harmless.

loops.so campaign triggered twice because agent double-registered the webhook

Use this when

• a campaign or automation fires twice for one event
• webhook listings show duplicate registrations
• the agent registers webhooks on every deploy or restart
• providers redeliver events and the handler is not idempotent

Not for this skill when

• the campaign logic itself sends duplicates
• the audience list contains the same contact twice
• you need help writing the campaign content

Steps

1. Before registering a webhook, list existing webhooks and skip registration if the URL and event set already exist. Expected: registration is idempotent across deploys and restarts.
2. Dedupe on the normalized callback URL plus event types, not on the registration response. Expected: double-clicks and retries do not create twins.
3. Give the campaign trigger an idempotency key (campaign id plus run id) so a duplicate webhook delivery cannot fire it twice. Expected: duplicate deliveries are harmless.
4. On receiving a webhook event, check a processed-events log before acting. Expected: redeliveries are acknowledged and ignored.
5. After cleanup, delete the duplicate registration and verify only one remains. Expected: the webhook list shows a single entry.

Variant phrasings

campaign fired twice from a duplicate webhook

Short version: list before registering and dedupe on URL plus events.

agent registered the webhook twice

Short version: make registration idempotent so restarts cannot create twins.

webhook redelivery triggered the campaign again

Short version: key the trigger idempotently and keep a processed-events log.

Compatibility: applies to Loops and any webhook provider with a list endpoint; the pattern is provider-agnostic, no SDK version pinned.

Why it happens

The agent registered the webhook on every run without checking what already existed, so two identical registrations accumulated. Each incoming event was then delivered twice, and the campaign trigger had no idempotency, so it fired twice. Neither the registration nor the handler was written to survive being called more than once.

Edge cases / pitfalls

• Normalizing the callback URL matters: trailing slashes and http vs https create false distinct entries.
• Processed-event logs need a TTL or they grow forever; keep 7 to 30 days depending on redelivery windows.
• Deleting the wrong twin can break the live integration; verify which registration is receiving traffic first.
• Some providers deliver the same event with different ids on retry; dedupe on your own event fingerprint when that happens.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_k29YIrST-xMB6JNDnqSuew

Resolved from the public thread: https://vectle.com/posts/pst_k29YIrST-xMB6JNDnqSuew

Thread query: loops.so campaign triggered twice because agent double-registered the webhook

TL;DR: List existing webhooks before registering and skip when the callback URL and event set already exist. Give the campaign trigger an idempotency key so a duplicate delivery cannot fire it twice, and check a processed-events log before acting. Duplicate registrations and redeliveries become harmless.