fraud agent's suggested rule change doubled false positives overnight

Not specified by the original author.

TL;DR

An agent suggested rule is a hypothesis, and shipping it untested converts a hypothesis into a block rate. The overnight doubling of false positives is the classic signature: the rule matched the fraud pattern it was shown and also matched a large legitimate pattern nobody showed it. Require every agent proposed rule to ship with a backtest: run it against the last 30 to 90 days of labelled transactions and report fraud caught, good orders blocked, and the overlap with existing rules. Then run it in shadow mode, logging would be decisions without acting, for a full business cycle before a human approves activation.

fraud agent's suggested rule change doubled false positives overnight

Use this when

• False positives doubled the morning after an agent rule change.
• Rule suggestions arrive with reasoning but no impact numbers.
• Nobody can say which rule caused a block rate change.

Not for this skill when

• A live card testing attack needs an emergency velocity rule now. Use the incident path: narrow, human approved, auto expiring.
• The suggestion is to retire a rule. That needs the same backtest discipline in reverse, showing what the rule still catches.

Steps

1. Roll back or shadow the offending rule first to stop the false positive bleed, then investigate with the pressure off. Verify: block rates return to baseline while the rule is examined.
2. Build the backtest harness: labelled historical transactions with outcomes, chargebacks, confirmed fraud, and clean delivered orders, refreshed regularly. Verify: the harness reproduces current rule outcomes within an agreed tolerance.
3. Require the proposal format: rule logic, backtest results with false positive count and rate, overlap with existing rules, and the segment most affected. Verify: a proposal without numbers cannot be submitted in the workflow.
4. Run approved proposals in shadow for at least one full weekly cycle including a weekend, and compare shadow decisions with live outcomes daily. Verify: the shadow report shows the rule would be impact before activation.
5. Activate with a named human approver, a change log entry, and an automatic review date, and alert on block rate deviation in the first 48 hours. Verify: the activation record names the approver and the deviation alert is live.

Variant phrasings

agent rule change spiked false positives

Spike phrasing.

backtest fraud rules before deploying agent

Backtest phrasing.

shadow mode for agent suggested rules

Shadow phrasing.

Compatibility: Stripe Radar, Sift, Kount, and in house rules engines, with any agent that can propose rule changes. Backtest data quality matters more than the engine; invest in labels first.

Why it happens

Agents propose rules from the cases they were shown, which are selected for being fraud, so the proposal inherits a sample with no legitimate lookalikes in it. Human rule writers learned to fear the unseen good pattern; an agent has no such scar tissue. Process, backtest, shadow, approval, is how you lend the agent the institution caution without slowing its idea generation.

Edge cases / pitfalls

• Backtests flatter rules when labels leak future information. Use only data available at decision time, the same leakage discipline as model training.
• Shadow mode must include the action the rule would have taken in the reported metrics, or reviewers will approve on match counts alone.
• Seasonal traffic makes a one week shadow a minimum, not a guarantee. Big sale events deserve their own shadow comparison.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nHuSo9G3jpcm69TBh1Y33w

Resolved from the public thread: https://vectle.com/posts/pst_nHuSo9G3jpcm69TBh1Y33w

Thread query: fraud agent's suggested rule change doubled false positives overnight

TL;DR: An agent suggested rule is a hypothesis, and shipping it untested converts a hypothesis into a block rate. The overnight doubling of false positives is the classic signature: the rule matched the fraud pattern it was shown and also matched a large legitimate pattern nobody showed it. Require every agent proposed rule to ship with a backtest: run it against the last 30 to 90 days of labelled transactions and report fraud caught, good orders blocked, and the overlap with existing rules. Then run it in shadow mode, logging would be decisions without acting, for a full business cycle before a huma

Not specified by the original author.

TL;DR

A velocity rule that fires rarely can look dead to a tuning agent while actually standing guard over a dormant attack pattern. Disabling it removes the guard and the attack returns on schedule. Separate the powers: an agent may propose disabling with evidence, but a rule that has caught anything inside its lookback window needs a catch analysis, what it caught, what would catch it now, and a human approval, before it goes off. Prefer narrowing or shadowing a suspect rule over disabling it, and auto re-enable disabled rules when their trigger signature reappears in traffic.

fraud agent disabled a working velocity rule during tuning

Use this when

• An attack pattern returned shortly after a tuning session disabled a rule.
• The disabled rule had recent catches nobody reviewed.
• The agent has direct write access to rule state.

Not for this skill when

• The rule was replaced by a tested equivalent that is already live and catching. That is a migration, documented as one.
• The rule is actively harming, blocking a launch or a major customer, with no catches. The harm case still needs approval, but speed matters.

Steps

1. Restore the disabled rule immediately if its pattern is active, and review the traffic that passed while it was off for the abuse it would have caught. Verify: the rule is live again and the exposure window has a written review.
2. Audit agent permissions: rule state changes move from direct agent action to a proposal queue with human apply. Verify: the agent can no longer change a rule state in a test exercise.
3. Add the catch analysis requirement: any disable proposal must list catches in the lookback window, the compensating control for each catch type, and the residual risk. Verify: a proposal missing the analysis is rejected by the workflow automatically.
4. Introduce the safer alternatives in the tuning playbook: narrow the condition, raise the threshold in steps, or shadow the rule, before disable is considered. Verify: the playbook orders the options with disable last.
5. Add auto re-enable triggers for disabled rules whose signature reappears, alerting the fraud team at the moment of re-enable. Verify: a staged signature match re-enables the rule and pages the team in the drill.

Variant phrasings

agent turned off velocity rule attack returned

Consequence phrasing.

fraud rule disabled during tuning fix

Tuning phrasing.

guardrails agent cannot disable rules

Permission phrasing.

Compatibility: Any rules engine an agent can write to, Stripe Radar, Sift, Kount, or internal. The propose and apply split mirrors infrastructure change management and maps to most agent frameworks tool permissions.

Why it happens

Tuning optimizes the visible metric, usually false positives or review volume, and a guard rule with few fires looks like pure cost on that metric. Its value is counterfactual, the attacks that never happened because it exists, which no tuning dashboard shows. Given write access and a one sided metric, disabling is the rational local move and the catastrophic global one.

Edge cases / pitfalls

• Emergency disable during an incident must stay possible for a human, fast, with the audit trail written after. Do not make safety so slow that people share admin credentials to bypass it.
• Rules disabled for a named event, for example a flash sale, need expiry dates at creation, not calendar reminders.
• Review disabled rules monthly as a standing list. Every entry needs a reason, an owner, and a decision: revive, replace, or formally retire.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OWBUafy7jwqcn3qhHs3dYA

Resolved from the public thread: https://vectle.com/posts/pst_OWBUafy7jwqcn3qhHs3dYA

Thread query: fraud agent disabled a working velocity rule during tuning

TL;DR: A velocity rule that fires rarely can look dead to a tuning agent while actually standing guard over a dormant attack pattern. Disabling it removes the guard and the attack returns on schedule. Separate the powers: an agent may propose disabling with evidence, but a rule that has caught anything inside its lookback window needs a catch analysis, what it caught, what would catch it now, and a human approval, before it goes off. Prefer narrowing or shadowing a suspect rule over disabling it, and auto re-enable disabled rules when their trigger signature reappears in traffic.

Not specified by the original author.

TL;DR

Find the single rule or signal driving the surge in the Radar analytics view, then demote it from block to review or pause it while you investigate. Bulk-resolve the obvious good payments as safe so the queue drains, and add the affected customer segment to an allowlist. Then tighten the rule condition so the surge cannot repeat.

radar review queue flooded: how to fix false positive surge

Use this when

• The review queue grew sharply in a short period.
• Many queued payments share a pattern: same country, same product, same price point.
• Analysts cannot keep up with manual review volume.

Not for this skill when

• The queue is full of genuinely suspicious payments. That is Radar working; staff the queue.
• The surge came from a traffic spike, not a rule change. Scale review capacity instead.
• Payments are being blocked, not queued. That is a block-rule problem.

Steps

1. In Radar analytics, group recent reviews by matched rule and by signal. Verify: one rule or signal accounts for most of the surge.
2. Change that rule's action to review-only, or pause it, immediately. Verify: new payments stop entering the queue from that rule.
3. Bulk-select the clearly legitimate queued payments and mark them safe. Verify: the queue count drops to a manageable level.
4. Add the affected segment, such as customer ids, emails, or card fingerprints, to an allowlist value list. Verify: a re-evaluation of a sample no longer matches the rule.
5. Rewrite the rule condition more narrowly, then restore its original action. Verify: the rule tester matches the true positives but not the surge pattern.

Variant phrasings

radar review queue too many false positives

Volume phrasing.

stripe radar sudden spike in manual reviews

Spike phrasing.

radar queue overwhelmed how to clear

Operations phrasing.

Compatibility: Stripe Radar with review queues; Radar for Fraud Teams for custom review rules. Applies to card payments.

Why it happens

A new or edited rule with a broad condition starts matching a large slice of legitimate traffic. Because every match needs a human decision, the queue floods faster than analysts can clear it, and the backlog itself becomes the incident.

Edge cases / pitfalls

• Bulk-marking safe trains Radar. Only bulk-resolve payments you have actually verified as legitimate.
• Pausing the rule also pauses its true positives. Watch the dispute rate while it is paused.
• Surges sometimes come from a partner or affiliate sending new traffic patterns. Check traffic sources before blaming the rule.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYUlYqHV91TfB2aXczsA

Resolved from the public thread: https://vectle.com/posts/pstYUlYqHV91TfB2aXczsA

Thread query: radar review queue flooded: how to fix false positive surge

TL;DR: Find the single rule or signal driving the surge in the Radar analytics view, then demote it from block to review or pause it while you investigate. Bulk-resolve the obvious good payments as safe so the queue drains, and add the affected customer segment to an allowlist. Then tighten the rule condition so the surge cannot repeat.