agent wrote health scores to the wrong Salesforce org: sandbox credentials used in the production run

TL;DR

Writing perfectly good scores to the wrong org is worse than a crash, because production stays stale while a sandbox fills with convincing data and every log says success. Sandbox and production Salesforce orgs look identical to an API client, the only difference is the login host and the org identity behind the credentials the agent loaded. Separate the credential sets physically, have the agent verify the org identity before writing a single record, and audit both orgs so the mistake is fully understood and cannot repeat quietly.

agent wrote health scores to the wrong Salesforce org: sandbox credentials used in the production run

Use this when

• Health scores updated in a sandbox org while production scores went stale, or production updates simply never appeared
• The run logs show successful writes with record counts that reconcile somewhere, just not in production
• A sandbox refresh, a new environment, or a credential rotation happened shortly before the bad run

Not for this skill when

• Writes failed with authentication errors, meaning no org received them, a simpler credential expiry
• The scores reached production but the wrong accounts, an identity mapping issue rather than an org issue

Steps

1. Establish which org received the writes. Query the org identity through the API using the exact credentials and configuration the run used, and compare the returned org ID and org type with productions known values, then find the written scores in that org by their timestamps and batch markers to confirm the destination beyond argument. Verify: the receiving org is proven by its org ID and the located score records
2. Recover production honestly. Do not simply rerun and hope, first check whether the scores are still valid, health scores age fast, so recompute them fresh for production where inputs moved, and only replay the original values for accounts whose inputs are unchanged. Record the stale period in production so CSMs know which days of score history are a gap rather than a trend. Verify: production receives current scores and the gap period is documented per account
3. Decide the sandbox cleanup with the org owners. The stray scores in the sandbox may pollute other teams testing, so agree whether to delete them or leave them marked, and check nothing else consumed them, such as a sandbox based report or a training environment demo that now shows real customer health information it should not hold. Verify: the sandbox data is handled per the owners decision and any consumers are identified
4. Separate the environments physically. Store sandbox and production credentials in distinct named entries with the org ID recorded beside each, remove any shared fallback in the agent configuration that silently substitutes one set for another, and ensure production credentials are only injectable by the production schedule or deployment, never present in a developers test environment. Verify: the credential stores are separated and the configuration contains no cross environment fallback
5. Add a pre-write org verification gate. Before the first write of every run, the agent queries the org identity and compares it with the expected production org ID held in configuration, aborting the run and alerting if they differ, and it logs the verified org ID with the run so every score batch is traceable to the org it landed in. Verify: a test run pointed at the sandbox aborts before writing and alerts the owner
6. Sweep for sibling mistakes. Check every other agent and integration that uses Salesforce credentials for the same shared store or fallback pattern, and add org identity to their run logs too, because credential stores are usually shared and the same trap is armed in more than one place. Verify: the sweep results for sibling integrations are recorded with fixes scheduled

Why it happens

Salesforce orgs are intentionally identical in shape, and a sandbox refreshed from production even contains the same record IDs, so an agent holding sandbox credentials writes successfully, updates plausible records, and gets plausible responses. Every technical signal says success while the business system rots. The only reliable discriminator is the org identity itself, which must be checked by the machine at run time, because humans cannot see the difference in a log line that says the update completed.

Compatibility: Salesforce production and sandbox orgs, API credentials and connected apps, and agent configuration management, in current Salesforce versions.

Variants and edge cases

agent wrote to Salesforce sandbox instead of production

['The direct phrasing. Org identity verification before writing is the control, credential naming alone is a hope, not a check.']

health scores missing production sandbox got them

['The incident variant. Recompute rather than replay when scores are stale, and document the production gap so trend reporting stays honest.']

wrong org credentials agent Salesforce safety

['The design variant. Distinct stores, no fallbacks, and an expected org ID in configuration are the three layers that together make the mistake structurally impossible.']

Edge cases to work through before you close the incident:

• A sandbox refreshed after the bad run may contain the stray writes fossilised inside it. Note the refresh timeline when you audit, or you may find the evidence in an unexpected place.
• Record IDs matching between orgs can mislead debugging for hours. Always confirm identity with the org endpoint, not by recognising familiar account IDs.
• Real customer data written to a sandbox can be a compliance issue depending on your sandbox policy and who has access. Involve your security contact if the sandbox is widely shared.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5t9F6M93haTZTdU54WwcA