agent duplicated save-play tasks in Totango: retry after a timeout created two tasks per account

TL;DR

A timeout does not mean the first attempt failed, it means you do not know, and retrying a creation as if it failed is how one save play becomes two tasks. Totango accepted the first creation, the response was lost or slow, the agent retried, and the second creation succeeded on top of the first. Check for an existing task before creating, key retries on a stable play identifier, clean up the duplicates without closing real work, and every retry becomes safe instead of generative.

agent duplicated save-play tasks in Totango: retry after a timeout created two tasks per account

Use this when

• Duplicate save-play tasks appear for the same account and play, created seconds or minutes apart
• The duplicates correlate with slow responses, timeouts, or retried runs in the agent logs
• CSMs are working or closing one copy while the other sits open, splitting the record

Not for this skill when

• Duplicates arise from two different triggers firing the same play, a rule design question rather than a retry problem
• Tasks look duplicated but cover different products or owners and are intentional

Steps

1. Correlate the duplicates with the logs. Take several duplicated accounts, line up the task creation timestamps in Totango against the agents attempt and timeout records, and confirm the pattern, a timed out attempt followed by a successful creation while the first creation also landed, so the fix targets the retry logic and not the play design. Verify: the timeout then duplicate pattern is confirmed on sampled accounts
2. Define what makes a save-play task the same task. Choose the identity the agent will check before creating, normally the account, the play or playbook identifier, and the play instance or trigger date, and confirm Totango fields can carry or expose that identity, using a tag, an attribute, or the task title convention if the API offers no native key. Verify: the task identity definition is written and mappable to Totango fields
3. Implement create with a preceding check. Before creating, the agent queries Totango for an open task matching that identity and skips creation if one exists, recording the existing task ID in its run log. Where the platform supports it, prefer an upsert style call, and where it does not, the check then create pattern with the identity stored on the task is the reliable substitute. Verify: a forced retry in testing produces one task and the run log cites the pre-existing task ID
4. Handle the ambiguous timeout window explicitly. When a creation call times out, the agent must re-query for the task by identity before any retry, and only create if the query proves absence, with a short waiting period for indexing lag if Totango search is eventually consistent, because the entire bug lives in treating unknown as absent. Verify: a simulated timeout followed by the recheck path creates no duplicate in testing
5. Clean up the existing duplicates safely. List all duplicate groups, keep the task with real CSM activity, notes, or the earliest due work as the survivor, close the copies with a reason that names this defect and links the survivor, and notify assignees whose queues change, since closing a task someone is working is its own incident. Verify: duplicate groups are resolved with survivors kept and assignees notified
6. Add a duplicate watchdog. Run a scheduled check grouping open save-play tasks by the identity definition, alert the agent owner when a group exceeds one, and track the count as the agents quality metric, because idempotency bugs creep back whenever retry logic or timeout values are touched. Verify: the watchdog runs and reports a planted duplicate group in a test

Why it happens

Distributed systems cannot distinguish a lost request from a lost response, and a timeout is the second case as often as the first. An agent that retries creations blindly converts that ambiguity into duplicates with total reliability under load, and Totango, like most task systems, happily holds both copies because each creation was individually valid. Idempotency is therefore the agents responsibility, carried by a stable identity it checks before and after every uncertain attempt.

Compatibility: Totango tasks and SuccessPlays, task search and creation APIs, and agent retry logic, in current Totango and agent framework versions.

Variants and edge cases

duplicate Totango tasks agent retry timeout

['The direct phrasing. Unknown is not absent, the recheck before retry is the whole fix in one sentence.']

save play task created twice idempotency

['The design variant. Creation endpoints without native idempotency keys need a stored identity on the created object and a disciplined check, document both.']

agent retry created duplicate tasks cleanup

['The cleanup variant. Survivor selection must respect human work in progress, automating cleanup by earliest timestamp alone can close the task the CSM is actually working.']

Edge cases to work through before you close the incident:

• Indexing lag can make a just created task invisible to search for a short window. Combine the pre-check with a brief settle delay on the retry path rather than abandoning the pattern.
• Identity based on trigger date breaks when a play legitimately re-triggers the next day. Include the play instance or a business window in the identity, chosen with the CS team.
• Retries of updates are usually safe, retries of creations are not. Audit the agents other create calls, tasks, notes, tickets, with the same lens while you are here.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YNuNizhyi5NC6axN2mlbvg