agent's heartbeat to the orchestrator failed and the test was marked failed

TL;DR

A heartbeat proves the network path and the agent process are alive; it does not prove the test failed. When the orchestrator marks tests failed on heartbeat loss alone, network blips manufacture backup failures. Buffer test results locally on the agent, have the orchestrator mark heartbeat-lost tests as unknown rather than failed, and reconcile the true outcome when the agent reconnects.

agent's heartbeat to the orchestrator failed and the test was marked failed

Use this when

• Tests are marked failed during network maintenance while the restores themselves completed.
• Agent logs show a finished test whose result never reached the orchestrator.
• Failure timestamps align with heartbeat gaps, not with restore errors.

Not for this skill when

• The agent process itself crashed, stopping the test. Then failure is honest, though the cause is the agent.
• The orchestrator correctly waits and the report is accurate. No problem to fix.

Steps

1. Review the orchestrator rule that converts heartbeat loss into a test outcome. Verify: the exact mislabelling rule is found
2. Add local result buffering on the agent that persists the outcome before reporting it. Verify: a disconnected agent holds its result durably
3. Change the orchestrator to record unknown or unreported for heartbeat-lost tests and to accept late results. Verify: a simulated disconnect yields unknown, then reconciles to the real outcome
4. Alert separately on agent connectivity loss so the infrastructure issue stays visible. Verify: connectivity alerts fire without touching test outcomes
5. Run a restore test and drop the agent network mid-run. Verify: the final record shows the true test outcome after reconnection

Variant phrasings

heartbeat loss test marked failed unknown state

The state phrasing.

orchestrator agent disconnect false failure reconcile

The reconcile variant.

liveness vs outcome agent reporting buffer

The distinction wording.

Compatibility: Agent fleets reporting to central orchestrators over heartbeat protocols, with local disk result buffering.

Why it happens

Conflating liveness with outcome is a category error: silence from an agent is absence of evidence, and systems that translate silence into failure corrupt their own statistics and send teams chasing backups that actually restored fine. Unknown is an honest state; building reconciliation around it keeps both the infrastructure problem and the test truth visible.

Edge cases / pitfalls

• Compliance reports must never count unknown as passed either. Reconciliation deadlines matter.
• If an agent never returns, its tests stay unknown and should be re-run elsewhere, with the gap documented.
• Heartbeat design should include the current test identifier, so a reconnecting agent and orchestrator agree on what was in flight.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lIMsmc2FCUJJPTztf6mjNQ