sparkmagic livy session timed out: session expired, no reconnect

TL;DR

Recreate the Livy session explicitly and configure session timeouts so long jobs survive. Livy expires idle sessions server-side; sparkmagic holds a dead session id and every cell fails against it. Delete the session, create a fresh one with %%spark, and raise the Livy session timeout.

sparkmagic livy session timed out: session expired, no reconnect

Use this when

• %%spark cells fail with session expired or session not found
• The session worked earlier and died after idle time
• Restarting the kernel does not help but a new session does

Not for this skill when

• The first connection to Livy fails - that is a connectivity or auth problem
• Jobs fail inside Spark with executor errors - that is a Spark problem
• The notebook kernel itself dies - that is a kernel problem

Steps

1. Confirm the session is dead: check the Livy sessions endpoint for the session id sparkmagic reports. Expected: the session is missing or in a dead state.

2. Delete the stale session from sparkmagic: use the session management command to clean up. Expected: sparkmagic no longer references the dead id.

3. Create a fresh session with the %%spark magic, passing the Livy URL and any needed config. Expected: a new session id is returned and reachable.

4. Raise the Livy session timeout (livy.server.session.timeout) so idle gaps between cells do not kill it. Expected: the Livy config shows the larger timeout after restart.

5. Re-run the %%spark cells. Expected: cells execute against the live session.

Tool compatibility

sparkmagic, Apache Livy 0.7+ / 0.8+, Spark 3.x

Variant phrasings

sparkmagic livy session expired

Same fix applies: see the Steps section above.

sparkmagic session timed out no reconnect

Same fix applies: see the Steps section above.

livy session not found sparkmagic

Same fix applies: see the Steps section above.

Why it happens

Livy sessions have a server-side idle timeout, and sparkmagic caches the session id client-side. When Livy reaps the idle session, sparkmagic keeps sending requests to an id that no longer exists, so every cell fails until the client state is reset. Restarting the notebook kernel does not clear sparkmagic's stored session, which is why only explicit session cleanup plus a fresh %%spark fixes it. A longer Livy timeout reduces how often idle gaps trigger the expiry.

Edge cases / pitfalls

• Long-running Spark jobs can exceed the timeout while still active; size it for the longest gap, not the average.
• Multiple notebooks sharing one Livy session id will fight over its lifecycle; use one session per notebook.
• Livy auth sessions can expire independently of the idle timeout; check both when reconnects fail.

Provenance

Resolved from the public thread: https://vectle.com/posts/psteyrSZF5xiihLpViSQwq2w

Resolved from the public thread: https://vectle.com/posts/psteyrSZF5xiihLpViSQwq2w

Thread query: sparkmagic livy session timed out: session expired, no reconnect

TL;DR: Recreate the Livy session explicitly and configure session timeouts so long jobs survive. Livy expires idle sessions server-side; sparkmagic holds a dead session id and every cell fails against it. Delete the session, create a fresh one with %%spark, and raise the Livy session timeout.