pipeline agent deadlocked waiting on its own sensor: timeout and fail-open
Unsticks a pipeline agent deadlocked on a sensor for data it produces itself. Use when a sensor waits on a partition or file the same pipeline creates, when sensors have no timeout, and when you need an explicit fail-open vs fail-closed policy. Not for sensors waiting on genuinely external upstream data.
TL;DR
A sensor watching for data the same pipeline produces is a dependency cycle: it can never succeed. Confirm the self-reference, give every sensor a timeout, encode fail-open vs fail-closed explicitly, and turn internal handoffs into real task dependencies.
Symptom
The pipeline sits in sensor-poke forever. Logs show the sensor waiting on a partition, file, or key that the pipeline's own earlier (or later) task was supposed to produce, so the data can never exist at poke time.
Steps
- Confirm the deadlock: check whether the sensor's target is produced by the same DAG and run, and whether task ordering means it can never exist yet.
Expected output: you can point at the cycle (sensor in task B waits on output of task C, which runs after B).
- Give every sensor a timeout. A sensor without one holds a worker slot forever:
sensor = FileSensor(task_id="wait_for_input", filepath="[path]", timeout=3600, poke_interval=60)Expected output: the wait ends after the timeout instead of hanging the scheduler indefinitely.
- Decide fail-open vs fail-closed per pipeline and encode it. Fail-open proceeds treating the data as absent; fail-closed aborts the run. Do not leave the choice to the agent's judgment each time.
Expected output: a timeout produces a deterministic, reviewed behavior instead of an agent improvising.
- Restructure so sensors only watch truly external inputs. Internal handoffs become explicit task dependencies (task B depends on task C) instead of sensor polling.
Expected output: the cycle disappears because the scheduler orders the tasks itself.
When to use
- A sensor waits on data the same pipeline produces
- Sensors were added without timeouts
- You need a standing policy for what a sensor timeout means
When not to use
- The sensor watches genuinely external upstream data (thats what sensors are for; check the upstream instead)
- Tasks fail for other reasons while the sensor waits fine (fix the task)
- You want event-driven triggering (use dataset-aware scheduling, not polling sensors)
Variant phrasings
sensor times out every run but the data eventually appears
The timeout is shorter than the upstream delay, or the sensor watches the wrong path. Fix the path or the timeout, not the sensor's existence.
two pipelines sensor-wait on each other
Same cycle across DAG boundaries. Break it with an explicit dependency or a single orchestrating pipeline.
Why it happens
Agents add sensors defensively ("wait until the data is there") without checking who produces the data. A sensor on self-produced data is a dependency the scheduler cannot resolve, so it polls forever.
Edge cases
- Poke intervals too tight hammer the metastore or filesystem; 60 seconds is a sane floor for most.
- Reschedule mode frees the worker slot while waiting; poke mode holds it. Use reschedule for long waits.
- Sensor timeouts need alerting; a silently skipped sensor looks like a successful run.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_e8WbuUwT6uMSTC9Sr0QCgA