buildkit scheduler error: return leaving incoming open

TL;DR

Upgrade (Satellites auto-update overnight if enabled). [alexcb (contributor)]: The intermittent hang on COPY plus wait-end was a buildkit race condition, fixed upstream and released in Earthly v0.7.21.

Steps

1. [alexcb (contributor)]: The intermittent hang on COPY plus wait-end was a buildkit race condition, fixed upstream and released in Earthly v0.7.21. Upgrade (Satellites auto-update overnight if enabled). If a buildkitd gets stuck in the bad state, restarting it with docker stop earthly-buildkitd recovers.

When to use

You are seeing this: We've been seeing the following error mostly in CI, however, we've also seen it on a developer's machine as well. Use this skill when you run into "buildkit scheduler error: return leaving incoming open".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: Earthly v0.7.21. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In gh:earthly/earthly#2957 (https://github.com/earthly/earthly/issues/2957), the reporter ran into this: We've been seeing the following error mostly in CI, however, we've also seen it on a developer's machine as well. The issue reproduced on the developer machine several times and only went away after changing the base image of the target (seems sort of random, but perhaps the team can make more sense of it). On CI, the job often fails multiple times in a row if you just re-run it as is. The verified resolution: [alexcb (contributor)]: The intermittent hang on COPY plus wait-end was a buildkit race condition, fixed upstream and released in Earthly v0.7.21. Upgrade (Satellites auto-update overnight if enabled). If a buildkitd gets stuck in the bad state, restarting it with docker stop earthly-buildkitd recovers.