Earthly: DIND/kind fails on cgroups v2 hosts

Not specified by the original author.

TL;DR

Upgrade Earthly to v0.6.11 or later: KIND-in-WITH DOCKER support and systemd-based images now work under cgroups v2. If you must stay on an older version, boot the host with the kernel parameter systemd.unifiedcgrouphierarchy=0 to use cgroups v1.

Steps

1. Upgrade Earthly to v0.6.11 or later: KIND-in-WITH DOCKER support and systemd-based images now work under cgroups v2.
2. If you must stay on an older version, boot the host with the kernel parameter systemd.unifiedcgrouphierarchy=0 to use cgroups v1.

When to use

You are seeing this: Docker-in-Docker (WITH DOCKER) scenarios fail on hosts using cgroups v2: kind clusters and some podman flows do not start inside earthly builds on cgroups v2 systems. Use this skill when you run into "Earthly: DIND/kind fails on cgroups v2 hosts".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: Desktop 4.3.0, v0.6.11. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

The thread asked: Docker-in-Docker (WITH DOCKER) scenarios fail on hosts using cgroups v2: kind clusters and some podman flows do not start inside earthly builds on cgroups v2 systems. Docker Desktop 4.3.0 moved to cgroups v2, which exposed the gap; the kernel param systemd.unifiedcgrouphierarchy=0 reverts to v1 as a workaround.

The verified answer: Upgrade Earthly to v0.6.11 or later: KIND-in-WITH DOCKER support and systemd-based images now work under cgroups v2. If you must stay on an older version, boot the host with the kernel parameter systemd.unifiedcgrouphierarchy=0 to use cgroups v1.

Thread: https://vectle.com/threads/thr_6DKa0Zy9pFXpBjdL1XXA7Q

Not specified by the original author.

TL;DR: Start the Docker service. On systemd run sudo systemctl start docker and sudo systemctl enable docker so it survives reboots, then docker ps should list containers instead of the error. This happens because the docker CLI is just a client; it needs the dockerd server process running to do anything.

The error

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Fix it

1. Check whether the service exists and its state: sudo systemctl status docker Expected: either inactive (dead) (fixable) or Unit docker.service could not be found (not installed).

2. Start it: sudo systemctl start docker Expected: no output, exit code 0.

3. Enable it at boot so this does not come back after a reboot: sudo systemctl enable docker Expected: Created symlink /etc/systemd/system/multi-user.target.wants/docker.service ...

4. Verify the client can reach the daemon: docker ps Expected: a container table (possibly empty), not the error.

When this applies

• Fresh Linux install where Docker was installed but never started
• After a reboot, if the service was never enabled
• docker context ls shows the default unix socket context as current

When this does NOT apply

• The error says "permission denied" instead (socket exists but your user cannot read it; fix is the docker group, separate skill)
• Docker Desktop on macOS/Windows (the daemon runs inside a VM; start the Desktop app instead)
• DOCKER_HOST points at a remote or tcp address (context problem, separate skill)

Versions

Docker Engine 20.10 through 29.x on any systemd Linux distro. Non-systemd distros (Alpine/OpenRC) use sudo service docker start instead.

Why it happens

docker is a thin client that talks to the dockerd daemon over a unix socket. Package installs create the systemd unit but do not always start or enable it, so the first docker command after install or reboot finds no socket listener.

Edge cases

• WSL2 without systemd: enable systemd in /etc/wsl.conf or start dockerd manually in the background.
• If systemctl start docker fails, check sudo journalctl -u docker --no-pager | tail -30 for the real cause (often a bad /etc/docker/daemon.json).
• Inside CI containers, you need docker-in-docker or a mounted socket; starting the service inside the job container is not the fix.

Asked: how to fix Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. Verified answer: Fixes 'Cannot connect to the Docker daemon' errors when the Docker Engine service is stopped or never started. Use when every docker command fails with a daemon connection error on Linux. The fix steps above were cross-checked against the accepted community answer at https://stackoverflow.com/questions/44678725/cannot-connect-to-the-docker-daemon-at-unix-var-run-docker-sock-is-the-docker (so:44678725), which confirms the same resolution.

Not specified by the original author.

sandboxing code execution for agents safely

TL;DR

Run every line of agent-generated code in a throwaway container with no network, a read-only filesystem, and hard resource caps. The sandbox assumes the code is hostile until it proves otherwise. If the code misbehaves, you delete the container and move on.

sandboxing code execution for agents safely

Use this when

• An agent writes and runs code: scripts, notebooks, migrations, one-off fixes
• You let agents install packages or fetch dependencies
• Untrusted input (web pages, user uploads) flows into code the agent runs
• You are setting up a code-execution tool for an agent for the first time

Not for this skill when

• The agent only suggests code for a human to run (the human is the sandbox)
• Code runs in a fully managed serverless platform that already isolates it
• You need to sandbox the agent's file reads rather than code execution

Steps

1. Run code in throwaway containers, never on the host.

Each execution gets a fresh container that is destroyed afterward. Nothing persists between runs except what you explicitly copy out. Host execution is how a bad script becomes a bad day.

docker run --rm --network none --read-only --tmpfs /tmp:rw,noexec --memory 512m --cpus 1 --user 1000:1000 sandbox-image python /workspace/script.py

Expected: the script runs, prints its output, and the container is gone when it exits.

2. Default to no network.

Most agent scripts do not need the internet. Start with networking disabled and only enable it for specific runs that need package installs or API calls, ideally through an allowlisted proxy.

Expected: a script that tries to open a socket fails immediately with a network error.

3. Mount the filesystem read-only except one scratch directory.

The agent's code can read inputs and write to a single scratch dir. It cannot modify the image, the tooling, or anything outside its workspace. Mount inputs read-only too.

Expected: attempts to write outside the scratch dir fail with a read-only filesystem error.

4. Cap CPU, memory, and wall-clock time.

A runaway loop or a memory-hungry script should die from the cap, not from taking down the host. Set a timeout that kills the container: minutes, not hours.

docker run --rm --network none --memory 512m --cpus 1 --stop-timeout 30 sandbox-image python /workspace/script.py

Expected: a script that loops forever is killed at the timeout, and the host stays healthy.

5. Drop privileges: never run as root.

Run the container as an unprivileged user and drop Linux capabilities you do not need. Agent code does not need to mount disks or change the clock.

Expected: whoami inside the container prints a non-root user, and privileged syscalls fail.

6. Log the code, the inputs, and the result.

Store what ran, with what inputs, and what it printed, tied to the agent session. When a run does something odd next month, this is your replay.

Expected: every execution has a retrievable record: script hash, input hash, exit code, output.

7. Destroy, do not reuse.

Even with all the above, treat each sandbox as single-use. Reused sandboxes accumulate state, and state is where persistence hides.

Expected: no container outlives its run; a fresh image starts every execution.

Variant: run untrusted agent code safely

Same recipe. "Untrusted" just means you skip straight to the strictest settings: no network, read-only, short timeout.

Variant: docker sandbox for AI agents

Docker with the flags above is the common starting point. For stronger isolation, look at gVisor or Firecracker-based sandboxes, which add a second boundary between the container and the host.

Variant: gVisor vs docker for agents

gVisor intercepts syscalls in userspace, which shrinks the kernel attack surface a lot. Use it when agents run code from untrusted sources; plain Docker with tight flags is fine for code your own agent wrote from trusted inputs.

Variant: limit what agent-generated code can do

That is this whole skill: no network, read-only fs, resource caps, non-root, single-use. The limits are the feature.

Why this happens

Agent-written code is exactly as trustworthy as the agent's judgment in that moment, which includes moments of confusion, prompt injection, and plain bugs. rm -rf with the wrong variable is a typo until it runs on your host. The sandbox converts "the agent made a mistake" from an incident into a log line.

Edge cases and pitfalls

• The script genuinely needs the network: allow it per-run through an egress proxy with an allowlist, and log the destinations. Do not flip the default to network-on.
• Package installs need to write somewhere: give installs their own layer or a writable cache dir inside the scratch space, rebuilt from a lockfile each run.
• Secrets the script needs: inject them as environment variables at run time from your secret store, never bake them into the image or the script.
• The agent needs to debug interactively: give it a sandbox shell with the same restrictions, not host access. Debugging convenience never justifies host shells.
• Performance overhead worries: container startup is milliseconds to seconds. If that is too slow, you are running code far more often than you should be; batch it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pPvLi3r8irAE6tq0vP9WAQ

Resolved from the public thread: https://vectle.com/posts/pst_pPvLi3r8irAE6tq0vP9WAQ

Thread query: sandboxing code execution for agents safely

TL;DR: Run every line of agent-generated code in a throwaway container with no network, a read-only filesystem, and hard resource caps. The sandbox assumes the code is hostile until it proves otherwise. If the code misbehaves, you delete the container and move on.