## TL;DR
The runner cannot create the cache volume because the user it runs as lacks permission on the Docker socket or the volume path. Fix the socket access for the runner user, and make sure user namespace remapping is not silently changing volume ownership.

```text
ERROR: Preparation failed: adding cache volume: permission denied
```

## Steps

1. Confirm the runner user can talk to the Docker daemon:
```bash
sudo -u [RUNNER_USER] docker ps
```
Expected: a container list. A permission denied here reproduces the job failure exactly at the daemon level.

2. Check socket ownership and the runner user's groups:
```bash
ls -l /var/run/docker.sock
id [RUNNER_USER]
```
Expected: the socket is owned by root and the docker group, and the runner user is a member of the docker group. If not, add the user and restart the runner service so the group takes effect.

3. Add the runner user to the docker group if it is missing:
```bash
sudo usermod -aG docker [RUNNER_USER]
sudo systemctl restart gitlab-runner
```
Expected: `id [RUNNER_USER]` now lists the docker group and the permission denied is gone on the next job.

4. Check for user namespace remapping changing volume ownership:
```bash
docker info --format '{{.SecurityOptions}}'
ls -ld [CACHE_VOLUME_PATH]
```
Expected: if userns remapping is active, the volume path must be writable by the remapped UID, not by the runner user directly. Adjust ownership to the subordinate UID range.

5. Rerun the job and confirm preparation succeeds:
```bash
# retry the job from the GitLab UI
```
Expected: the preparation stage creates the cache volume and the job proceeds to the script.

## When to use
- Jobs fail in the preparation stage with the cache volume permission denied
- The failure started after moving the runner to a new host or hardening the Docker install
- Shell executor jobs work but Docker executor jobs fail the same way

## When not to use
- The failure is during the script phase; that is a job problem, not volume setup
- The error names a missing image or a pull failure; that is registry auth or naming
- Cache restore fails later with corrupt archive errors; that is cache content, not volume permissions

## Why it happens
The Docker executor creates a cache volume before the job container starts, and that creation runs as the runner's OS user against the Docker socket. Hardened hosts often restrict socket access to root or the docker group, and user namespace remapping silently rewrites the UIDs that own created volumes, so the volume ends up owned by a UID the job cannot write to.

## Edge cases
- SELinux in enforcing mode can deny the volume mount even with correct ownership; check the audit log for denials before loosening anything
- Rootless Docker moves the socket to the user's runtime dir; the runner config must point at that socket path
- The `privileged` flag does not fix socket permission problems; it only affects what the job container can do once it exists
- Group membership changes need a full service restart, not just a job retry, because the runner process keeps its old groups

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yAaROjgHWn7Tar1G2PHh-Q