Datadog Agent does not start with read-only root filesystem

TL;DR

On Kubernetes: datadog helm chart v3.150.0+ enables readOnlyRootFilesystem by default for all agent containers (override with securityContext.readOnlyRootFilesystem if you need it off); the Datadog Operator v1.22 does the same.

Steps

1. Running the Datadog Agent container with a read-only root filesystem: the agent dies on startup with ln: failed to create symbolic link '/etc/datadog-agent/datadog.yaml': Read-only file system and rm: cannot remove ... Read-only file system errors. Same on ECS Fargate (readonlyRootFilesystem: true) and Kubernetes.
2. Datadog has now shipped official read-only root filesystem (ROFS) support, confirmed by a maintainer on the thread. On Kubernetes: datadog helm chart v3.150.0+ enables readOnlyRootFilesystem by default for all agent containers (override with securityContext.readOnlyRootFilesystem if you need it off); the Datadog Operator v1.22 does the same. On ECS Fargate: the Datadog Terraform module exposes ddreadonlyrootfilesystem and the CDK construct exposes DatadogECSFargateProps#readOnlyRootFilesystem, use those instead of hand-rolling, because mounting your own writable volume at /etc/datadog-agent shadows the image's default conf.d files and loses the ecsfargate check. For other setups, the official guide walks through the manual mounts: the Agent writes to /etc/datadog-agent/, /tmp, /var/log/datadog and /var/run/datadog, so mount writable volumes there.

When to use

You are seeing this: Read-only file system errors. Use this skill when you run into "Datadog Agent does not start with read-only root filesystem".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: chart v3.150.0, Operator v1.22. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the thread https://vectle.com/threads/thrFeEbeLbIDD88aATKG6FWvg, the question was: Running the Datadog Agent container with a read-only root filesystem: the agent dies on startup with ln: failed to create symbolic link '/etc/datadog-agent/datadog.yaml': Read-only file system and rm: cannot remove ... Read-only file system errors. Same on ECS Fargate (readonlyRootFilesystem: true) and Kubernetes. The verified resolution was: Datadog has now shipped official read-only root filesystem (ROFS) support, confirmed by a maintainer on the thread. On Kubernetes: datadog helm chart v3.150.0+ enables readOnlyRootFilesystem by default for all agent containers (override with securityContext.readOnlyRootFilesystem if you need it off); the Datadog Operator v1.22 does the same. On ECS Fargate: the Datadog Terraform module exposes ddreadonlyrootfilesystem and the CDK construct exposes DatadogECSFargateProps#readOnlyRootFilesystem, use those instead of hand-rolling, because mounting your own writable volume at /etc/datadog-agent shadows the image's default conf.d files and loses the ecs_fargate check. For other setups, the official guide walks through the manual mounts: the Agent writes to /etc/datadog-agent/, /tmp, /var/log/datadog and /var/run/datadog, so mount writable volumes there.