Immutable solution version
Log routing: agent, forwarder, or API, one path per source
Complete solution
Design log routing deliberately: agent for host/container logs, forwarder for S3 and cloud services, direct API for the rest, and know which path each log takes.
Log routing architecture: pick the path per source
Logs reach Datadog by three paths. Each source should use exactly one, chosen deliberately.
The three paths
- Agent: host logs, container stdout/stderr, anything on a machine you control. The agent tails files and container logs, applies the source/service mapping, and ships. Default choice for your own compute.
- Forwarder (Lambda): AWS services that write to S3 or CloudWatch (ELB logs, CloudTrail, RDS, VPC flow logs). The forwarder subscribes and ships. Give it reserved concurrency (docs recommend at least 10) so a log flood does not starve it.
- Direct API: apps that cannot run an agent or forwarder. Fine for low volume; at scale you are reimplementing batching the agent does for free.
Design rules
- One path per source. The same log arriving via agent and forwarder is a duplicate-bill and a double-count. Audit for overlaps after every architecture change.
- Tag at the edge. Set source/service/env as close to the origin as possible (agent config, forwarder env). Routing and pipelines key off these; late tagging means misrouted logs.
- S3 archival: decide retention per log type up front. Everything into Datadog with long retention is the expensive default; archive cold logs to S3 and rehydrate on demand.
- Cloud service logs via the integration, not DIY forwarders: the AWS integration wires CloudWatch and S3 sources with the right parsing out of the box.
Verify
For each log source, name its path and prove it: one live-tail check per path, no source appearing twice, and the forwarder concurrency metrics flat during a traffic spike.
Evidence
The log collection docs cover the Agent, the Lambda forwarder function for AWS services and S3, and direct API submission; forwarder reserved concurrency guidance comes from the same docs. Verified against live docs 2026-09-26.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_IQkxxpl_fKoSWEtPqTf1Mg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Log routing: agent, forwarder, or API, one path per source <span>— Vectle</span></a>