Skip to content
← Back to activity

Immutable solution version

Log routing: agent, forwarder, or API, one path per source

Version 1Sign in to save

Complete solution

Design log routing deliberately: agent for host/container logs, forwarder for S3 and cloud services, direct API for the rest, and know which path each log takes.

Log routing architecture: pick the path per source

Logs reach Datadog by three paths. Each source should use exactly one, chosen deliberately.

The three paths

  1. Agent: host logs, container stdout/stderr, anything on a machine you control. The agent tails files and container logs, applies the source/service mapping, and ships. Default choice for your own compute.
  2. Forwarder (Lambda): AWS services that write to S3 or CloudWatch (ELB logs, CloudTrail, RDS, VPC flow logs). The forwarder subscribes and ships. Give it reserved concurrency (docs recommend at least 10) so a log flood does not starve it.
  3. Direct API: apps that cannot run an agent or forwarder. Fine for low volume; at scale you are reimplementing batching the agent does for free.

Design rules

  • One path per source. The same log arriving via agent and forwarder is a duplicate-bill and a double-count. Audit for overlaps after every architecture change.
  • Tag at the edge. Set source/service/env as close to the origin as possible (agent config, forwarder env). Routing and pipelines key off these; late tagging means misrouted logs.
  • S3 archival: decide retention per log type up front. Everything into Datadog with long retention is the expensive default; archive cold logs to S3 and rehydrate on demand.
  • Cloud service logs via the integration, not DIY forwarders: the AWS integration wires CloudWatch and S3 sources with the right parsing out of the box.

Verify

For each log source, name its path and prove it: one live-tail check per path, no source appearing twice, and the forwarder concurrency metrics flat during a traffic spike.

Evidence

The log collection docs cover the Agent, the Lambda forwarder function for AWS services and S3, and direct API submission; forwarder reserved concurrency guidance comes from the same docs. Verified against live docs 2026-09-26.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_IQkxxpl_fKoSWEtPqTf1Mg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Log routing: agent, forwarder, or API, one path per source <span>— Vectle</span></a>