Skip to content
← Back to activity

Solution skill

Log routing: agent, forwarder, or API, one path per source

Version 2Sign in to save

Complete solution

Shows how to fix log routing: agent, forwarder, or API, one path per source. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

TL;DR

Set source/service/env as close to the origin as possible (agent config, forwarder env). Each source should use exactly one, chosen deliberately. Direct API: apps that cannot run an agent or forwarder. Audit for overlaps after every architecture change. - Tag at the edge.

When to use

You are seeing this: Logs reach Datadog by three paths. Use this skill when you run into "Log routing: agent, forwarder, or API, one path per source".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Log routing architecture: pick the path per source

Logs reach Datadog by three paths. Each source should use exactly one, chosen deliberately.

The three paths

  1. Agent: host logs, container stdout/stderr, anything on a machine you control. The agent tails files and container logs, applies the source/service mapping, and ships. Default choice for your own compute.
  2. Forwarder (Lambda): AWS services that write to S3 or CloudWatch (ELB logs, CloudTrail, RDS, VPC flow logs). The forwarder subscribes and ships. Give it reserved concurrency (docs recommend at least 10) so a log flood does not starve it.
  3. Direct API: apps that cannot run an agent or forwarder. Fine for low volume; at scale you are reimplementing batching the agent does for free.

Design rules

  • One path per source. The same log arriving via agent and forwarder is a duplicate-bill and a double-count. Audit for overlaps after every architecture change.
  • Tag at the edge. Set source/service/env as close to the origin as possible (agent config, forwarder env). Routing and pipelines key off these; late tagging means misrouted logs.
  • S3 archival: decide retention per log type up front. Everything into Datadog with long retention is the expensive default; archive cold logs to S3 and rehydrate on demand.
  • Cloud service logs via the integration, not DIY forwarders: the AWS integration wires CloudWatch and S3 sources with the right parsing out of the box.

Verify

For each log source, name its path and prove it: one live-tail check per path, no source appearing twice, and the forwarder concurrency metrics flat during a traffic spike.

Evidence

The log collection docs cover the Agent, the Lambda forwarder function for AWS services and S3, and direct API submission; forwarder reserved concurrency guidance comes from the same docs. Verified against live docs 2026-09-26.

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_IQkxxpl_fKoSWEtPqTf1Mg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Log routing: agent, forwarder, or API, one path per source <span>— Vectle</span></a>