cost agent's tag-based attribution assigned 60% of spend to "unknown" - the tagging Lambda failed on resources...

TL;DR: Two separate fixes are needed. First, activate your user-defined cost allocation tags in the Billing console, because tags are invisible to Cost Explorer and the CUR until activated, and activation is not retroactive. Second, fix the tagging Lambda so it handles CloudFormation-created resources, then backfill tags with the Tag Editor. Attribution recovers going forward from the fix date.

The symptom in Cost Explorer:

Grouped by Cost Center:  unknown  $48,200  (61% of spend)

1. Check whether the tags are even activated: in the Billing console open Cost allocation tags and look at the status column. Expected: most of your tags show Inactive, which explains why tagged resources still report as unknown.

2. Activate the user-defined tags you need for attribution. Expected: status flips to Active; new charges from this point on carry the tags, while historical data stays unknown.

3. Find why the Lambda fails on CloudFormation-created resources: check its CloudWatch logs for access denied or unsupported resource type errors, and confirm its event pattern includes CloudFormation stack events rather than only direct API calls. Expected: a concrete error, usually missing tagging permissions for specific resource types or an event source that never fires for stack-created resources.

4. Fix the Lambda permissions and event pattern, then verify new stack-created resources get tagged within minutes of creation. Expected: fresh resources appear with tags on describe calls.

5. Backfill existing resources with the Tag Editor: filter by resource type, select all, and apply the missing tags in bulk. Expected: the unknown share starts shrinking on new billing data within 24 hours.

6. Add a Config rule that flags untagged resources non-destructively so coverage never silently decays again. Expected: a compliance dashboard instead of a surprise.

Use this when

• Cost Explorer grouped by tag shows a large unknown slice
• Resources have tags in the console but billing reports ignore them
• A tagging Lambda exists but coverage keeps decaying
• CloudFormation stacks create resources that arrive untagged

Not for this skill when

• The tags are AWS-generated rather than user-defined, which activate differently
• Spend is unattributed because the payer account structure changed, which is an Organizations mapping issue
• You need historical months re-tagged, which is impossible: activation only affects new data

Variant phrasings

• cost allocation tags not showing in cost explorer
• 60 percent of spend unknown tag
• tagging lambda not tagging cloudformation resources
• activate user defined cost allocation tags

Why it happens

Tagging has two independent gates and teams usually fix only one. The Lambda is the write path: it must catch every creation event including CloudFormation stack operations, and it fails silently on resource types its IAM role cannot tag. Activation is the read path: AWS ignores user-defined tags in billing until a human flips them active in the console. Either gate closed means unknown spend, and here both were closed.

Edge cases

• Tag activation is not retroactive: the unknown months stay unknown, so annotate reports rather than rewriting history
• Some services do not support cost allocation tags at all, so a residual unknown slice is normal; chase it to zero only after checking the supported list
• Tags are case-sensitive and the Lambda must match the exact key the finance team activates, or you get two parallel tag universes
• Stack-level tags propagate only to resources whose CloudFormation definitions support them; a drifted resource keeps its old tags silently

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FA7WNgvNsLDcJomjrbdvNQ