agent's budget action to stop EC2 instances failed silently, the action's IAM role was missing

TL;DR

Budget Actions fail silently when their execution role lacks permission: the action runs, gets denied, and tells nobody. Create the role with a trust policy for budgets.amazonaws.com, grant the exact stop permissions, and test the action on a sacrificial instance before trusting it.

agent's budget action to stop EC2 instances failed silently, the action's IAM role was missing

Use this when

• A budget action (stop EC2, deny IAM, apply SCP) is attached to a budget but nothing happens at the threshold.
• CloudTrail shows AccessDenied from the budgets service principal around the threshold time.
• An agent configured the action and marked the guardrail "done."

Not for this skill when

• The budget uses SNS/email alerts only. Those are notifications, not actions, and fail via the subscription path.
• You want the action to run in a different account. Cross-account actions need the role in the target account.

Steps

1. Find the action's execution role in the budget definition: aws budgets describe-budget-actions-for-budget. Verify: note the role ARN (or its absence).
2. Create the role with a trust policy allowing budgets.amazonaws.com to assume it, and attach a policy with ec2:StopInstances scoped to the target instance ARNs or a tag condition.
3. Update the budget action to reference the role.
4. Test immediately: temporarily lower the budget threshold so the action fires, or trigger it manually against a tagged test instance.
5. Check CloudTrail for the stop call with the budgets principal. Verify: the instances actually stopped.
6. Restore the real threshold and document the test date. Re-test after any role or policy change.

Variant phrasings

aws budget action not stopping instances

Symptom-level search.

budgets.amazonaws.com access denied stop instances

The CloudTrail shape of this failure.

budget action failed silently no error

How it feels when nothing tells you.

Compatibility: AWS Budget Actions require an execution role; available in regions where Budget Actions are supported.

Why it happens

Budget Actions execute as a service principal, and the console does not force you to attach a working role when you create the action. The action fires, IAM denies it, and the denial goes into CloudTrail where nobody set up an alarm. Agents skip the test step because "the API accepted the configuration."

Edge cases / pitfalls

• Scope the role tightly: a role that can stop any instance is a loaded footgun if the budget misfires.
• The action fires once per threshold evaluation, not continuously. If someone restarts the instances, the action will not stop them again until the next evaluation.
• SCP-based actions apply at the org level and are even harder to test. Use a sandbox OU first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wt2wy8DLvumAfPoBBg1BUg