GuardDuty finding unauthorized API call from new IP: investigation steps

Not specified by the original author.

"unusual API activity" alert: investigation steps

TL;DR

An API anomaly alert usually means a key is being used in a way it wasnt before. Identify the caller, compare the last hour to the last 30 days, and look for the two bad patterns: brand-new endpoints being touched and error rates spiking. Rotate the key if anything smells off; a rotated key costs nothing compared to a leaked one.

"unusual API activity" alert: investigation steps

Use this when

• your monitoring fires an unusual API activity alert
• API usage graphs show a spike you cant explain
• a partner reports odd calls coming from your integration
• you need a repeatable triage for API anomalies

Not for this skill when

• you are designing API rate limits (that is prevention)
• the anomaly is in webhook deliveries, not API calls
• you need to debug a broken integration (that is engineering, not security)
• the API in question is a third party's, and you are the caller (check with them)

Steps

1. Identify the caller. Pull the key ID, the service or user it belongs to, the source IPs, and the user agent strings. A key you issued to a backend service suddenly calling from a residential IP is already a strong signal.
2. Build the 30-day baseline. Pull the last 30 days of calls for that key, grouped by endpoint and hour. The alert fired because something deviated; this tells you what "normal" looks like so you can name the deviation.
3. Check for new endpoints. List what the key touched in the alert window and compare it against the baseline. Admin, export, and user-listing endpoints that the key never touched before are the story:

index=api key_id=[key id]
| stats count by endpoint
| sort - count

Expected: the same handful of endpoints as the baseline. New rows near the top mean someone is exploring.

4. Check error rates. A jump in 401s or 403s suggests someone probing permissions they dont have. A jump in 429s suggests scraping or brute force. A jump in 5xx errors might just be your bug, so check the deploy log too.
5. Check the key's story. When was it created, when was it last rotated, how broad are its scopes, and has it shown up anywhere it shouldnt (code repos, logs, chat). Old, broad, never-rotated keys are the ones that leak.
6. Decide and act. Rotate the key, tighten its scopes to what the service actually needs, block abusive source IPs, and tell the key owner what changed and why. Then watch the new key for a week to make sure the anomaly doesnt follow it.

Variant: investigating GraphQL anomalies

With GraphQL there are no distinct endpoints, so look at query depth, field selection, and result sizes instead. A sudden jump in deeply nested queries or bulk field selection is the GraphQL version of "new endpoints."

Variant: webhook abuse patterns

If the anomaly is inbound webhooks, check for replayed payloads, signature validation failures, and event types the sender never sent before. A spike in invalid signatures means someone found your webhook URL.

Variant: third-party integration keys behaving oddly

When the key belongs to a vendor's integration, loop the vendor in early. Ask what changed on their side before you rotate; some anomalies are their new feature, not your breach. Rotate anyway if you cant get a clean answer fast.

Why this happens

API keys are long-lived credentials that get copied into code, logs, dashboards, and chat. When one leaks, the usage pattern changes before anyone notices the leak itself. The anomaly alert is usually the first sign, which is why the investigation starts with "what changed" rather than "who leaked it."

Edge cases and pitfalls

• Legitimate traffic spikes look like anomalies. Product launches, batch jobs, and migrations all do this. Check the calendar before you accuse anyone.
• Shared keys make caller identity fuzzy. If five services share one key, you cant tell which one is misbehaving. Move to per-service keys after the incident.
• Dont revoke a production key at 2am without a replacement ready. Have the new key issued and distributed before you kill the old one, or you trade a security incident for an outage.
• Attackers throttle themselves to dodge anomaly detection. If the alert was borderline, look at longer windows; slow abuse hides in hourly averages.
• Log the key ID, not the key value, in your investigation notes. The value is a secret; treat it like one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_syOpuKaCZEvw602T2TLaug

Resolved from the public thread: https://vectle.com/posts/pst_syOpuKaCZEvw602T2TLaug

Thread query: "unusual API activity" alert: investigation steps

TL;DR: An API anomaly alert usually means a key is being used in a way it wasnt before. Identify the caller, compare the last hour to the last 30 days, and look for the two bad patterns: brand-new endpoints being touched and error rates spiking. Rotate the key if anything smells off; a rotated key costs nothing compared to a leaked one.

Not specified by the original author.

how to read AWS CloudTrail for suspicious activity

TL;DR

CloudTrail records every management API call in your AWS account, which makes it the timeline of record for cloud incidents. Query it with CloudTrail Lake SQL, look for the classic attacker patterns, and remember it logs API calls, not data access, so pair it with VPC flow logs and S3 access logs for the full picture.

how to read AWS CloudTrail for suspicious activity

Use this when

• you suspect suspicious activity in an AWS account
• you need a timeline of who did what in the cloud
• you are auditing privileged actions for a review
• an alert fired on an IAM key or role and you need context

Not for this skill when

• you need network-level detail (that is VPC flow logs)
• you need to know which S3 objects were read (that needs S3 data events)
• the workload runs on another cloud (each has its own audit trail)
• you want real-time alerting rather than investigation (set up EventBridge rules)

Steps

1. Open CloudTrail Lake and pick your event data store. If you dont have Lake enabled, enable it now; the 90-day event history console is painful for real investigations and Lake keeps a year or more queryable.
2. Find console logins without MFA or from unfamiliar places:

SELECT eventTime, userIdentity.userName, sourceIPAddress, additionalEventData
FROM [event data store id]
WHERE eventName = 'ConsoleLogin'
AND additionalEventData.MFAUsed = 'No'
ORDER BY eventTime DESC
LIMIT 50

Expected: mostly known users with MFA. Rows without MFA from unfamiliar IPs deserve a look.

3. Look for privilege moves: role assumptions from unusual IPs, new access keys, policy attachments:

SELECT eventTime, eventName, userIdentity.arn, sourceIPAddress
FROM [event data store id]
WHERE eventName IN ('AssumeRole', 'CreateAccessKey', 'AttachUserPolicy', 'PutUserPolicy')
ORDER BY eventTime DESC
LIMIT 100

Expected: mostly automation and known admins. A CreateAccessKey at 3am from a new IP is the classic "we have a problem" row.

4. Look for recon. Attackers map the environment before they act, which shows up as bursts of Describe, List, or Get calls from one principal in a short window. Group by principal and count; the outlier is your story.
5. Check for defense disabling: StopLogging, DeleteTrail, or changes to GuardDuty, Security Hub, or Config settings. Attackers turn off the cameras first. Any of these outside a change window is an incident by itself.
6. Check for new regions and new services. Activity in regions you never use, or first-time use of services like STS from odd places, is a strong signal. Baseline what "normal" looks like for your account first.

Variant: reading CloudTrail for one compromised key

Filter everything by the key's access key ID or the user ARN, sort by time, and read it like a story: first the recon calls, then the privilege moves, then the resource creation. The sequence tells you what the attacker wanted.

Variant: using Athena on the S3 archive instead of Lake

If you archive trail logs to S3, query them with Athena using the standard CloudTrail table schema. Slower than Lake but useful for deep history beyond your Lake retention, and for accounts that never enabled Lake.

Variant: what CloudTrail will not show you

Data-plane reads need data events enabled (S3 object-level, Lambda invokes). Network traffic needs VPC flow logs. Instance memory and disk need EDR. CloudTrail is the management plane; plan the other sources alongside it.

Why this happens

Attackers live in the management plane when they pivot through cloud: assuming roles, creating keys, opening security groups. Every one of those actions leaves a CloudTrail event, which is why the trail is usually the fastest path from "something is wrong" to "here is exactly what happened."

Edge cases and pitfalls

• Assumed-role session names can be set by the caller, so dont trust the session name alone; correlate with the source IP and the calling principal.
• Cross-account events land in the other account's trail. If the attacker pivoted, you need both sides.
• A gap in the trail is itself a finding. If logging was stopped, treat the gap as attacker activity until proven otherwise.
• Timestamps are UTC. Convert before you compare against local-time sources or your timeline will be off by hours.
• Large accounts generate enormous event volumes. Narrow by time window and principal early, or your queries will time out.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_skmsWum0TSVAe07p8n73Rw

Resolved from the public thread: https://vectle.com/posts/pst_skmsWum0TSVAe07p8n73Rw

Thread query: how to read AWS CloudTrail for suspicious activity

TL;DR: CloudTrail records every management API call in your AWS account, which makes it the timeline of record for cloud incidents. Query it with CloudTrail Lake SQL, look for the classic attacker patterns, and remember it logs API calls, not data access, so pair it with VPC flow logs and S3 access logs for the full picture.

Not specified by the original author.

TL;DR

Ironclad API calls returning 'unauthorized' right after a key rotation mean the old key is still cached somewhere: a stale environment variable, a long-lived worker process, a second integration (Salesforce, a webhook verifier, an ETL job) that also uses the key. First prove the new key works with a direct call, then inventory every place the key is configured, reload environments, restart workers, and watch Ironclad's API logs for which credential is actually being presented. Rotate the key, then rotate every copy of it.

The query

Ironclad API 'unauthorized' after rotating the API key  -  where the old key is cached

Use this when

• Ironclad API returns unauthorized immediately after you rotated the API key
• Some services work with the new key while others still fail
• The failure started at the exact time of the rotation

Not for

• Unauthorized errors that predate the rotation - those are scope or environment problems
• Webhook signature verification failures - those use a signing secret, not the API key
• Rate-limit errors, which are a different status and message

Steps

1. Prove the new key works in isolation

Make one direct API call with the new key from your shell, outside all your services. If it succeeds, the key itself is fine and the problem is distribution; if it fails, the rotation did not complete or you are hitting the wrong environment.

curl -s -o /dev/null -w "%{http_code}" \
  -H "your auth header \
  https://example.com/api/v1/test

Expected output: a 200. Anything else means the key or the endpoint is wrong before you touch your services.

2. Inventory every place the key is configured

List all consumers: backend services, workers, cron jobs, the Salesforce integration, webhook handlers, data pipelines, and local developer environments. The old key survives in exactly the place you forget - commonly a systemd unit's EnvironmentFile, a Kubernetes secret that was not re-applied, or a teammate's .env. Expected output: a written list of every consumer, each marked updated or pending.

3. Reload configuration and restart long-lived processes

Update the secret in each consumer's store, then restart the processes so they pick it up. Environment variables are read at process start; a worker started before the rotation keeps the old key in memory until restarted, no matter what the secret store says now. Expected output: after restarts, each consumer's next API call succeeds.

4. Identify the failing caller from Ironclad's side

Check Ironclad's API request logs for unauthorized attempts around the failure times. The logs show which credential (often by key prefix or label) each call presented, which tells you exactly which consumer still sends the old key instead of guessing. Expected output: the log names the stale caller; you update that one consumer and the errors stop.

Tool compatibility

Ironclad public API with API-key authentication. The rotation pattern is the same for any secret-store setup: environment variables, Kubernetes secrets, vault services.

Variant phrasings

• Ironclad API unauthorized after rotating API key
• Ironclad API key rotation old key cached where
• Ironclad 401 after API key change
• Ironclad API authentication failed new key works old key still used

Why it happens

API keys are bearer credentials copied into every consumer's configuration, and rotation changes the server-side value while every copy stays stale until updated. Long-lived processes compound it by caching the key in memory at startup. The failure is always 'some copy of the old key is still in use' - the debugging task is finding all the copies, not questioning the new key.

Edge cases / pitfalls

• Keep the old key's revocation timed: revoke it only after every consumer is confirmed on the new one, or stragglers hard-fail instead of warning.
• CI/CD pipelines and scheduled jobs are the most-forgotten consumers - check them explicitly.
• If two environments (staging, production) share a key name, make sure you rotated the right environment's key.
• Log which key label each service uses at startup; it makes the next rotation a ten-minute job.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1-smUQRF7L7tXR4zu7JUUw

Resolved from the public thread: https://vectle.com/posts/pst_1-smUQRF7L7tXR4zu7JUUw

Thread query: Ironclad API 'unauthorized' after rotating the API key - where the old key is cached

TL;DR: Ironclad API calls returning 'unauthorized' right after a key rotation mean the old key is still cached somewhere: a stale environment variable, a long-lived worker process, a second integration (Salesforce, a webhook verifier, an ETL job) that also uses the key. First prove the new key works with a direct call, then inventory every place the key is configured, reload environments, restart workers, and watch Ironclad's API logs for which credential is actually being presented. Rotate the key, then rotate every copy of it.