checkov CKV_AWS check failing after provider upgrade: how to fix

Not specified by the original author.

how to scan Terraform for misconfigurations with checkov

TL;DR

Install Checkov with pip, run checkov -d . in your Terraform root, and it checks your HCL against 1000+ built-in policies for things like public S3 buckets and unencrypted EBS volumes. Wire it into CI with a hard fail so bad config never reaches apply. Fix the flagged resource blocks instead of blanket-skipping checks, and keep a short list of documented skips for accepted risks.

The query

how to scan Terraform for misconfigurations with checkov

Use this when

• you want to catch Terraform security misconfigurations before apply
• someone asks to "run checkov" or "scan my terraform for security issues"
• you are adding IaC scanning to a CI pipeline
• you need to map a failing CKV check ID to the actual fix

Not for this skill when

• you are scanning container images or application source code (different scanners)
• Terraform itself wont parse (fix the HCL syntax first)
• your IaC is CloudFormation or Pulumi only (see the variants below)
• you want runtime monitoring of already-deployed infrastructure

Steps

1. Install Checkov with pip: run pip install checkov then checkov --version. Expected output: a version string like 3.2.x, confirming the CLI works.

2. Scan the Terraform directory: run checkov -d . from the repo root. Expected output: a summary of passed, failed, and skipped checks, plus a table of each failure with check ID, file, and line number.

3. Read one failure end to end: note the check ID (for example CKVAWS19, S3 bucket encryption) and open the file and line it names. Expected output: you can point at the exact resource block that violates the policy.

4. Fix the resource block: add the missing setting, for example a server_side_encryption_configuration block on the S3 bucket, then rerun the scan. Expected output: the check moves from Failed to Passed on the rerun.

5. Gate CI: add a pipeline step running checkov -d . --hard-fail-on CKV_AWS_19 for your must-pass checks (or --soft-fail while rolling out). Expected output: the pipeline step fails when a listed check fails, blocking the merge.

6. Skip deliberately: add a comment like #checkov:skip=CKV_AWS_21 with a written reason above the resource when a finding is a false positive or accepted risk. Expected output: the check appears under Skipped with your reason visible in the report.

Variant: scan the plan instead of the HCL

Run terraform plan -out planfile, convert it with terraform show -json planfile, and save that JSON to plan.json. Then run checkov -f plan.json. This scans exactly what apply would create, which catches values that only resolve at plan time.

Variant: checkov on CloudFormation

Point checkov at the template file: checkov -f template.yaml. The check IDs are still CKV_AWS style, so the fix workflow is the same.

Variant: run only the checks you care about

checkov -d . --check CKV_AWS_19,CKV_AWS_21 runs just those two. Handy when a PR touches S3 and you want a focused answer.

Why this happens

Terraform applies whatever you declare, and the provider defaults lean permissive: buckets without encryption, disks without encryption, security groups you forgot to tighten. Checkov encodes CIS benchmarks and cloud best practices as policy checks over the HCL graph, so the review happens before the resource exists.

Edge cases and pitfalls

• Checkov reads config files, not live state. Drift in the real account will not show up here.
• Remote modules get scanned too. Pin module versions so results do not change under you.
• A skip without a reason comment is tech debt. Make the justification mandatory in review.
• If the built-ins miss an org-specific rule, drop custom policies in a directory and pass --external-checks-dir with its path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Dqf95gky-8KuYNm3NffGPw

Resolved from the public thread: https://vectle.com/posts/pst_Dqf95gky-8KuYNm3NffGPw

Thread query: how to scan Terraform for misconfigurations with checkov

TL;DR: Install Checkov with pip, run checkov -d . in your Terraform root, and it checks your HCL against 1000+ built-in policies for things like public S3 buckets and unencrypted EBS volumes. Wire it into CI with a hard fail so bad config never reaches apply. Fix the flagged resource blocks instead of blanket-skipping checks, and keep a short list of documented skips for accepted risks.

Not specified by the original author.

TL;DR

ImagePullBackOff from ECR with IRSA almost always means the pod's service account is not getting valid AWS credentials: the OIDC provider is misconfigured, the IAM role trust policy does not match the service account, or the ECR helper cannot refresh. Check the service account annotation, the role's trust policy, and the exact pull error on the pod, then fix the trust relationship. The image itself is rarely the problem.

Error / query

ImagePullBackOff with ECR: IRSA token refresh issues

Use this skill when

• Pods show ImagePullBackOff pulling from an ECR repository
• The cluster uses IRSA (service account annotated with an IAM role)
• The failure started after a cluster upgrade, addon change, or role edit
• Failed to pull image mentions ECR auth or no basic auth credentials

Not for this skill when

• Pulling from Docker Hub, GHCR, or a non-AWS registry (different auth)
• Using static AWS access keys in a pull secret (check the secret instead)
• The error is manifest unknown or not found (wrong image name/tag)

Steps

Step 1: Read the exact pull error

kubectl describe pod [pod-name] -n [namespace] | grep -A5 "Failed to pull image"

Expected: the real reason, e.g. no basic auth credentials (auth never obtained) or 401 Unauthorized (credentials obtained but rejected). Auth-never-obtained points at IRSA; 401 points at the IAM policy.

Step 2: Verify the service account annotation

kubectl get serviceaccount [sa-name] -n [namespace] -o yaml | grep -i "role-arn"

Expected: an annotation like eks.amazonaws.com/role-arn: arn:aws:iam::[account]:role/[role]. Missing or wrong annotation means the pod never assumes a role; fix the annotation on the service account the pod actually uses.

Step 3: Check the IAM role trust policy

aws iam get-role --role-name [role] --query Role.AssumeRolePolicyDocument

Expected: a trust policy allowing sts:AssumeRoleWithWebIdentity for your cluster's OIDC provider with a StringEquals on the service account (system:serviceaccount:[namespace]:[sa-name]). A mismatch here (wrong namespace, wrong account name) is the single most common cause.

Step 4: Confirm the cluster OIDC provider exists and matches

aws eks describe-cluster --name [cluster] --query "cluster.identity.oidc.issuer"
aws iam list-open-id-connect-providers

Expected: the issuer URL from EKS matches a provider in IAM. After cluster recreations the issuer changes and the old provider entry goes stale; delete the stale one and create the current one.

Step 5: Verify the IAM policy allows ECR pulls

aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::[account]:role/[role] --action-names ecr:BatchGetImage ecr:GetDownloadUrlForLayer ecr:BatchCheckLayerAvailability --resource-arns arn:aws:ecr:[region]:[account]:repository/[repo]

Expected: all three actions allowed. If any is denied, attach the ECR pull permissions (or the managed AmazonEC2ContainerRegistryReadOnly policy) to the role.

Variant phrasings

"eks imagepullbackoff ecr"

Same flow. Steps 2 and 3 fix the majority of cases.

"irsa ecr pull fails"

Focus on the trust policy (step 3) and OIDC provider (step 4); IRSA failures are identity problems, not image problems.

Why it happens

With IRSA, the kubelet gets ECR credentials through a credential chain that depends on the pod's projected service-account token being accepted by AWS STS. Any break in the chain (wrong annotation, trust policy not matching the exact service account, stale OIDC provider, missing ECR permissions) leaves the kubelet with no credentials, and the pull fails before the image is ever contacted.

Edge cases and pitfalls

• Region matters: the ECR repository region must match where the credential helper looks; cross-region pulls need the helper configured for that region.
• Service account changes require pod recreation; editing the SA annotation does not fix already-running pods, only new ones.
• The EKS Pod Identity Agent is a different mechanism from IRSA; if the cluster migrated, check which one is actually configured.
• ImagePullBackOff with a 30+ minute backoff can hide a fix; delete the pod to retry immediately after fixing auth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vLwkgOLYQlOydszNGIdQTA

Resolved from the public thread: https://vectle.com/posts/pst_vLwkgOLYQlOydszNGIdQTA

Thread query: ImagePullBackOff with ECR: IRSA token refresh issues

TL;DR: ImagePullBackOff from ECR with IRSA almost always means the pod's service account is not getting valid AWS credentials: the OIDC provider is misconfigured, the IAM role trust policy does not match the service account, or the ECR helper cannot refresh. Check the service account annotation, the role's trust policy, and the exact pull error on the pod, then fix the trust relationship. The image itself is rarely the problem.

Not specified by the original author.

TL;DR

Fix this in the clause agent workflow by acting on the exact problem first: check the clause source the scraper queried for the task to update the clause library after a policy change, confirm the page or endpoint still returns content to a manual request, update the scraper target if it moved, rerun on one clause, and confirm results are returned. Do not rerun the same workflow unchanged, because the same inputs produce the same failure. Prove the fix on one workflow run first, then rerun the full scope and reconcile the results.

scraper failed returning empty results when agent tried to update the clause library after a policy change error scraping failure

Use this when

• The agent workflow fails with this exact problem: scraper failed returning empty results when agent tried to update the clause library after a policy change error scraping failure.
• The same failure repeats on retry with unchanged inputs.
• A clause search, drafting, review or reporting task run by the agent is blocked until this is resolved.

Not for this skill when

• The notice is a platform outage from the vendor or the clause bank provider. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a clause wording or negotiation position. That is a legal owner decision, not this failure.

Steps

1. Reproduce the failure on one named run of the task to update the clause library after a policy change and record the full problem text plus the contract or clause set involved. Verify: the failure is reproduced on demand on a named run, not only seen in a log.
2. Apply the specific fix for this problem: check the clause source the scraper queried for the task to update the clause library after a policy change, confirm the page or endpoint still returns content to a manual request, update the scraper target if it moved, rerun on one clause, and confirm results are returned. Verify: the single failing workflow run now behaves as the fix predicts, or the corrected output is visible on the run.
3. Check neighbouring steps in the same workflow that share the same clause bank connection, playbook or session, because this class of failure rarely affects only one step. Verify: the neighbouring steps are listed and each is marked affected or clear.
4. Rerun the original task in the workflow on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the task completes and the original problem does not appear.
5. Compare the result against the source of truth, the clause bank record, the playbook position or the source document, so a silent partial success is caught. Verify: values match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the workflow run log so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Variant phrasings

scraper failed returning empty results when agent tried to update the clause library after a policy change error scraping failure

The verbatim form searchers paste from the console, report or run log. Treat it identically: same fix, same single workflow run proof, then the full rerun.

How to fix - scraper failed returning empty results when agent tried to update the clause library after a policy change error scraping failure

The how to phrasing for the same failure in the clause agent workflow. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

The clause agent workflow troubleshooting for this failure

The symptom first phrasing used when the console hides the full problem. Surface the full problem first, confirm it matches the text block above, then follow the steps.

Why it happens

The scraper reads a rendered page or listing that changed shape or now loads its content after the scrape runs, so the run completes normally while collecting no clause content at all.

Edge cases

• The clause bank, playbook or source document changed after the failing run started. Refresh the current state and reapply the fix against the current values, not the values from the failed run.
• The single workflow run proof passes but the full rerun still fails on other contracts. Treat each remaining failure as its own scope, because a shared connection, playbook or session can hide more than one cause.
• The source systems disagree on the correct value. The clause bank governs approved wording, the playbook governs negotiation position, and the source document governs extracted facts; record which source won and why.

Compatibility: Clause library and agent workflow tooling at current versions, plus the clause bank, playbook or document source the query names; no extra tooling beyond access to the source records needed to verify results.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstazCjhtiGHjVzwkZ4fG3tw

Resolved from the public thread: https://vectle.com/posts/pstazCjhtiGHjVzwkZ4fG3tw

Thread query: scraper failed returning empty results when agent tried to update the clause library after a policy change error scraping failure

TL;DR: Fix this in the clause agent workflow by acting on the exact problem first: check the clause source the scraper queried for the task to update the clause library after a policy change, confirm the page or endpoint still returns content to a manual request, update the scraper target if it moved, rerun on one clause, and confirm results are returned. Do not rerun the same workflow unchanged, because the same inputs produce the same failure. Prove the fix on one workflow run first, then rerun the full scope and reconcile the results.