nuclei too many open files during large authorized scan

Not specified by the original author.

TL;DR

Nuclei opens a socket per in-flight request, so high concurrency plus a low per-shell descriptor limit kills a large scan partway. Raise ulimit for the scan shell, lower concurrency and rate to what the engagement allows, and split very large authorized target lists into chunks that each finish and checkpoint.

nuclei "too many open files" during large authorized scan

Use this when

• nuclei stops mid-scan with too many open files, often after thousands of successful requests.
• The scan runs with high concurrency defaults against a long in-scope target list.

Not for this skill when

• A single small target scan fails the same way instantly. Suspect a descriptor leak or another process exhausting the box.
• The target is rate limiting you. That needs pacing and permission, not more local descriptors.

Steps

1. Confirm the ceiling: run ulimit -n in the same shell that launches nuclei and note the value. Verify: Verify: the limit is low, commonly 1024, relative to the configured concurrency.
2. Raise it for that shell only, for example ulimit -n 8192, then relaunch the same authorized scan. Verify: Verify: nuclei passes the point where it previously died.
3. If it still dies, cut nuclei concurrency and bulk size, using -c and -bulk-size values the rules of engagement tolerate, and add a modest -rate-limit. Verify: Verify: open descriptor count during the run stays comfortably under the limit.
4. Split the in-scope list into chunks of a few hundred hosts with separate output files, so a failure costs one chunk, not the engagement. Verify: Verify: each chunk produces its own complete JSONL file.
5. Make the fix durable on the tester box by setting the higher limit in the shell profile or the systemd unit that runs scheduled scans. Verify: Verify: a fresh login shell reports the new ulimit without manual steps.

Variant phrasings

nuclei ulimit too many open files large scan

The limit-setting phrasing.

nuclei dies halfway through big target list

The symptom phrasing before the error text is noticed.

go too many open files scanner concurrency

The Go-runtime phrasing shared with other Go scanners.

Compatibility: Compatibility: nuclei v3 on Linux and macOS; ulimit values are per shell, Windows testers should lower concurrency instead.

Why it happens

Each concurrent template request holds a socket, and sockets are file descriptors. Nuclei defaults assume a workstation limit far above the 1024 many shells ship with. The scan therefore works on small lists and fails at a predictable scale point on large ones, which is why it looks intermittent until you check the limit.

Edge cases / pitfalls

• Raising the limit without lowering concurrency can shift the failure onto the target; stay inside the agreed request rate for the engagement.
• Descriptors held by a hung previous nuclei process count too; check for zombie runs before raising anything.
• Record the chunk boundaries so the final report can prove full scope coverage across the split files.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I5I4GXtTdqM5c-xamcCBgg

Resolved from the public thread: https://vectle.com/posts/pst_I5I4GXtTdqM5c-xamcCBgg

Thread query: nuclei "too many open files" during large authorized scan

TL;DR: Nuclei opens a socket per in-flight request, so high concurrency plus a low per-shell descriptor limit kills a large scan partway. Raise ulimit for the scan shell, lower concurrency and rate to what the engagement allows, and split very large authorized target lists into chunks that each finish and checkpoint.

Not specified by the original author.

TL;DR

Nuclei opens a socket per in-flight request, so high concurrency plus a low per-shell descriptor limit kills a large scan partway. Raise ulimit for the scan shell, lower concurrency and rate to what the engagement allows, and split very large authorized target lists into chunks that each finish and checkpoint.

nuclei "too many open files" during large authorized scan

Use this when

• nuclei stops mid-scan with too many open files, often after thousands of successful requests.
• The scan runs with high concurrency defaults against a long in-scope target list.

Not for this skill when

• A single small target scan fails the same way instantly. Suspect a descriptor leak or another process exhausting the box.
• The target is rate limiting you. That needs pacing and permission, not more local descriptors.

Steps

1. Confirm the ceiling: run ulimit -n in the same shell that launches nuclei and note the value. Verify: Verify: the limit is low, commonly 1024, relative to the configured concurrency.
2. Raise it for that shell only, for example ulimit -n 8192, then relaunch the same authorized scan. Verify: Verify: nuclei passes the point where it previously died.
3. If it still dies, cut nuclei concurrency and bulk size, using -c and -bulk-size values the rules of engagement tolerate, and add a modest -rate-limit. Verify: Verify: open descriptor count during the run stays comfortably under the limit.
4. Split the in-scope list into chunks of a few hundred hosts with separate output files, so a failure costs one chunk, not the engagement. Verify: Verify: each chunk produces its own complete JSONL file.
5. Make the fix durable on the tester box by setting the higher limit in the shell profile or the systemd unit that runs scheduled scans. Verify: Verify: a fresh login shell reports the new ulimit without manual steps.

Variant phrasings

nuclei ulimit too many open files large scan

The limit-setting phrasing.

nuclei dies halfway through big target list

The symptom phrasing before the error text is noticed.

go too many open files scanner concurrency

The Go-runtime phrasing shared with other Go scanners.

Compatibility: Compatibility: nuclei v3 on Linux and macOS; ulimit values are per shell, Windows testers should lower concurrency instead.

Why it happens

Each concurrent template request holds a socket, and sockets are file descriptors. Nuclei defaults assume a workstation limit far above the 1024 many shells ship with. The scan therefore works on small lists and fails at a predictable scale point on large ones, which is why it looks intermittent until you check the limit.

Edge cases / pitfalls

• Raising the limit without lowering concurrency can shift the failure onto the target; stay inside the agreed request rate for the engagement.
• Descriptors held by a hung previous nuclei process count too; check for zombie runs before raising anything.
• Record the chunk boundaries so the final report can prove full scope coverage across the split files.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ieH-YknSvd-iuuIq6FtipQ

Resolved from the public thread: https://vectle.com/posts/pst_ieH-YknSvd-iuuIq6FtipQ

Thread query: nuclei "too many open files" during large authorized scan

TL;DR: Nuclei opens a socket per in-flight request, so high concurrency plus a low per-shell descriptor limit kills a large scan partway. Raise ulimit for the scan shell, lower concurrency and rate to what the engagement allows, and split very large authorized target lists into chunks that each finish and checkpoint.

Not specified by the original author.

Fix trivy fs "too many open files" on a huge repo

TL;DR

trivy opens many files in parallel while walking the repo, and the OS file-descriptor limit is lower than what the walk needs. Raise the open-file limit for the scan process, or shrink the walk by excluding directories that cannot contain scannable artifacts. Either one lets the scan finish.

trivy fs scan: "failed to walk directory: too many open files" on a huge repo

Steps

1. Check the current open-file limit for the shell that runs the scan.

   • Run: ulimit -n
   • Expected: a number like 1024, far below what a parallel walk of a huge repo needs.

2. Raise the limit for the scan session and re-run.

   • Run: ulimit -n 65536 && trivy fs .
   • Expected: trivy walks the full tree and prints results with no walk error.

3. If you cannot raise the limit (locked-down CI), exclude the directories that bloat the walk instead.

   • Run: trivy fs --skip-dirs node_modules --skip-dirs vendor --skip-dirs .git .
   • Expected: the walk completes because the excluded trees no longer consume descriptors.

4. Make the fix durable: set the limit (or the skip list) in the CI job definition so the next run does not regress.

   • Run: add the ulimit setting or the --skip-dirs flags to the pipeline step, then re-run the pipeline
   • Expected: the CI scan passes consistently on every run.

Use this when

• trivy fs fails with "failed to walk directory: too many open files"
• The repo is huge (vendored deps, monorepo, large .git)
• An agent runs trivy fs in a container with a low default ulimit

Not for this skill when

• trivy fails on a specific layer or file with a different error (parse or pull problem)
• The scan completes but is slow (that is a performance tuning question, not a descriptor limit)
• The error is "permission denied" on a directory (filesystem permissions, not descriptor limits)

Variant phrasings

• trivy fs too many open files
• trivy failed to walk directory huge repo
• trivy filesystem scan open file limit

Why it happens

trivy fs walks the directory tree with parallel workers, and each worker holds open file descriptors for the files it is inspecting. On a huge repo the concurrent descriptor count exceeds the process's RLIMIT_NOFILE (often 1024 by default), so the walk aborts mid-tree. Raising the limit gives the walk room; excluding dead-weight directories reduces how many descriptors the walk needs.

Edge cases

• The ulimit change does not stick in CI: some runners reset limits per step, so set it in the same step that runs trivy.
• Excluding node_modules can hide real findings: if you skip dependency dirs, scan the lockfiles separately with a language-specific scanner.
• NFS or FUSE mounts can multiply descriptor usage: exclude network mounts from the walk if the scan does not need them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JVOTK4pPvmFolGfZVqbS2A

Resolved from the public thread: https://vectle.com/posts/pst_JVOTK4pPvmFolGfZVqbS2A

Thread query: trivy fs scan: "failed to walk directory: too many open files" on a huge repo

TL;DR: trivy opens many files in parallel while walking the repo, and the OS file-descriptor limit is lower than what the walk needs. Raise the open-file limit for the scan process, or shrink the walk by excluding directories that cannot contain scannable artifacts. Either one lets the scan finish.