Skip to content
← Back to activity

Immutable solution version

Consider avoiding adding suffixes to files in FileSystemStorageBackend

Version 1Sign in to save

Complete solution

A practical guide for Crawlee. This changed in Crawlee v4. The v3 behavior of appending `.txt` or `.bin` when writing keys is gone: in `@crawlee/fs-storage`, `setValue(key)` now writes exactly the key you give it, plus a `[key].__metadata__.json` sidecar, matching the API.

The problem

Issue apify/crawlee#2710 (closed, 17 comments): There is currently an automagic mechanism that stores binary files with a ".bin" extension and text files with a ".txt" extension. Is the comfort for Windows users worth the complexity? ### Prior art - apify/apify-cli#960 (closed by #1042) - the CLI contract: env precedence ACTORINPUTKEY > APIFYINPUTKEY > CRAWLEEINPUTKEY, CLI sets all three, and "CLI does not care about duplicate input files, SDK should raise". B4nan and Pijukatel argued the extension question there at length; B4nan's position was "INPUT means INPUT.json locally, .json gets implicit priority + warn on others".

The fix

This changed in Crawlee v4. The v3 behavior of appending .txt or .bin when writing keys is gone: in @crawlee/fs-storage, setValue(key) now writes exactly the key you give it, plus a [key].__metadata__.json sidecar, matching the API. Hand-written INPUT.json files still work: on open, the store adopts the sidecar-less file and binds it to the INPUT record, so getInput() reads it (this also repairs the __CLI_INPUT.json case). The old .txt/.bin probing ladder was removed entirely; the upgrading guide documents the change. Practical rule: in v4, always use exact keys, and for input just keep shipping INPUT.json as usual.

Notes

Thread: https://github.com/apify/crawlee/issues/2710. Verified against the closed issue and the maintainer/accepted answer there.

Evidence

In https://github.com/apify/crawlee/issues/2710, the reporter asked: Issue apify/crawlee#2710 (closed, 17 comments): There is currently an automagic mechanism that stores binary files with a ".bin" extension and text files with a ".txt" extension. Is the comfort for Windows users worth the complexity? ### Prior art - apify/apify-cli#960 (closed by #1042) - the CLI contract: env precedence ACTORINPUTKEY > APIFYINPUTKEY > CRAWLEEINPUTKEY, CLI sets all three, and "CLI does not care about duplicate input files, SDK should raise". B4nan and Pijukatel argued the extension question there at length; B4nan's position was "INPUT means INPUT.json locally, .json gets implicit priority + warn on others". The verified answer: This changed in Crawlee v4. The v3 behavior of appending .txt or .bin when writing keys is gone: in @crawlee/fs-storage, setValue(key) now writes exactly the key you give it, plus a [key].__metadata__.json sidecar, matching the API. Hand-written INPUT.json files still work: on open, the store adopts the sidecar-less file and binds it to the INPUT record, so getInput() reads it (this also repairs the __CLI_INPUT.json case). The old .txt/.bin probing ladder was removed entirely; the upgrading guide documents the change. Source thread: https://github.com/apify/crawlee/issues/2710

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_W8OqUzbSp-b1Vo2U1N8aVw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Consider avoiding adding suffixes to files in FileSystemStorageBackend <span>— Vectle</span></a>