Immutable solution version
simple-salesforce: avoid memory blowups on big pulls with query_all_iter
Complete solution
When a simple-salesforce bulk query eats all your container memory, switch the read path.
When a simple-salesforce bulk query eats all your container memory, switch the read path. Use queryalliter from the REST API, which streams records through a generator instead of building one giant list. The Bulk API's per-call record advantage does not help if the client library holds every row in memory anyway; the generator keeps memory flat no matter how many rows come back.
Context: GitHub issue simple-salesforce/simple-salesforce#629 (closed, 11 comments): a daily job pulling 100k rows with 250+ columns via bulk queryAll ran out of memory because the library materialized every result into a Python object. The accepted resolution in the thread: switch to queryalliter, which returns a generator that yields records one at a time instead of loading everything into memory.
Evidence
Source: GitHub issue simple-salesforce/simple-salesforce#629 (closed, 11 comments): a daily job pulling 100k rows with 250+ columns via bulk queryAll ran out of memory because the library materialized every result into a Python object. The accepted resolution in the thread: switch to queryalliter, which returns a generator that yields records one at a time instead of loading everything into memory.
Verified resolution: When a simple-salesforce bulk query eats all your container memory, switch the read path. Use queryalliter from the REST API, which streams records through a generator instead of building one giant list. The Bulk API's per-call record advantage does not help if the client library holds every row in memory anyway; the generator keeps memory flat no matter how many rows come back.
Source URL: https://github.com/simple-salesforce/simple-salesforce/issues/629
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_VjD-OX1xG6c7L1caQAfj2w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">simple-salesforce: avoid memory blowups on big pulls with query_all_iter <span>— Vectle</span></a>