simple-salesforce: avoid memory blowups on big pulls with query_all_iter

TL;DR

Use queryalliter from the REST API, which streams records through a generator instead of building one giant list. When a simple-salesforce bulk query eats all your container memory, switch the read path.

Steps

1. When a simple-salesforce bulk query eats all your container memory, switch the read path. Use queryalliter from the REST API, which streams records through a generator instead of building one giant list. The Bulk API's per-call record advantage does not help if the client library holds every row in memory anyway; the generator keeps memory flat no matter how many rows come back.

When to use

You are seeing this: The Bulk API's per-call record advantage does not help if the client library holds every row in memory anyway; the generator keeps memory flat no matter how many rows come back. Use this skill when you run into "simple-salesforce: avoid memory blowups on big pulls with queryalliter".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: GitHub issue simple-salesforce/simple-salesforce#629 (closed, 11 comments): a daily job pulling 100k rows with 250+ columns via bulk queryAll ran out of memory because the library materialized every result into a Python object. The accepted resolution in the thread: switch to queryalliter, which returns a generator that yields records one at a time instead of loading everything into memory.

Verified resolution: When a simple-salesforce bulk query eats all your container memory, switch the read path. Use queryalliter from the REST API, which streams records through a generator instead of building one giant list. The Bulk API's per-call record advantage does not help if the client library holds every row in memory anyway; the generator keeps memory flat no matter how many rows come back.

Source URL: https://github.com/simple-salesforce/simple-salesforce/issues/629