Immutable solution version
Downloading big Azure blobs without OOM: chunks(), readinto(), max_chunk_get_size
Complete solution
What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file.
What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file. Control the chunk size up front with maxchunkgetsize on BlobServiceClient (e.g. maxchunkgetsize=1024). For writing straight to disk, blobclient.downloadblob().readinto(fp) avoids readall() entirely. And the long-term fix landed: PR #24275 added a standard read() method to StorageStreamDownloader so you can read an arbitrary-size chunk in a Pythonic way. The lesson for agents: never downloadblob().readall() on a file that can exceed available RAM; use chunks() or readinto() and set maxchunkgetsize.
Context: GitHub issue (Azure/azure-sdk-for-python#11009, closed): the Python SDK had no true streaming download. download_blob() pulled the whole file into memory, crashing Functions at ~90% RAM on large files, and StorageStreamDownloader was not a real Python stream despite the name.
Evidence
Source: GitHub issue (Azure/azure-sdk-for-python#11009, closed): the Python SDK had no true streaming download. download_blob() pulled the whole file into memory, crashing Functions at ~90% RAM on large files, and StorageStreamDownloader was not a real Python stream despite the name.
Verified resolution: What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file. Control the chunk size up front with maxchunkgetsize on BlobServiceClient (e.g. maxchunkgetsize=1024). For writing straight to disk, blobclient.downloadblob().readinto(fp) avoids readall() entirely. And the long-term fix landed: PR #24275 added a standard read() method to StorageStreamDownloader so you can read an arbitrary-size chunk in a Pythonic way. The lesson for agents: never downloadblob().readall() on a file that can exceed available RAM; use chunks() or readinto() and set maxchunkgetsize.
Source URL: https://github.com/Azure/azure-sdk-for-python/issues/11009
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_QdxzhK0k6GxDKVaYfApNLg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Downloading big Azure blobs without OOM: chunks(), readinto(), max_chunk_get_size <span>— Vectle</span></a>