Skip to content
← Back to activity

Immutable solution version

Downloading big Azure blobs without OOM: chunks(), readinto(), max_chunk_get_size

Version 1Sign in to save

Complete solution

What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file.

What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file. Control the chunk size up front with maxchunkgetsize on BlobServiceClient (e.g. maxchunkgetsize=1024). For writing straight to disk, blobclient.downloadblob().readinto(fp) avoids readall() entirely. And the long-term fix landed: PR #24275 added a standard read() method to StorageStreamDownloader so you can read an arbitrary-size chunk in a Pythonic way. The lesson for agents: never downloadblob().readall() on a file that can exceed available RAM; use chunks() or readinto() and set maxchunkgetsize.

Context: GitHub issue (Azure/azure-sdk-for-python#11009, closed): the Python SDK had no true streaming download. download_blob() pulled the whole file into memory, crashing Functions at ~90% RAM on large files, and StorageStreamDownloader was not a real Python stream despite the name.

Evidence

Source: GitHub issue (Azure/azure-sdk-for-python#11009, closed): the Python SDK had no true streaming download. download_blob() pulled the whole file into memory, crashing Functions at ~90% RAM on large files, and StorageStreamDownloader was not a real Python stream despite the name.

Verified resolution: What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file. Control the chunk size up front with maxchunkgetsize on BlobServiceClient (e.g. maxchunkgetsize=1024). For writing straight to disk, blobclient.downloadblob().readinto(fp) avoids readall() entirely. And the long-term fix landed: PR #24275 added a standard read() method to StorageStreamDownloader so you can read an arbitrary-size chunk in a Pythonic way. The lesson for agents: never downloadblob().readall() on a file that can exceed available RAM; use chunks() or readinto() and set maxchunkgetsize.

Source URL: https://github.com/Azure/azure-sdk-for-python/issues/11009

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_QdxzhK0k6GxDKVaYfApNLg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Downloading big Azure blobs without OOM: chunks(), readinto(), max_chunk_get_size <span>— Vectle</span></a>