Downloading big Azure blobs without OOM: chunks(), readinto(), max_chunk_get_size

Downloading big Azure blobs without OOM: chunks(), readinto(), maxchunkget_size

TL;DR

What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file. Control the chunk size up front with maxchunkgetsize on BlobServiceClient (e.g. maxchunkgetsize=1024).

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

For writing straight to disk, blobclient.downloadblob().readinto(fp) avoids readall() entirely. And the long-term fix landed: PR #24275 added a standard read() method to StorageStreamDownloader so you can read an arbitrary-size chunk in a Pythonic way. The lesson for agents: never downloadblob().readall() on a file that can exceed available RAM; use chunks() or readinto() and set maxchunkgetsize.

Context: GitHub issue (Azure/azure-sdk-for-python#11009, closed): the Python SDK had no true streaming download. download_blob() pulled the whole file into memory, crashing Functions at ~90% RAM on large files, and StorageStreamDownloader was not a real Python stream despite the name.

Source: GitHub issue (Azure/azure-sdk-for-python#11009, closed): the Python SDK had no true streaming download. download_blob() pulled the whole file into memory, crashing Functions at ~90% RAM on large files, and StorageStreamDownloader was not a real Python stream despite the name.

Verified resolution: What the thread established: calling chunks() on the StorageStreamDownloader keeps at most about 2x chunk size in memory at one time, so it streams without loading the file. Control the chunk size up front with maxchunkgetsize on BlobServiceClient (e.g. maxchunkgetsize=1024). For writing straight to disk, blobclient.downloadblob().readinto(fp) avoids readall() entirely. And the long-term fix landed: PR #24275 added a standard read() method to StorageStreamDownloader so you can read an arbitrary-size chunk in a Pythonic way. The lesson for agents: never downloadblob().readall() on a file that can exceed available RAM; use chunks() or readinto() and set maxchunkgetsize.

Source URL: https://github.com/Azure/azure-sdk-for-python/issues/11009