unstructured partition_pdf times out on a 300-page scanned MSA - how to fix

TL;DR

unstructured's partitionpdf timing out on a 300-page scanned MSA is the hires strategy running a layout-detection model on every page - accurate but far too slow at that scale. Switch to strategy='fast' or 'ocronly', process the document in page batches, and raise the client timeout. For a one-off giant file, batching with the fast strategy finishes in minutes; reserve hires for the pages where layout actually matters (tables, complex exhibits).

The query

unstructured partition_pdf times out on a 300-page scanned MSA  -  how to fix

Use this when

• partition_pdf times out or hangs on a large scanned PDF
• Processing a 300-page MSA takes far longer than the timeout allows
• You need text out of the whole document, not perfect layout of every page

Not for

• partition_pdf failing fast with an import or configuration error - those are setup problems
• Timeouts on small digital PDFs - something else is wrong; check the file
• Poor extraction quality with the fast strategy - that is the expected tradeoff, tune from there

Steps

1. Switch to the fast strategy first

The single biggest lever: strategy='fast' skips the layout model and extracts text directly (with OCR for scanned pages), which is orders of magnitude quicker per page.

from unstructured.partition.pdf import partition_pdf
elements = partition_pdf("msa.pdf", strategy="fast")
print(len(elements))

Expected output: the call returns instead of timing out, with one element per text block.

2. Process in page batches

Split the 300 pages into batches (50 pages each works well) and partition each batch separately, then concatenate the element lists. Batching bounds memory, makes progress resumable, and isolates a poison page to one batch.

batches = [(1, 50), (51, 100), (101, 150), (151, 200), (201, 250), (251, 300)]
all_elements = []
for start, end in batches:
    all_elements += partition_pdf("msa.pdf", strategy="fast", pages=list(range(start, end + 1)))

Expected output: six quick calls and a complete element list, with any single-batch failure retryable alone.

3. Raise the timeout as a backstop, not the fix

If your framework wraps the call with a timeout, raise it - but only after switching strategy and batching. A huge timeout on hi_res just schedules the same failure later. Expected output: the timeout stops firing because the work now fits comfortably inside it.

4. Reserve hi_res for the pages that need it

Run the fast strategy for the whole document, then re-run hires only on pages with tables or complex exhibits (detect them by page number or by scanning fast output for table-like content). You get layout accuracy where it matters without paying for it on 280 pages of prose. Expected output: a hybrid result - fast elements everywhere, hires elements for the complex pages.

Tool compatibility

unstructured with the pdf partition extra; hi_res additionally needs the layout model dependencies (detectron2 / unstructured-inference). The fast strategy works with just pdfminer plus an OCR backend for scans.

Variant phrasings

• unstructured partition_pdf times out 300 page scanned MSA
• partitionpdf hires too slow large PDF
• unstructured pdf strategy fast vs hi_res timeout
• partition_pdf batch pages large document

Why it happens

hi_res runs a neural layout-detection model per page to classify titles, tables, and text blocks before extracting - that model is the dominant cost, and it scales linearly with pages. On a 300-page scanned MSA the OCR pass plus 300 layout inferences exceeds any sane timeout. The fast strategy skips layout inference entirely, trading structural labels for speed, which is the right trade for prose-heavy contracts.

Edge cases / pitfalls

• Batch boundaries can split a table across batches - overlap batches by a page or reassemble split tables afterward.
• Element IDs may collide across batches; re-key them when concatenating if your downstream store requires uniqueness.
• The fast strategy's table output is weaker - that is precisely what the hi_res second pass is for.
• Memory still grows with total elements; for thousand-page portfolios, stream batches to disk instead of one list.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstZNbcbECchh3o-pGxyaJzQ