Immutable solution version
Document Splitter always returns 1 document for split_type="passage" in pdfs
Complete solution
Haystack: Root cause identified by contributor lbux: PDF converters store paragraph breaks as single newlines, but split_by='passage' splits on double newlines (\n\n), so passage splitting yields one document. Contributor davidsbatista's fix: use PyPDFToDocument(extraction_mode=PyPDFExtractionMode.LAYOUT) - 'works perfectly!' - since layout mode preserves paragraph spacing.
The problem
Issue deepset-ai/haystack#8491 (closed, 22 comments): Describe the bug When using Document Splitter with pdf and split_type="passage", the result is always one document. This is using pypdf. Expected behavior The understanding I have is that it splits based on at least two line breaks \n\n Additional context When I tested using plain text it seems to be splitting correctly To Reproduce dir = '...' files = [ {"filename": "rules.pdf", "meta": {"splitby" : "passage", "splitlength":1, "splitoverlap":0, "splitthreshold":0}}, {"filename": "rules.txt", "meta": {"splitby" : "passage", "splitlength":1, "splitoverlap":0, "splitt
The verified fix
Root cause identified by contributor lbux: PDF converters store paragraph breaks as single newlines, but splitby='passage' splits on double newlines (\n\n), so passage splitting yields one document. Contributor davidsbatista's fix: use PyPDFToDocument(extractionmode=PyPDFExtractionMode.LAYOUT) - 'works perfectly!' - since layout mode preserves paragraph spacing. Also noted: running DocumentCleaner with removeemptylines=True before the splitter strips the newlines passage splitting needs.
Evidence
In gh:deepset-ai/haystack#8491 (https://github.com/deepset-ai/haystack/issues/8491), someone reported: Issue deepset-ai/haystack#8491 (closed, 22 comments): Describe the bug When using Document Splitter with pdf and split_type="passage", the result is always one document. This is using pypdf. Expected behavior The understanding I have is that it splits based on at least two line breaks \n\n Additional context When I tested using plain text it seems to be splitting correctly To Reproduce dir = '...' files = [ {"filename": "rules.pdf", "meta": {"splitby" : "passage", "splitlength":1, "splitoverlap":0, "splitthreshold":0}}, {"filename": "rules.txt", "meta": {"splitby" : "passage", "splitlength":1, "splitoverlap":0, "splitt The verified resolution from the thread: Root cause identified by contributor lbux: PDF converters store paragraph breaks as single newlines, but splitby='passage' splits on double newlines (\n\n), so passage splitting yields one document. Contributor davidsbatista's fix: use PyPDFToDocument(extractionmode=PyPDFExtractionMode.LAYOUT) - 'works perfectly!' - since layout mode preserves paragraph spacing. Also noted: running DocumentCleaner with removeemptylines=True before the splitter strips the newlines passage splitting needs.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_k0KZJjNmlqaJzRLe0qcvZA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Document Splitter always returns 1 document for split_type="passage" in pdfs <span>— Vectle</span></a>