tabula read_pdf_with_template shifts columns on a scanned fee schedule

Not specified by the original author.

TL;DR

tabula's readpdfwith_template shifting columns on a scanned fee schedule happens because tabula reads text positions from the PDF - and a scanned PDF has no text positions, only images. The template's coordinates then map onto nothing, and columns drift. OCR the scan first (ocrmypdf adds a real, positioned text layer), then apply the same template to the OCR'd file. If columns still shift, re-measure the areas on the OCR'd version, because OCR can slightly change page geometry.

The query

tabula read_pdf_with_template shifts columns on a scanned fee schedule

Use this when

• tabula readpdfwith_template shifts or scrambles columns on a scanned schedule
• The template works on digital PDFs but fails on scanned ones
• Extracted columns contain text from neighboring columns

Not for

• Templates failing on digital PDFs - those need area re-measurement, not OCR
• tabula finding no tables at all - check the template areas and the PDF text layer first
• OCR'd PDFs where columns are right but rows are merged - a row-detection tuning problem

Steps

1. Prove the PDF has no text layer

Try selecting text in a PDF reader, or extract with a text tool. If nothing selects, tabula has no positions to work with and the template cannot function - this check takes ten seconds and prevents an hour of template tuning.

pdftotext scanned-fees.pdf - | head -5

Expected output: empty output, confirming the scan has no text layer.

2. OCR the scan to add a positioned text layer

Run ocrmypdf on the scanned file. The output is a PDF with the original images plus an invisible, positioned text layer - exactly what tabula needs.

ocrmypdf scanned-fees.pdf ocr-fees.pdf

Expected output: ocr-fees.pdf, in which text selects and pdftotext returns the fee rows.

3. Apply the template to the OCR'd file

Run readpdfwith_template against ocr-fees.pdf with your existing template. In most cases the columns now land correctly because the coordinates finally have text to attach to.

import tabula
tabula.read_pdf_with_template("ocr-fees.pdf", "template.json", pages="all", output_path="fees.csv")

Expected output: fees.csv with columns aligned to the visible schedule grid.

4. Re-measure areas if columns still drift

OCR can deskew and slightly rescale pages, moving content relative to the template's coordinates. Open the OCR'd PDF in tabula's template designer (or measure with pdfplumber) and adjust the areas, then re-run. Expected output: the adjusted template produces clean columns on the OCR'd file.

5. Bake OCR into the pipeline before tabula

Make the OCR step unconditional for scans: detect text-layer presence at ingest, OCR when absent, then run tabula. Never let a raw scan reach the template step again. Expected output: scanned schedules flow through the same template path as digital ones.

Tool compatibility

tabula-py with a Java runtime; ocrmypdf for the OCR step; pdftotext (poppler) for the text-layer check. The template JSON format is unchanged - only the input PDF changes.

Variant phrasings

• tabula readpdfwith_template shifts columns scanned fee schedule
• tabula template wrong columns scanned PDF
• tabula OCR PDF before template extraction
• tabula columns drift scanned schedule no text layer

Why it happens

tabula is a positional extractor: it places the template's rectangles onto the PDF's text objects and reads what falls inside. A scanned PDF contains images, not text objects, so the rectangles land on empty space and tabula grabs whatever fragments are nearest - producing shifted, interleaved columns. OCR inserts the missing text objects with real coordinates, which is the only input tabula's algorithm can consume.

Edge cases / pitfalls

• OCR quality bounds extraction quality: a bad OCR pass gives well-positioned garbage - check the OCR text first.
• Handwritten annotations on the scan become text objects too and can pollute columns; mask or accept them.
• Multi-page scans need per-page OCR; a single bad page's geometry can throw off a shared template.
• Keep the original scan archived - the OCR'd file is a working copy, and re-OCRing with better settings later is normal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstt-LkhAgHu049qL7FkIu1w

Resolved from the public thread: https://vectle.com/posts/pstt-LkhAgHu049qL7FkIu1w

Thread query: tabula readpdfwith_template shifts columns on a scanned fee schedule

TL;DR: tabula's readpdfwith_template shifting columns on a scanned fee schedule happens because tabula reads text positions from the PDF - and a scanned PDF has no text positions, only images. The template's coordinates then map onto nothing, and columns drift. OCR the scan first (ocrmypdf adds a real, positioned text layer), then apply the same template to the OCR'd file. If columns still shift, re-measure the areas on the OCR'd version, because OCR can slightly change page geometry.

Not specified by the original author.

Tabula and camelot misalign rows on PDF table extraction

TL;DR

Misaligned rows from tabula and camelot mean the table's visual grid does not match the extractor's assumptions: multi-line cells, missing ruling lines, or merged headers shift everything. The fix is picking the right detection mode for the table's actual structure and post-processing the dataframe, not endlessly tuning one mode. When both tools misalign the same table, the table is genuinely irregular and needs rule-based cleanup.

The error

(misaligned output, no exception)
tabula/camelot rows shifted: values land in wrong columns, header merged into data rows

When this helps

• tabula or camelot return misaligned rows
• values land in wrong columns after extraction
• troubleshooting table extraction on a new report
• deciding between lattice and stream modes

When it doesn't

• the PDF is scanned; OCR first, then extract
• tables span pages; stitch fragments before aligning
• the table is an image inside the PDF; extraction cannot see it

Works with

python 3.8+ with camelot-py (ghostscript for lattice) or tabula-py (java). Behavior varies by PDF producer.

Steps

1. Compare lattice versus stream on the same page

import camelot
for flavor in ["lattice", "stream"]:
    t = camelot.read_pdf("report.pdf", pages="5", flavor=flavor)
    print(flavor, t.n, "tables")
    if t.n:
        print(t[0].df.shape)

Expected: Shapes from both flavors. The flavor whose shape matches the visible table is the right starting point.

2. Inspect where the misalignment starts

import camelot
t = camelot.read_pdf("report.pdf", pages="5", flavor="stream")
df = t[0].df
for i, row in df.head(8).iterrows():
    print(i, list(row)[:4])

Expected: The first shifted row. Misalignment usually starts at a multi-line cell or a merged header; everything after shifts.

3. Repair with column-aware cleanup rules

import camelot, re
df = camelot.read_pdf("report.pdf", pages="5", flavor="stream")[0].df
df = df[df.apply(lambda r: sum(1 for c in r if str(c).strip()) != 0, axis=1)]
print("non-empty rows:", len(df))
print("cols:", df.shape[1])

Expected: A dataframe with empty artifact rows removed. Row filtering by non-empty cell count fixes the most common shift source.

4. Validate against a known total from the table

import camelot
df = camelot.read_pdf("report.pdf", pages="5", flavor="stream")[0].df
txt = df.to_string()
print("mentions total:", "total" in txt.lower())
print("row count:", len(df))

Expected: Confirmation the totals row survived. If the total row is missing or shifted, the alignment is still wrong.

Other ways people phrase this

camelot rows misaligned pdf table

Usually multi-line cells or merged headers. Stream plus row cleanup handles most cases.

tabula columns shifted extraction

Tabula's lattice mode invents columns from ruling lines; stream mode follows whitespace. Try both.

pdf table values wrong columns

The alignment half of table extraction. Detect the shift start, then clean.

Why it happens

Extractors assume tables are regular grids, but real financial tables have multi-line cells, merged headers, and inconsistent ruling. One irregular row shifts every row after it in stream mode, while lattice mode hallucinates gridlines from stray rules. The tools are not broken; the table violates their assumptions and needs cleanup rules.

Edge cases

• Merged header cells never align cleanly; extract headers separately from body rows.
• Currency symbols in their own column defeat whitespace detection; strip them before aligning.
• Tables with both ruled and unruled sections need per-section settings.
• When both flavors fail identically, the table is irregular enough to warrant manual mapping.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fdhQuTh-K8MLmwnXgOELcg

Resolved from the public thread: https://vectle.com/posts/pst_fdhQuTh-K8MLmwnXgOELcg

Thread query: tabula camelot pdf table extraction misaligned rows, troubleshooting

TL;DR: Misaligned rows from tabula and camelot mean the table's visual grid does not match the extractor's assumptions: multi-line cells, missing ruling lines, or merged headers shift everything. The fix is picking the right detection mode for the table's actual structure and post-processing the dataframe, not endlessly tuning one mode. When both tools misalign the same table, the table is genuinely irregular and needs rule-based cleanup.

Not specified by the original author.

TL;DR

camelot's readpdf returning empty tables on a bordered fee schedule means its lattice mode could not detect the table's line grid: the borders are too thin, broken, or drawn as background shading rather than real lines. Switch to flavor='stream' (which uses text alignment instead of lines), tune linescale if you stay on lattice, or pin table_areas to the fee schedule's coordinates. If neither flavor finds it, the 'borders' are decorative - fall back to pdfplumber's text-based extraction.

The query

camelot read_pdf returns empty tables on a bordered fee schedule  -  how to fix

Use this when

• camelot read_pdf returns an empty list on a PDF with a visibly bordered table
• Lattice mode finds tables on some pages but not on the fee schedule page
• The borders look light gray or hairline in the PDF

Not for

• Tables found but with wrong cell contents - that is a parsing-accuracy problem, not detection
• Scanned PDFs with no text layer - camelot needs real text; OCR the scan first
• Stream mode finding tables but merging columns - tune columns or use table_areas

Steps

1. Check what the 'borders' actually are

Zoom into the PDF at high magnification: real vector lines stay crisp, while hairline borders drawn as thin rectangles or background shading break lattice detection. Also confirm the PDF has selectable text - a scanned fee schedule defeats both flavors until OCR'd. Expected output: you know whether the grid is vector lines, shading, or a scanned image.

2. Try stream flavor

Stream mode ignores lines and groups text by alignment, which handles shaded or broken borders well. It is the right first experiment because it needs no tuning.

import camelot
tables = camelot.read_pdf("fees.pdf", flavor="stream", pages="1")
print(tables.n)

Expected output: tables.n is 1 or more, with the fee rows roughly right.

3. Tune lattice before giving up on it

If the grid is real but thin, lower linescale so fainter lines count, and try processbackground to catch lines drawn behind shading. Adjust one parameter at a time and re-check the table count.

tables = camelot.read_pdf("fees.pdf", flavor="lattice", line_scale=30, process_background=True)
print(tables.n)

Expected output: the bordered schedule is detected with cells aligned to the visible grid.

4. Pin the table area explicitly

When detection is the problem but parsing is fine, give camelot the coordinates: measure the table's bounding box (any PDF viewer with coordinates, or pdfplumber's debugging) and pass table_areas. This skips detection entirely.

tables = camelot.read_pdf("fees.pdf", flavor="lattice", table_areas=["50,700,550,100"])
print(tables[0].df.head())

Expected output: the fee schedule DataFrame with the expected rows and columns.

5. Fall back to pdfplumber for decorative borders

If the borders are shading rather than lines, neither flavor is reliable. pdfplumber's extract_table with explicit line strategies, or plain text extraction with column splitting, handles these schedules more predictably. Expected output: a correct table from the fallback, proving the issue was detection, not the document.

Tool compatibility

camelot-py with either the PyMuPDF or pdfminer backend; pdfplumber as the fallback. Applies to text-based PDFs - scanned schedules need OCR first regardless of flavor.

Variant phrasings

• camelot read_pdf returns empty tables bordered fee schedule
• camelot lattice no tables found thin borders
• camelot flavor stream vs lattice empty result
• camelot table_areas bordered table not detected

Why it happens

Lattice mode works by detecting line segments and intersecting them into a grid; it is precise but fragile, because it depends on the PDF's drawing commands containing real lines. Many 'bordered' schedules use cell shading or hairline rules that do not register as lines at default sensitivity. Stream mode uses a completely different signal (whitespace between text columns), so it succeeds exactly where lattice fails.

Edge cases / pitfalls

• line_scale is relative to the page's line lengths - the right value varies per document; there is no universal setting.
• Multi-page fee schedules need per-page areas; one table_areas box rarely fits all pages.
• Rotated or slightly skewed pages break both flavors - deskew or rotate before extraction.
• Always inspect tables[0].df visually once; a detected table with shifted columns is worse than no table.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_42naGZxbBZpNQkEJZbjfFA

Resolved from the public thread: https://vectle.com/posts/pst_42naGZxbBZpNQkEJZbjfFA

Thread query: camelot read_pdf returns empty tables on a bordered fee schedule - how to fix

TL;DR: camelot's readpdf returning empty tables on a bordered fee schedule means its lattice mode could not detect the table's line grid: the borders are too thin, broken, or drawn as background shading rather than real lines. Switch to flavor='stream' (which uses text alignment instead of lines), tune linescale if you stay on lattice, or pin table_areas to the fee schedule's coordinates. If neither flavor finds it, the 'borders' are decorative - fall back to pdfplumber's text-based extraction.