Why PDF tables are hard to extract

PDF grew out of PostScript, a language for telling a printer where to put marks on paper. A page in a PDF is a content stream: a list of operators that set a font, set a text matrix, and then draw a string of glyphs at a position. There is an operator for showing text and an operator for drawing a line, but there is no operator that says “this is a table with five columns”. The format was designed so that a page looks identical everywhere, not so that a program could later understand it.

The standard does have an answer to this: tagged PDF, introduced with PDF 1.4 and required by the PDF/UA accessibility standard, stores a logical structure tree with real table, row and cell elements. In practice you almost never meet it. Exports from accounting systems, bank statements, scientific papers and government forms are overwhelmingly untagged, because tagging costs the producer effort and nothing visible is lost by skipping it.

So a table in a typical PDF is a table only in the eye of the reader. The columns exist because the numbers happen to start at the same horizontal position; the rows exist because the baselines happen to line up. Every extraction tool has to rediscover that arrangement from scratch.

When a PDF library reads a page it does not get words or lines. It gets text runs: a run ends wherever the producer changed the font, changed the size, applied kerning, or simply chose to emit a new instruction. A single printed word can arrive as three runs, and a whole table row can arrive as one run with wide spacing baked into the positioning array rather than into actual space characters.

Worse, the order of those runs is drawing order, not reading order. A typesetter that fills the first column, then the second, then the third, produces a stream in which the whole of column one precedes the whole of column two. That is exactly why copy and paste from a PDF reader so often yields every value of the first column, then every value of the second — technically faithful to the file, useless to you.

Readers apply heuristics to hide this, and their heuristics are tuned for prose. They are good at reconstructing a paragraph and poor at reconstructing a grid, because a paragraph only needs the right order while a grid needs two coordinates for every piece of text.

Extraction tools have settled on two approaches, named by the Tabula and Camelot projects. The lattice strategy looks for the ruling lines actually drawn on the page — the vector strokes that form a grid — and uses their intersections as cell boundaries. It is exact when the lines are there, which makes it the right choice for forms, invoices from ERP systems and most government documents.

The stream strategy ignores graphics entirely and works from the text alone: rows are groups of characters sharing a baseline, columns are vertical corridors of the page where no character ever appears. It is the only option for the many tables that are drawn with nothing but alignment — annual reports, price lists, scientific tables with a single rule under the header.

Neither is universally better. A lattice tool fails on a table with no lines; a stream tool fails on a dense table whose columns are packed tight enough that no corridor is ever fully empty. TableExtractor implements the stream strategy and then lets you fix the result, which covers the awkward middle ground where a tool that only guesses would be wrong and you would have no way to say so.

For every text run, pdf.js reports a transformation matrix, an advance width and a height. Multiply the run's matrix by the page viewport matrix and you get its position in page space: points from the top-left corner, with any page rotation already applied. From the same matrix you get the effective font size, which is the natural unit for every threshold that follows — a tolerance of four points means something quite different on a 6 pt footnote and a 14 pt heading.

Rows come first. Sort the runs by baseline, then walk down the page and start a new row whenever the next baseline is further from the current row's mean than the row tolerance. Superscripts, footnote markers and slightly raised currency symbols are what that tolerance is for; set it too low and a single printed line splits in two, set it too high and two printed lines merge.

Columns come from a projection. Walk along the x axis in one-point bins and mark every bin covered by any run; the unmarked stretches are the corridors between columns. Every corridor at least as wide as the column-gap threshold becomes a cut line, placed in the middle of the gap. Assign each run to the cell whose column and row contain its centre, join the runs of a cell in reading order, and the grid is a table.

Four failures account for most bad results. A column that is empty for most of its rows leaves a corridor where the software sees no column at all — add the line by hand. A heading that spans two columns bridges the corridor and merges them — raise the column-gap threshold, or set the columns on a body row and leave the header to be edited afterwards. Centred text in wide cells wanders across the corridor; left- or right-aligned numbers rarely do.

The fourth is vertical: a cell whose content wraps over two printed lines is seen as two rows. Delete the row line between them and the two lines join into one value, separated by a space. Rotated column headers, common in wide scientific tables, are skipped entirely — they are not axis-aligned, so no whitespace projection can place them, and they are quicker to type than to rescue.

The practical method is the same every time: let the detector do the first pass, look at the lines it drew over the page, and fix what is wrong before you look at the text. Correcting geometry takes seconds; correcting a thousand misplaced cells in a spreadsheet does not.