Skip to content
terbium

terbium / Guides

File formats

How terbium reads PDF, PPTX, XLSX, CSV and image files into one normalized view, and how to add an adapter for a new format.

On this page
  1. PDF
  2. Beyond matrices
  3. PPTX
  4. XLSX
  5. CSV
  6. Images
  7. Adding a format

terbium ships adapters for Portable Document Format (PDF), PowerPoint (PPTX), Excel (XLSX), comma-separated values (CSV) and image files. Each one turns bytes on disk into the same normalized shape: pages holding positioned Words, ImageRefs and, where the format has one, native tables. Everything after that works on this uniform view.

Format Extensions Read with
PDF pdf PyMuPDF, word geometry
PPTX pptx python-pptx, native structure
XLSX xlsx, xlsm openpyxl, native cells
CSV csv, tsv Python csv module
Images png, jpg, jpeg, webp, tiff, tif Pillow and Tesseract

PDF gets the full geometry engine because a PDF throws its structure away. PPTX, XLSX and CSV already carry structure, so terbium reads it directly. The reasoning is laid out on How it works.

For the reverse question, which kinds of documents exist and which lane reads each, see the document taxonomy.

PDF

The hard case, and where the geometry engine runs in full.

  1. Word geometry. PyMuPDF gives text spans with boxes and font sizes; terbium splits them into positioned tokens, placing each token by its character offset inside the span. Private-use icon glyphs are stripped from the text.
  2. De-heading. Text that repeats in the same band near the top or bottom edge across many pages is a running header or footer, and is stripped. Mid-page repeats, such as an axis label like “length x width x height”, are deliberately kept.
  3. Spread splitting. A page more than 1.7 times as wide as it is tall, with an empty vertical gutter near its centre, is a two-page spread. terbium cuts it there, unless content genuinely crosses the gutter, so the left product does not merge with the right one.
  4. Table reconstruction (content-agnostic). A run of consecutive lines that share the same column structure is a table. Column anchors come from the x-positions of the cells, the first row becomes the header, and every cell is placed by x-alignment. The detector makes no assumption about what the cells contain, so a financial table, a spec sheet and a furniture matrix reconstruct the same way. Multi-column prose is rejected, because its cells are sentence fragments rather than data. The furniture schema swaps in a specialized cross-tab detector for size by finish matrices.

Beyond matrices

Not every PDF is a matrix.

  • Label grids. A lookbook is a grid of product photos with a name under each. On any page that produced no table and carries a photo (or at least two images), terbium segments each text row into separate labels, stitches wrapped names back together, and groups them under the page’s collection title. The decision is made page by page, not for the whole document.
  • Text-only tables on photo pages. On a page with three or more images, a detected “table” in which fewer than 15% of the cells contain a digit is treated as a label grid rather than a data table.
  • Image-only pages. A page with images and fewer than five words that yields nothing cannot be read algorithmically. terbium records these pages and reports them, so an image catalogue returns an honest “these pages need the vision lane” message instead of nothing.
  • Optical character recognition (OCR). When Tesseract is installed, parse and build_catalog run it by default (ocr="auto") on pages that have images and fewer than three words, at 300 dots per inch (DPI), and feed the recognized words through the same pipeline.

PPTX

python-pptx exposes shapes with positions, native tables (rows and cells, no reconstruction needed) and pictures with real pixels. terbium passes native tables straight through, titled with the slide title, and tokenizes text boxes into positioned words. Those words feed classification and the catalog lane’s naming, but geometric table reconstruction runs only on PDF pages, so a table drawn with loose text boxes on a slide is not rebuilt.

Only shapes placed directly on the slide are read. Shapes inside a group, and pictures inserted into a picture placeholder, are skipped. Ungroup them in PowerPoint if they matter.

For product decks, build_catalog treats each picture as a row anchor. See The catalog table.

XLSX

openpyxl reads cell values (not formulas) and merged ranges. A merged cell’s value is copied across its whole range so multi-column headers survive. Each sheet becomes one page holding one native table, titled with the sheet name, with the first non-empty row as the header. Reading stops at 5,000 rows and 200 columns per sheet.

The values are the ones Excel stored the last time it calculated the sheet. A file written by a script may have no stored values, so its formula cells come out empty. Open and save it in Excel or LibreOffice first.

CSV

The delimiter (comma, semicolon, tab or pipe) and whether a header row exists are sniffed from the first 8 kilobytes (KB). If sniffing fails, terbium assumes a comma, or a tab for a .tsv file. Without a header row, the columns are named col1, col2 and so on. The file is decoded as UTF-8 (with or without a byte order mark), falling back to Latin-1. Values become a single native table with no title, so a complete CSV table scores 0.88 rather than 0.98. Reading stops at 20,000 rows. Cell values stay strings; the CSV adapter does not infer column types.

Images

A standalone image file becomes a single page holding one image. Words are synthesized with the local Tesseract bridge when Tesseract is installed. They feed classification and the transaction and resume lanes; table reconstruction does not run on image files. See Invoices, receipts and resumes.

Adding a format

Subclass terbium.documents.base.DocumentAdapter, set extensions, implement parse(path) returning a list of Page objects, and decorate the class with @register. If your format has native tables, attach them to Page.native_tables and terbium skips geometry reconstruction for that page.

from typing import List

from terbium.documents.base import DocumentAdapter, register
from terbium.model.elements import Page


@register
class MyAdapter(DocumentAdapter):
    extensions = ("myext",)

    def parse(self, path: str) -> List[Page]:
        ...

The adapter must be imported before you call terbium.parse, because registration happens when the class is defined.

Set source_kind on each Page to your format’s name. The default is "pdf", which turns on the PDF-only steps (geometric reconstruction, label grids and OCR), and the OCR step opens the original file as a PDF.