Skip to content
terbium

terbium / Guides

Schemas

The built-in terbium schemas (product, generic, furniture, transaction, resume), how the default is chosen, and how to write your own.

On this page
  1. How the schema is chosen
  2. Built-in schemas
  3. product
  4. generic
  5. furniture
  6. transaction and resume
  7. Writing your own

A schema is the one place that knows what a row of output should look like. Everything upstream is domain-agnostic: swap the schema and keep the engine.

How the schema is chosen

Pass schema= to terbium.parse to choose one. When you leave it as None (the default), terbium picks it from the document type:

  • catalog, lookbook, table: product
  • transaction: transaction
  • resume: resume
  • deck, unknown: generic

With doc_type="auto" (the default) the type comes from auto-classification. On the command line, --schema applies to the records view.

The schema shapes records; it does not decide which page-level readers run. The transaction and resume readers that work from labels and headings run when the document type is transaction or resume, so pass doc_type= as well when you force one of those schemas on a document the classifier labelled differently.

Built-in schemas

product

Universal and category-agnostic. It maps each column header to a canonical product field by meaning, using synonym lists (FIELD_SYNONYMS in schema/product.py). The longest matching synonym wins, so “pack size” maps to quantity rather than dimensions.

Canonical fields: sku, name, price, dimensions, color, material, category, quantity, weight, brand.

The synonym lists are broad, so check the mapping on a new supplier’s file. For example, Description and Item map to name, Finish and Shade to color, Code and Ref to sku, Rate and Amount to price, and Type and Range to category.

  • A price cell is also split into price_amount and currency (from $, £, €, ₹, Rs, or a code such as USD).
  • A column with no canonical match keeps its own header as the field name, so a bag’s “capacity” or a lamp’s “wattage” survives as an attribute.
  • The table title becomes collection.
  • If no column is named like a stock keeping unit (SKU) code, the first compact code-like cell containing a digit is used, skipping price, quantity and weight columns.
  • Normalized keys are added alongside the raw values: color_family, material_family and dimensions_mm. See Normalization and feeds.
doc = terbium.parse("stock.xlsx")        # classified as "table", so schema "product"
doc.records[0].fields
# {"sku": "35155", "name": "Sequence table", "quantity": "12", "collection": "Stock"}

generic

Works for any table without domain knowledge.

  • Grid tables (Excel, comma-separated values (CSV), PowerPoint, and tables rebuilt from a Portable Document Format (PDF) page): one record per body row; columns become fields named after their headers. A compact, digit-bearing code of 3 to 16 characters in the row becomes the sku. The table title is added as title, and any shared table attributes are copied in.
  • Matrix and list tables (from the furniture detector): one record per filled cell, with row, column and title fields and the cell value as the sku.
doc = terbium.parse("stock.xlsx", schema="generic")
doc.records[0].fields
# {"Article": "35155", "Product": "Sequence table", "Qty": "12", "title": "Stock"}

furniture

One record per article number, tuned for a catalogue size by finish matrix. Choosing it also switches PDF reconstruction to the specialized cross-tab detector.

doc = terbium.parse("catalogue.pdf", schema="furniture")
doc.records[0].fields
# {"product": "Barrow pouf", "size": "40 × 40 × 40 cm", "finish": "off white",
#  "dimensions_cm": "40 x 40 x 40 cm", "dimensions_in": '15.5" x 15.5" x 15.5"'}

When the table carries them, composition and height are added too. The repository’s examples/furniture_catalogue.py runs this schema on a catalogue PDF you supply.

transaction and resume

transaction reads invoices, bills, receipts, purchase orders and quotes into summary, line_item and total records. resume reads a curriculum vitae (CV) into a header record and one record per line under each section; rows from a table become records with section set to table. Both are described in Invoices, receipts and resumes.

Writing your own

Subclass terbium.schema.Schema, give it a name, implement build_records(tables) and register it:

import terbium
from terbium.model.record import Record
from terbium.schema import Schema, register_schema


@register_schema
class PricelistSchema(Schema):
    name = "pricelist"

    def build_records(self, tables):
        records = []
        for t in tables:
            for rh, ch, val in t.iter_cells():
                records.append(Record(
                    sku=val, fields={"item": t.title, "variant": ch},
                    source_page=t.source_page, confidence=t.confidence,
                    reasons=list(t.reasons),
                ))
        return records


doc = terbium.parse("prices.pdf", schema="pricelist")

build_records receives ExtractedTable objects. Each has title, row_headers, col_headers, cells (cells[row][col]), source_page (0-based), kind (matrix, grid or list), attributes, confidence, reasons and origin. iter_cells() yields (row_header, col_header, value) for every non-empty cell.

Lookbook label grids do not pass through the schema. Whatever schema you choose, each label becomes a record with a name field and, when the page has a collection title, a collection field. These records have a fixed confidence of 0.82.

You can also pass a Schema instance directly: terbium.parse(path, schema=PricelistSchema()). An unknown schema name raises ValueError listing the registered names.