Skip to content
terbium

terbium / Project

Evaluation

What the terbium repository measures: the category benchmark in eval/, the test suite, and reported results on real vendor catalogues.

On this page
  1. Category benchmark (reproducible)
  2. Real vendor catalogues (reported, not reproducible from the repository)
  3. The worked example document
  4. Test suite

This page reports only numbers that exist in the repository and says where each one comes from.

A parser’s accuracy depends heavily on the documents it is given, so a number means little without its inputs. Each result below therefore names its inputs, whether you can reproduce it, and what it does not show.

Category benchmark (reproducible)

eval/category_bench.py generates a representative lookbook in Portable Document Format (PDF) for each of six product categories, runs build_catalog on it with no artificial intelligence (AI) key, and scores recall of name, stock keeping unit (SKU), materials and image. Each PDF has four real embedded photos laid out in a grid, with the product name directly beneath each photo and SKU: and Material: (or Ingredients:) lines below that. The PDFs are synthetic but follow the structure of a real vendor lookbook. The script passes only_photos=False to the image extractor.

Run it from a clone:

pip install -e .
python eval/category_bench.py

Output from a run of 0.10.0 (Python 3.12, September 2026), matching the table in the old docs. Re-runs with Python 3.12 and PyMuPDF 2.0, of 0.10.0 on 25 October 2026 and of 2.0.0 on 26 October 2026, printed the same table and layout checks:

category   products   name    sku  material  image
----------------------------------------------------
rugs              4      4      4         4      4
lamps             4      4      4         4      4
bags              4      4      4         4      4
handbags          4      4      4         4      4
cushions          4      4      4         4      4
handwash          4      4      4         4      4
----------------------------------------------------
TOTAL            24     24     24        24     24

Every field was recovered for every product: 24 of 24 names, SKUs, materials and images.

The same script then runs three layout checks and asserts a minimum row count:

  • dense_grid: 12 products on one page. The run produced 12 rows (expected at least 12).
  • sparse_photo: one product on one page. The run produced 1 row (expected at least 1).
  • mixed_layout: a six-product grid on page 1 and one product on page 2. The run produced 7 rows (expected at least 2).

This measures the deterministic engine on clean lookbook layouts. Messy real-world brochures are where the AI pass is meant to help.

Real vendor catalogues (reported, not reproducible from the repository)

The old documentation (docs/catalog.md) reports a run of build_catalog, with no AI key, on six vendor catalogue PDFs downloaded from the open web, one per category. The PDFs are not in the repository, the script used is not in the repository, and the terbium version used is not recorded, so these numbers cannot be reproduced from the repository. They are carried over as reported.

1,010 products in total. Per category (products; then how many had a name, image, SKU and materials):

  • Furniture (a furniture retailer): 70 products; name 70, image 70, SKU 10, materials 18
  • Rugs (The Rug Furnish): 124 products; name 97, image 124, SKU 103, materials 41
  • Lamps (Sahil & Sarthak, 227 pages): 484 products; name 415, image 484, SKU 229, materials 191
  • Cushions (Domkapa): 121 products; name 40, image 121, SKU 8, materials 66
  • Handbags (Zoosch): 183 products; name 108, image 183, SKU 59, materials 112
  • Handwash (English Soap Co): 28 products; name 0, image 28, SKU 0, materials 0
  • Total: 1,010 products; name 730, image 1,010, SKU 409, materials 428

What the old docs conclude from it:

  • Image extraction recovered a photo for 1,010 of 1,010 products.
  • Deterministically, name coverage was 72%, SKU 40% and materials 42%. Coverage was strong where the catalogue prints them beside the photo (lamps with codes like SSVSNL26, rugs with MRP-962) and thin where it does not. A SKU count below the product count usually means the catalogue does not print a per-product code.
  • The handwash brochure is the floor: images yes, text none. Its names and ingredients are set as stylised artwork rather than selectable text. That is the case the AI vision pass is designed for; no AI-assisted numbers are recorded in the repository.

The worked example document

The design notes (docs/DESIGN.md) and the old landing page describe the furniture catalogue terbium was first built against (an Ethnicraft 2026 catalogue): 192 pages, 964 unique SKUs, 505 dimension rows, 0 prices and 1,157 images (562 photos, 439 swatches and 156 material icons). Each product page is a two-dimensional matrix with sizes as rows, finishes as columns and article numbers in the cells. These figures describe the input document, not a parse result, and the PDF is not in the repository.

Test suite

tests/test_smoke.py holds 51 pytest tests in 2.0.0 (46 in 0.10.0). They cover SKU and dimension signals, matrix and content-agnostic table reconstruction, rejection of prose columns, label grids and image association, the product schema and normalization, the catalog table end to end, catalog escalation, AI fill and enrichment plumbing with stub providers, feed exporters, comma-separated values (CSV) round trips, classification of invoices and resumes, transaction and resume fields, the render helpers, receipts, dense and sparse catalogues, key-value forms, provider selection order, command line routing of an invoice, the transaction AI fill (filling summaries, skipping complete ones, not reporting failed calls), used_ai, terminal header truncation, the message shown without Tesseract, and the token parameter for OpenAI o-series models. The five tests added in 2.0.0 cover the PyMuPDF import without the fitz warning and its fallback to fitz, the dominant colour without the deprecated Pillow call, and the resume heading rule.

pip install -e ".[dev]"
python -m pytest -q

On 2.0.0 all 51 pass (Python 3.12 and PyMuPDF 2.0, 26 October 2026). On 0.10.0 all 46 passed (Python 3.12, September 2026; re-run on 25 October 2026 with PyMuPDF 2.0, still 46 passed). The repository has no continuous integration (CI) workflow, so tests only run when someone runs them locally. The tests use small generated files and stub providers; no test calls a real model.