A design extractor makes a tempting promise: give it a public Uniform Resource Locator (URL), get a file that helps an agent reproduce the look and feel. The output is useful even when it is imperfect. It can save time identifying colors, type, spacing, and recurring components. It can turn a page that exists only in a browser into a starting point for implementation.
The difficult question is not whether this can be useful. It is what the file is faithful to. A website is not a single style guide. It is rendered HyperText Markup Language (HTML), Cascading Style Sheets (CSS) variables, loaded fonts, responsive rules, stateful components, experiments, image assets, and choices that may be visible only after an interaction. A screenshot contains less than the site. Computed styles contain values but not necessarily intent. A large language model (LLM) can explain a system but may infer roles that were never encoded.
Our September 2026 research survey found no canonical fidelity benchmark for any of these paths. No surveyed extractor publishes an accuracy number against a known ground-truth design system. An agent receiving a generated file cannot tell whether a token set is close to source truth, a credible interpretation, or a well-written approximation.
The caveats are already in the market
Several products already say enough to show they understand the problem. Refero Styles is the clearest example. Its Brex reference page says that source measurements are normalized, roles and recommendations are interpreted, font summary lists are independent rather than position-paired, and HTML examples are reconstructions rather than source components. That is unusually useful language. It does not pretend a style reference is a source repository.
getdesign.md says its catalog entries are “reference material, not exact reproductions” and not official analyses from the named companies. Design.md Store calls its specs independent, unofficial works inspired by public websites. designmd.supply has the literal generation prompt in source and tells the model to use extracted styleguide data as the primary source, screenshots and page Markdown as supporting evidence, and conservative prose for missing data. Design Extractor documents the format but did not expose a real public extraction output in the research.
These statements are honest. They are not an evaluation. “Interpreted” tells a user what sort of document they have. It does not tell them whether the main color is right, whether the font is identified correctly, whether component radii are consistent, or whether the file covers mobile states.
This distinction matters because extraction outputs look authoritative. They arrive with precise hexadecimal color values, named tokens, tables, and component prescriptions. Precision in the document can be mistaken for precision in the measurement. A line such as radius: 8px is either a confirmed source value or an estimate that happens to be expressed in pixels. A reader needs a way to tell which.
What is the ground truth?
There is no single answer, which is why a benchmark needs a carefully limited scope. The most defensible benchmark set would use design systems whose source values are available publicly and whose license permits the test. It should not begin by reverse-engineering a commercial site and then call an internal guess “truth.”
A useful corpus could contain open-source projects with a public CSS token file, published component source, or an explicit first-party design file. Each test case would define what counts as ground truth for a category. Colors might come from a CSS custom-property file. Typography might come from loaded CSS declarations and a public font configuration. Spacing and radii might come from named tokens or component source. Component behavior could be limited to visible static states if interactive states cannot be recorded fairly.
This limits the benchmark, but that is a feature. A benchmark should measure something that can be checked. It should say, for example, “resolved color values matched the canonical token file” rather than “the result feels like the product.” The latter is a valid design review question. It is not a reproducible extraction metric.
The research already shows why system boundaries matter. CollectUI is a browsable screenshot gallery with no structured export, no application programming interface (API), and no agent-facing interface. It can help a person find visual references; it cannot by itself prove a token value. INSANE0777/Awwards-mcp is a live Model Context Protocol (MCP) tool for screenshot search, page capture, and “design DNA” extraction, not a static dataset. A page image and a source token are different kinds of evidence.
Accuracy has more than one axis
An extraction score should not collapse the whole task into one percentage. A number can conceal whether a file got colors right but invented component behavior, or found the font but missed the responsive scale. The output should report separate measures.
Color coverage could compare declared values and aliases to canonical values. Typography could compare font family, weight, size, line-height, and letter spacing when those are exposed in the target source. Layout could compare named spacing and breakpoint values where an authoritative token source exists. Shape and elevation could compare radii, border values, and shadow declarations. Components could be scored against a limited checklist: whether a documented button, input, or card is described using values found in canonical source.
Each axis needs a distinction between exact retrieval and interpretation. Extracting a CSS custom property is a different operation from looking at a rendered button and concluding that its role is “primary.” The first can be tested mechanically. The second should be labelled as interpretation and assessed by a reviewed rubric, if it is assessed at all.
Coverage matters as much as precision. An extractor can get five colors right and omit the rest of a system. It can identify a body font and miss the display face used in the most visible parts of a product. It can output a polished document with no evidence about hover, focus, error, or disabled states. A file with a high precision score on a narrow subset should state the subset.
Confidence should therefore be a field, not a tone. A section-level label such as high, medium, or low lets a generator say that colors came from Document Object Model (DOM) inspection while components were inferred from screenshots. A numeric score can be useful only if its calculation is published. Otherwise it has the same authority problem as an unlabeled extraction.
A proposed benchmark procedure
The procedure can be simple enough to inspect.
- Select the corpus. Use a documented set of public, licensed design systems. Pin the commit or release used as ground truth.
- Define the target. Name the pages and viewports. A tool that processes a single marketing page should not be evaluated as if it analyzed a whole application.
- Run each extractor as a user would. Use its normal public workflow. Record version, date, input URL, settings, and any paid tier or manual intervention.
- Normalize the output. Convert each generated file into a common comparison record. This is one reason token fields shaped like the Design Tokens Community Group (DTCG) format matter: a color, dimension, font family, or alias can be compared without writing a special parser for every frontmatter convention. A Markdown component description is judged separately, not forced into a false numeric match.
- Publish the evidence. Release the raw generated artifact, the source artifact, the comparison method, and per-axis results. A leaderboard should link to the evidence, not publish only a rank. A reader should be able to see why a tool lost points and decide whether that error matters for their use case.
- Rerun when products change. Extraction is a moving target. designmd.me had 15 dated changelog entries from 2026-04-15 through 2026-09-14 in the research. An evaluation from an old version does not describe the current one. Freshness belongs in the result alongside score.
The incentives are wrong without a benchmark
Current product incentives reward the appearance of a complete answer. More named tokens, richer prose, a rendered preview, or a Figma import can make an output feel finished. designmd.me offers single-page and multi-page generation, an optional HTML preview, and Figma import. Design Extractor offers web, extension, and MCP paths. These are useful distribution choices. They do not tell a user whether the generated artifact is closer to source than a simpler tool.
The absence of a benchmark also makes comparison hard for buyers. Exact pricing is opaque in parts of the category: Design Extractor’s credit packs did not expose dollar prices in fetched static HTML; Design.md Store’s /pricing page returned 404; TypeUI’s Pro price was unavailable behind a bot checkpoint; Refero displayed inconsistent pricing widgets in the same research session. A user asked to buy “brand accuracy” often cannot inspect either price or accuracy before committing.
Publishing weak results would be uncomfortable. A tool could show that it reliably finds colors but performs poorly on components. That would still be useful information. It would let a team select the tool for token discovery and keep component work manual. Honest boundaries are a better product surface than a single implicit claim that an LLM understood an entire visual identity.
Provenance is the first, smaller step
A complete benchmark takes work. Provenance can happen now. Every generated profile can name its source URL, extraction method, generation date, tool version if applicable, and confidence by section. A manually written first-party file can say manual. A file read from the page’s DOM can say DOM-scraped. A summary of screenshots and page content can say LLM-inferred.
This does not make an inaccurate output accurate. It makes its limits machine-readable and reviewable. It also creates data that a future benchmark can use. If an extractor changes its method, results can be grouped by method. If a file has low confidence in layout, an agent can avoid treating spacing values as strict requirements.
Velofy’s draft DESIGN.md profile makes this provenance block mandatory. That is not a claim of a benchmark, and it should not be read as one. It is a small discipline: no design file should arrive without an account of how it was made.
What Velofy is doing next
Velofy intends to propose a public, inspectable extraction benchmark based on first-party, licensed source systems if the corpus and method can be defined responsibly. We intend to publish per-axis results, raw artifacts, and method notes rather than a single unsupported score. Until then, any future generated design profile should carry its source URL, method, confidence, generator, and date.