Skip to content
Troy

Troy / Guides

How Troy reads a page

The read pipeline: settle, extract, cover, transcribe and fuse, what each stage decides, the output it produces, and how to run it from the panel or a script.

On this page
  1. The problem it is aimed at
  2. The reading model
  3. The five stages
  4. 1. Settle
  5. 2. Extract
  6. 3. Cover
  7. 4. Transcribe
  8. 5. Fuse
  9. Output
  10. The document
  11. Markdown
  12. What the agent panel reads today
  13. In v0.1.5
  14. Reading from a script
  15. With scripts/read.mjs (main only)
  16. From your own code
  17. Limits
  18. Test fixtures for the pipeline

Troy is named for reading a page the way a person sees it: structure from the Document Object Model (DOM), and pixels only where the DOM cannot say what is on screen. This page explains the pipeline that does this, the output it produces, and its limits.

Release status. The pipeline is on the main branch and not yet in a release. In v0.1.5, the current release, the agent panel reads the page’s text only (see In v0.1.5). To use the pipeline today, run Troy from source (Installation).

The problem it is aimed at

An agent pointed at the web has two poor options. Dumping the HyperText Markup Language (HTML) costs tens of thousands of tokens of wrappers and noise. Taking a screenshot shows the page but loses every link, form field and selector.

Some real text is also never in the DOM: numbers painted into a canvas dashboard, text baked into an image, an embedded Portable Document Format (PDF) file, a cross-origin iframe. Only the pixels say what those contain.

The reading model

Three facts shape the pipeline.

  1. Only the DOM can be acted on. Selectors, roles and links exist only there, and an agent that wants to click needs them. So the DOM is read first, for everything it can explain.
  2. The DOM also holds text nobody can see. Text can be display: none, zero-sized, clipped, fully transparent, or white on white. Hidden text is the most common way scraped output gets poisoned. So every run of text gets a visibility verdict, and invisible text never reaches the output.
  3. Some painted text has no DOM. Reading it needs optical character recognition (OCR), which is slow and can cost money. So OCR runs only on the regions the DOM fails to explain, and a plain article costs zero OCR calls.

The five stages

The pipeline is readPage in src/read/pipeline.js. It needs only two things from the browser: a way to evaluate an expression in the page, and, for OCR, a way to take a screenshot of a box. That narrow port is why the same code runs in the agent panel, in scripts/read.mjs and in the tests.

1. Settle

Reading a page mid-render gives half an article, fallback fonts and boxes that move after they were measured. So the pipeline waits for three signals:

  • document.readyState is complete.
  • document.fonts.ready has resolved.
  • Two animation frames in a row show no change in a cheap layout signature (scroll size plus element count).

After 10 seconds it reads anyway and marks the result as not settled. A page with a ticking clock or an endless spinner must still be readable. To do this, settle installs a small probe on the page (window.__troySettleProbe) that stops itself once it has seen two quiet frames.

2. Extract

One evaluation walks the whole document, including open shadow roots, and returns:

  • Blocks: runs of text tied to the element that owns them, each with a role (heading, paragraph, link, list item, button, cell and so on), a box in viewport pixels, a selector and a visibility verdict. A link gets its own block and its address, so each anchor can be clicked by selector.
  • Candidates: regions that might need pixels: canvas, video, object and embed elements, images, Scalable Vector Graphics (SVG) elements, iframes, and elements with a background image of at least 10,000 square pixels.

A block is invisible when its element has display: none or visibility: hidden, is smaller than 2 pixels in either direction, is clipped to nothing by clip or an overflow-hiding ancestor, sits entirely at negative page coordinates or is text-indented out of view, has an effective opacity at or below 0.02, or has a text colour that matches the solid background behind it. Invisible blocks are kept in the extraction with the rule that condemned them (hiddenReason), so a drop can be audited.

Form controls are read from their value, aria-label, placeholder or name. A password field is reported only as [password field]; its value is never read.

Selectors are anchored to the nearest ancestor with a unique id and positional (:nth-of-type) below it, so they survive restyling. A selector that crosses into a shadow root joins the hops with >>>.

3. Cover

Cover decides which candidates the extracted text fails to explain:

  • A canvas, video, object or embed is always a gap.
  • An image is a gap when it has no alt text.
  • An SVG element is a gap when it has no accessible text.
  • An iframe is a gap when it is cross-origin.
  • A painted background area is a gap when visible text covers less than 5 percent of it.

Overlapping gaps are merged into one region, so a composite graphic costs one OCR call, not five. Regions are clamped to the viewport, because that is what a screenshot can show. Regions smaller than 1,200 square pixels, or under 16 pixels on a side, are dropped as noise.

4. Transcribe

Each region is cropped from a screenshot and passed to an OCR engine. Recognised lines are scaled back from image pixels to viewport pixels, so everything downstream uses one coordinate space.

When no engine is available, each region becomes a block that says so, with its size and position:

[untranscribed canvas region, 482x222px at (40, 117); no OCR engine available]

That is deliberate. An agent told “there is an unread canvas here” can take a screenshot, ask a vision model or tell the person. An agent shown nothing believes the page was empty there. A region whose crop or recognition fails also becomes this marker, so one bad region cannot fail the whole read. A region where a working engine finds no text adds nothing: a blank chart area reads as empty.

5. Fuse

Fuse drops every invisible block, then puts the rest in reading order. Blocks whose horizontal extents overlap are grouped into one column, each column is ordered top to bottom, and columns are emitted left to right. When a full-width header joins everything into one column, the order falls back to plain top to bottom.

Every fused block keeps its box and its source (dom or ocr). DOM blocks keep their selector. OCR blocks have none, because no element is behind them.

Output

The document

readPage returns one object. scripts/read.mjs --json prints it as JavaScript Object Notation (JSON). An abridged example, with illustrative values:

{
  "url": "http://localhost:3000/dashboard",
  "title": "Operations",
  "blocks": [
    {
      "text": "Operations Dashboard",
      "box": { "x": 40, "y": 61, "w": 1160, "h": 37 },
      "source": "dom",
      "selector": "html > body > h1",
      "role": "heading",
      "headingLevel": 1
    },
    {
      "text": "[untranscribed canvas region, 482x222px at (40, 117); no OCR engine available]",
      "box": { "x": 40, "y": 117, "w": 482, "h": 222 },
      "source": "ocr",
      "untranscribed": true
    }
  ],
  "regions": [
    {
      "box": { "x": 40, "y": 117, "w": 482, "h": 222 },
      "kinds": ["canvas"],
      "selectors": ["#dashboard"]
    }
  ],
  "stats": {
    "settled": true,
    "settleMs": 140,
    "elapsedMs": 210,
    "regionCount": 1,
    "ocrEngine": "none",
    "transcribed": 0
  }
}

Block fields:

  • text, box (x, y, w, h in viewport pixels) and source are always present.
  • selector, role, href (links) and headingLevel (headings) appear on DOM blocks.
  • untranscribed: true marks an OCR region no engine read.

stats.ocrEngine is "none" when no engine ran, and stats.transcribed counts the regions an engine actually read.

Markdown

The default output is Markdown for reading:

  • The page title becomes a level 1 heading and headings keep their level.
  • Fragments that share a visual line are joined back together, and links render as [text](href).
  • OCR text is fenced and labelled, never blended in, so an agent can tell which words have a selector behind them.
  • A footer reports what the read cost, for example > read: 14 blocks, 1 region for ocr (engine: none), 210ms, with , page never settled added when settle timed out.

What the agent panel reads today

Open the panel with Cmd+Shift+A (Ctrl+Shift+A on Windows and Linux), View > Toggle Agent Panel, or the aperture button at the right of the toolbar, then choose read this page.

On main, the panel runs the full pipeline on the active tab:

  1. It attaches Electron’s debugger to the tab over the Chrome DevTools Protocol (CDP), unless one is already attached.
  2. It runs settle, extract, cover, transcribe and fuse. No OCR engine is passed, so any cover region appears as an untranscribed marker.
  3. It detaches the debugger again if it attached it, even when the read fails, so DevTools is never locked out of the tab.

The panel shows the address, the title, the block count split into DOM and OCR blocks, the region count, the character count, the time taken, whether settle finished or timed out, and then the Markdown. If the read fails, it shows the error instead.

In v0.1.5

The released panel has the same button but a simpler read:

  1. It attaches the debugger as above.
  2. It evaluates document.body.innerText, the text that is laid out and painted.
  3. When innerText is empty but the DOM holds text, which happens on a tab that has never been shown, it falls back to textContent and marks the result as degraded.
  4. It returns the address and the first 4,000 characters of the title followed by the text, then detaches.

The panel labels this “dom only for now”. There is no OCR in this path.

Reading from a script

With scripts/read.mjs (main only)

scripts/read.mjs runs the same pipeline from the command line against a running Troy. It needs a source checkout with npm install done, and Troy started with --cdp-port.

node scripts/read.mjs                                                    # Markdown for one open tab
node scripts/read.mjs --url http://localhost:3000/                       # the tab at exactly this address
node scripts/read.mjs --url http://localhost:3000/ --json --out page.json  # the full document, to a file
  • --url <address> picks the open tab whose address equals this string exactly. It does not navigate, and https://example.com does not match a tab at https://example.com/. With no arguments at all, the script reads the last http or https tab in the list Playwright returns, which is not necessarily the tab in front.
  • Give --url whenever you give any other option. The script takes the word after --url as the address, and when --url is missing it takes the first argument instead, so node scripts/read.mjs --json looks for a tab whose address is --json and exits with code 2.
  • --json prints the whole document instead of Markdown.
  • --out <file> writes the output to a file instead of printing it.
  • --fetch-mode=reap --url <address> skips the browser. It fetches the address over plain Hypertext Transfer Protocol (HTTP) by running python3 -m curl_reap.cli get <address> and prints what that returns. It needs Python 3 and curl_reap installed.

The script finds Troy through agent-endpoint.json. It looks in ~/Library/Application Support/Troy on macOS and AppData/Roaming/Troy under your home folder on Windows. On Linux that default is wrong, so set TROY_ENDPOINT_FILE:

TROY_ENDPOINT_FILE="$HOME/.config/Troy/agent-endpoint.json" node scripts/read.mjs

It exits with code 2 when no tab matches (or --fetch-mode=reap has no --url), 1 on any other error, and 0 on success. It always detaches without closing Troy. It passes no OCR engine, so cover regions come back untranscribed.

From your own code

To check that something is visible from an attached CDP client, read the painted text:

const text = await page.evaluate(() => document.body.innerText)

Text that is in the DOM but not painted is not page content. Assert on innerText or a role query, never on textContent or innerHTML, which return text a person could not see.

To run the pipeline yourself, give readPage a port with evaluate and, for OCR, screenshot. This sketch runs from the root of a Troy checkout and attaches over CDP with Playwright:

import { chromium } from 'playwright'
import { readPage } from './src/read/pipeline.js'
import { toMarkdown } from './src/read/render.js'
import { appleVisionEngine } from './src/read/apple-vision.js'

const browser = await chromium.connectOverCDP('http://127.0.0.1:9333')
const page = browser.contexts()[0].pages().find((p) => p.url().startsWith('http'))
const session = await page.context().newCDPSession(page)

const port = {
  async evaluate(expression) {
    const { result } = await session.send('Runtime.evaluate', { expression, returnByValue: true })
    if (result.subtype === 'error') throw new Error(String(result.description))
    return String(result.value ?? '')
  },
  async screenshot(box) {
    const clip = box && { x: box.x, y: box.y, width: box.w, height: box.h, scale: 1 }
    const { data } = await session.send('Page.captureScreenshot', { format: 'png', clip })
    return Buffer.from(data, 'base64')
  },
}

const doc = await readPage(port, { settleTimeoutMs: 10000, ocr: appleVisionEngine() })
console.log(toMarkdown(doc))
await browser.close()   // detaches only

appleVisionEngine() is the one real OCR engine in the code, and it is macOS only. On first use it compiles a small Swift helper with swiftc (from the Xcode command line tools) and caches it in the temporary folder. On any other platform, or without swiftc, it reports itself unavailable and the read degrades to untranscribed markers. Nothing in the app calls it yet, and the test suite does not exercise it. Leave ocr out to get the default engine, which reads nothing.

Limits

  • OCR sees the viewport only. Regions are clamped to what is on screen, so a chart below the fold is not nominated. DOM text below the fold is still read.
  • No OCR engine is wired in. Neither the panel nor scripts/read.mjs passes one, so cover regions appear as untranscribed markers.
  • Same-origin iframes are flattened. Their text is read as one block of up to 20,000 characters with the frame’s selector, and elements inside them get no selectors. Cross-origin iframes become cover regions.
  • Shadow DOM selectors cannot be acted on yet. Closed shadow roots are not walked. Selectors that cross a shadow boundary use >>>, which document.querySelector rejects, so the action scripts and the tool layer cannot use them. The project’s September 2026 spec lists this as bug B5.
  • The visibility rules have edges. Text over a background image or a translucent background is not checked for camouflage and counts as visible. Text covered by another element is not detected.
  • Very large pages are capped. A read collects at most 5,000 text blocks and 500 candidates. Past that the walk stops adding, and the output does not say so.
  • Settle has a deadline. After 10 seconds the page is read as it stands and flagged as not settled.

Test fixtures for the pipeline

test/fixtures/ holds three pages, each with an expected-output JSON file listing text that must and must not appear:

  • An article. On main, test/read.test.ts checks that every expected fact appears, that the page settles, and that it produces zero OCR regions.
  • A page with hidden text. The visible line must appear and three hidden traps must not. The test also checks that the extraction keeps all three with a hiddenReason.
  • A canvas dashboard. The canvas must be nominated as a cover region while the heading is still read from the DOM.

Unit tests cover fuse ordering (two columns read column by column, hidden blocks dropped, OCR blocks kept without an invented selector) and cover decisions (a canvas becomes a region, an image far below the viewport does not).

test/score.ts is a scoring helper for the same fixtures. It counts expected strings recovered, output lines that match nothing expected, and forbidden strings that leaked through.