How it works

This page explains why Troy is built the way it is. It starts with the problem, sets out the constraints that follow from it, then walks through each mechanism: what it does, what it costs, and how it fails.

Architecture lists the same parts file by file. How Troy reads a page covers the read pipeline stage by stage.

The problem

An agent working on the web needs two things: the page a person would see, and a way to act on it. Most browser automation gets both wrong.

• It starts a fresh, empty browser. The pages worth automating (a staging site, an admin panel, a dashboard) sit behind a login. A fresh profile has no session, so the agent lands on a sign-in wall. A person then has to walk it through the login, hand over credentials, or give up.
• It hands the agent the page in a poor form. Raw HyperText Markup Language (HTML) runs to tens of thousands of tokens of wrappers and noise. A screenshot shows the page but loses every link, form field and selector, so the agent can read but not act.

Some real text is also never in the Document Object Model (DOM) at all: numbers drawn on a canvas, text inside an image, a cross-origin iframe. Only the pixels say what those contain.

First principles

Five facts about browsers and agents decide most of Troy’s design.

1. A session is state in a profile. Cookies and site storage live in the browser’s profile folder. You cannot give an agent your session without giving it that profile, so the agent has to work inside the browser that already holds it. Troy is therefore a browser you use yourself, and the agent attaches to it.
2. Control of a Chromium browser is one protocol. Navigation, input, script evaluation and screenshots all go through the Chrome DevTools Protocol (CDP). Any process that reaches the debugging port controls every tab in that profile. So the port must stay closed unless you ask for it, listen on this machine only, and be opened per launch.
3. A page is untrusted input. It can carry hidden text, try to reach the browser’s own interface, or address instructions to the agent. So refusals have to live in code the page cannot reach, and a read has to drop text a person could not see.
4. Elements are what an agent acts on; pixels are what a person sees. Only the DOM has selectors, roles and links. Only pixels show text drawn on a canvas or baked into an image. So Troy reads the DOM first and spends optical character recognition (OCR) only on regions the DOM cannot explain.
5. The person and the agent share one window. Anything that crashes the browser, closes a tab or navigates away costs the person their work. So the browser must survive its own errors, and the agent rules forbid closing the person’s tabs or navigating them away without asking.

The rest of this page is what follows from those five facts.

A real browser with its own chrome

Troy is an Electron app. The window’s own web contents draw Troy’s chrome: the tab strip, the toolbar and the agent panel. Each tab is a separate Chromium view (WebContentsView) laid out below the top 88 pixels.

What follows from that split:

• Pages render exactly as Chrome renders them, because they are Chromium.
• A page cannot repaint or script the chrome. It is a different frame tree in a different process, with context isolation and the sandbox on and no Node.js integration.
• Every tab already speaks CDP through Electron’s webContents.debugger, which is how the agent panel reads the tab you are looking at.

Trade-off. The chrome and the page do not share a compositor. Cascading Style Sheets (CSS) effects that need to see the page, such as a blur behind the toolbar, are impossible, so on macOS the glass comes from the window’s system vibrancy instead. Anything drawn at the seam, like the loading bar, has to sit inside the chrome’s 88 pixels or the page view covers it.

An agent port that is off by default

--cdp-port=<port>, or TROY_CDP_PORT, turns on Chromium’s remote debugging on that port and accepts connections from the http://127.0.0.1 origin. Troy then writes agent-endpoint.json into its profile folder, so an agent can find the port without anyone copying it between terminals. The file is deleted when Troy quits.

An agent attaches with any CDP client. Playwright’s connectOverCDP returns the tabs already open, including the one you signed into, and browser.close() on that connection only detaches.

Trade-off. Once open, the port is full control of every signed-in tab for any process on the machine. Troy adds no token on top. The protection is that the port does not exist unless you ask for it, for that launch only.

Failure modes.

• A port value that is not an integer from 1 to 65535 opens nothing, without an error.
• A crash or force quit skips the clean-up, so agent-endpoint.json can outlive the browser and point at a dead port.
• Troy does not hold a single-instance lock. A second copy on the same profile overwrites the file, and deletes it when it quits.

Driving Troy from an agent shows how to check that the endpoint is live.

Refusals in one resolver

Everything typed into the address bar or the new tab search box goes through one function, resolveOmnibox. It returns one of five results: load a web address, run a search, hand the address to another app, refuse it, or do nothing.

Refused schemes (javascript:, data:, blob:, filesystem:, chrome:, devtools:, chrome-extension:) are checked before the generic “looks like a web address” rule. That ordering is the point: javascript://x/%0aalert(document.cookie) passes a naive address check, and pasted into a signed-in tab it would run script in that site.

The same idea applies to what pages do on their own:

• A page that tries to move its top frame to a scheme other than http, https, file or about:blank is cancelled and the address goes to the operating system.
• window.open becomes a Troy tab, never an uncontrolled window.
• Every permission request is denied except fullscreen and sanitized clipboard writes.

Why one resolver. Several entry points that each check schemes will drift apart. On the main branch, the agent tool layer’s page_navigate calls the same resolver, so an agent is refused exactly what a person is.

Trade-off. There is no override. A site that needs the camera, or a data: address you meant to open, needs another browser.

Reading a page: structure first, pixels second

On the main branch, not yet in a release, a read runs five stages:

1. Settle. Wait until the document has loaded, its fonts are ready and two animation frames pass with no layout change. Give up after 10 seconds and read anyway.
2. Extract. Walk the whole DOM in one evaluation. Record each run of text with its role, box, selector and a visibility verdict.
3. Cover. Choose only the regions the DOM cannot explain: canvases, images without alt text, wordless Scalable Vector Graphics (SVG), cross-origin iframes, large painted areas with no text over them.
4. Transcribe. Send those regions to an OCR engine, or mark each one as untranscribed when no engine is available.
5. Fuse. Drop invisible text and order the rest into reading order, column by column.

Why one evaluation. Every round trip to the page is a chance for it to change. Fifty questions get fifty answers about fifty slightly different pages.

Why OCR is the exception. It is slow, and a hosted engine costs money. A plain article should cost zero OCR calls, and the test suite checks exactly that on its article fixture.

Trade-offs and failure modes. The visibility rules are rules, not vision. Text over a background image or a translucent colour cannot be checked for white-on-white, so it counts as visible. Text covered by another element is not detected. OCR regions are limited to the visible viewport. And on main no OCR engine is passed to the panel or the read script yet, so cover regions come back as labelled “untranscribed” markers rather than text. How Troy reads a page lists every limit.

Verified actions

The action scripts in scripts/, and the tool layer on main, follow three rules:

• Inspect before acting. Every refusal happens before anything that could change the page runs.
• Refuse ambiguity. In click.mjs and the tool layer, a target that matches two elements is an error, not a coin flip. The action scripts also refuse to choose between two tabs on an allowed host.
• Check the result. A fill reads its value back. A click compares a snapshot of the page before and after, and reports NOT VERIFIED when nothing observable changed.

Submit-like controls and password fields are refused in code. Submitting is left to the person.

Why. An agent that reports “done” for an action that never landed is worse than one that reports a failure, because the person stops checking.

Trade-off. Some real clicks change nothing the snapshot measures, such as a click that only redraws a canvas. Those report NOT VERIFIED. Give click.mjs an explicit expectation (--expect-text, --expect-visible and so on) for those cases.

Staying open

An uncaught exception in Electron’s main process normally ends the app, and every tab and signed-in session with it. Troy installs a safety net as soon as the app is ready, before anything else. It logs the error to troy-errors.log in the profile, shows a notice, and keeps running.

Two more guards back it up:

• Reads of a tab first check that its renderer still exists, because every getter on a destroyed web contents throws.
• A tab whose renderer crashes shows a failure page with the reason instead of staying blank.

Trade-off. An error the net catches may leave one tab or feature in an odd state. The log entry is where to look, and what to attach to an issue.

Defaults that keep nothing

• No history unless asked. In v0.1.5 the Remember history switch is stored but records nothing. On main it records up to 500 visits in history.json, and turning it off deletes the file.
• Nothing sent while typing. The search address is built once, when you press Enter. No suggestion service is asked anything.
• Trackers blocked. Third-party analytics and ad requests are cancelled and tracking parameters are stripped, on by default.

Each default is the one that collects nothing, because the profile holds real accounts. Privacy and permissions covers the details and where each protection stops.

What the design costs

• A large download. Electron ships its own Chromium, and Chromium is most of the size.
• No auto-update. Updates come from Homebrew or a new download.
• One gesture on first launch. Troy is ad-hoc signed but not notarised. The Homebrew cask clears the quarantine flag, so installing that way skips it.
• One profile per install. Every session lives in one folder, which is also what an attached agent works in.

When something goes wrong, Troubleshooting is organised by symptom.