Labs · Scoped

Orrery

Tech news, ranked by corroboration

There is no shortage of places to read tech news. There is a shortage of places where reading it takes fifteen minutes instead of an hour and leaves you genuinely informed rather than vaguely anxious.

Hacker News surfaces what a specific crowd upvoted, which is a signal and also a monoculture. Techmeme is excellent and reads like a wire service. Feed readers give you everything, which is the same as giving you nothing. AI newsletters summarise, and mostly summarise each other.

Orrery is a reader that ranks by corroboration and novelty rather than popularity, and shows its working. An orrery is a clockwork model of the solar system: the point is seeing how the parts move relative to each other, not staring at one planet.

The thesis

A story matters when several independent sources report it and it is not a restatement of something you already read. Both of those are measurable without a language model, and measuring them is more honest than counting votes.

Upvote-ranked feeds have a known pathology: they surface what is entertaining to a community, on that community's schedule, with a heavy recency bias and a strong pull toward drama. Corroboration ranking has a different failure mode, press-release amplification, which is at least detectable.

How ranking works, deterministically

No model call is in the ranking path. The pipeline is arithmetic and set operations, which means it is reproducible, debuggable and free to run.

ingest        pull ~300 feeds on a schedule (RSS/Atom, a few APIs)
normalise     canonical URL, strip trackers, resolve redirects, dedupe
cluster       group articles about the same event
score         corroboration + independence + novelty - recency decay
present       one card per cluster, sources listed

Clustering

Same-event detection without embeddings: shingle the title and lede into character n-grams, MinHash them, and bucket with locality-sensitive hashing. Two articles land in a cluster when their Jaccard similarity clears a threshold, with a boost when they share an outbound link, since three outlets linking the same filing are almost certainly covering the same filing.

This is a decades-old technique, it runs in milliseconds, and it is good enough. Embeddings would be marginally better at paraphrase and would add a model dependency to the hot path for that margin.

Scoring

score = corroboration * independence * novelty * decay

corroboration = log(1 + distinct_sources_in_cluster)
independence  = 1 - (shared_ownership + syndication_overlap)
novelty       = 1 - max_similarity(cluster, everything_from_last_7_days)
decay         = 0.5 ^ (hours_old / half_life)

Independence is the term that does the real work. Twelve outlets covering a story means nothing if nine of them are the same wire copy or share a parent company. We maintain an ownership map and detect syndication by near-identical body text, then discount accordingly. A story with four genuinely independent sources outranks one with twelve syndicated ones.

Novelty is what stops the same story from occupying the top slot for three days as new outlets file late follow-ups.

Where a model is allowed

Exactly one place: writing the two-line cluster summary, after ranking has already decided what appears. The summary is cosmetic. If the model is unavailable, the card falls back to the highest-scoring source's own lede and the product still works.

Ranking never calls a model. That is the line.

What the reader sees

One screen, roughly twenty cards, refreshed a few times a day rather than continuously. A card carries the headline, a two-line summary, the source count, and the sources themselves as small links.

The differentiating detail: every card can show why it ranked. Tap the score and you see the corroboration count, the independence discount, the novelty measure and the age. If you disagree with the ranking, you can see exactly which term you disagree with.

No infinite scroll. No engagement metrics. No comment section. The feed ends, deliberately, because a reader that ends is one you can finish.

Build shape

PieceChoiceWhy
IngestPython, curl_reapFeeds are easy; the publisher pages behind them block stock clients. Already solved.
SchedulerPostgres queue, SKIP LOCKEDOne less service than a broker, and durable.
StorePostgres, full-text plus pg_trgmClustering and search without a second datastore.
APIFastAPIReads are cacheable and the payload is small.
WebStatic HTML + summit.jsA reader should not ship 200KB of framework. Dogfoods our own.
MobileAndroid, laterWeb first. The value is ranking, not a native shell.

The whole thing is designed to run on one small box. Three hundred feeds polled a few times a day is a trivial workload; the interesting engineering is entirely in the clustering and scoring, not the infrastructure.

Honest risks

  • Corroboration rewards press releases. A well-distributed announcement gets covered everywhere by definition. The independence term helps and does not fully solve it. Likely mitigation: a press-release detector, since that copy has recognisable structural tells.
  • Feeds are shrinking. More publishers ship truncated or no RSS. Falling back to fetching articles raises both cost and the ethical bar, and we would follow the same posture as curl_reap: obey robots.txt, honour crawl-delay, identify honestly.
  • Cold start on novelty. The novelty term needs a week of history before it means anything. The first week will rank worse and we should say so in the product rather than hide it.
  • It may just be Techmeme. The genuine risk is that a good editor already solves this better than an algorithm. The test is whether transparent ranking plus a feed that ends is worth switching for. If it is not, this should not ship.

Status

Scoped, not started. First milestone is a two-week offline evaluation: ingest 300 feeds, cluster them, and compare the top twenty against Techmeme's front page for the same day. If corroboration ranking does not beat or match a human editor on a blind read, the thesis is wrong and this stays a document.