How it works

This page explains the design of curl_reap from first principles: the problem, the constraints, the mechanism, what it costs and how it fails. The guides cover how to use each part; this page covers why each part exists.

The problem

A scraper does three jobs, and each one is harder than it looks:

1. Get the page. The server decides what to send based on how the request looks, not only on what it asks for. A request that looks like a script can get a block page while a browser gets the content.
2. Find the data. Selectors are written against today’s markup. When the site renames a class, the selector quietly finds nothing.
3. Do it many times. Thousands of requests to one host are load on someone else’s server. A crawl needs limits, or it hurts the site and gets itself blocked.

curl_reap has one layer for each job: a transport, a parser with self-healing selectors and extractors, and a crawl engine. Most of the design work is in the transport, so most of this page is about it.

What a server can see

A request over Hypertext Transfer Protocol Secure (HTTPS) reaches the server in layers. Each layer carries details the client chose, and each one can be compared with what a real browser sends.

• The TLS ClientHello. Transport Layer Security (TLS) starts with the client’s ClientHello, which is sent before any encryption is set up, so every field in it is readable on the wire: the cipher suites in order, the extensions in order, the supported groups and key shares, the signature algorithms, the Application-Layer Protocol Negotiation (ALPN) list, and Generate Random Extensions And Sustain Extensibility (GREASE) placeholder values. Fingerprint formats such as JA3 and JA4 reduce these fields to a short string. Different TLS libraries produce different hellos, so the fingerprint names the library: Python’s ssl module (OpenSSL) and Chrome (BoringSSL) look nothing alike.
• The HTTP/2 connection preface. HTTP/2 is version 2 of the Hypertext Transfer Protocol (HTTP). Before the first request, an HTTP/2 client sends a SETTINGS frame, a WINDOW_UPDATE and sometimes PRIORITY frames. The values, their order, the order of the pseudo headers (:method, :authority, :scheme, :path) and the header compression choices differ from browser to browser. The Akamai HTTP/2 fingerprint reduces these to a short string.
• The headers. Which headers are sent, in which order, with which casing. A browser sends a different set over HTTP/2 than over HTTP/1.1: Chrome 124 and newer and Safari 18.0 and newer add priority: u=0, i on HTTP/2.
• Behavior. Request rate, cookies, whether JavaScript runs, and the reputation of the Internet Protocol (IP) address the request comes from.

The constraint that follows: every layer must agree. A Chrome User-Agent header on an OpenSSL handshake is a contradiction that a server can detect with one comparison. Copying the headers is not enough, so a curl_reap profile covers the handshake, the HTTP/2 framing and both header sets together.

Why not the standard library or a curl binding

Python’s ssl module wraps OpenSSL. It lets you choose ciphers and ALPN protocols, but it has no controls for extension order, GREASE, the set of key shares, certificate compression or padding. It cannot produce a browser’s ClientHello.

curl-impersonate and curl_cffi solve this by compiling curl against the browser’s own TLS library. That works, and curl_reap used curl_cffi before 1.0.0, but it ties every user to a compiled binary. Since 1.0.0 the transport is written in Python. The project README gives the reason: no compiled curl binding, no runtime downloads, and a readable engine a maintainer can adapt the day a browser changes its hello.

The mechanism

Captured templates

A browser’s fingerprint cannot be derived from a specification; it can only be recorded. Each of the 38 concrete targets is a recording:

• tools/capture_clienthello.py opens a local listener, lets the reference client (curl_cffi 0.16.3) connect as one target, and records the raw ClientHello. Repeated runs classify every field as fixed or variable.
• tools/capture_h2.py runs a local HTTP/2 server and records every frame the client sends: SETTINGS pairs in wire order, the WINDOW_UPDATE increment, PRIORITY frames, HEADERS flags, and how each header was compressed. It also records the HTTP/1.1 header order.
• Generator scripts turn the captures into plain Python data modules (curl_reap/_tls/templates_data.py, curl_reap/_h2/templates_data.py and curl_reap/_profiles_data.py).

Adding or refreshing a target is therefore a new capture plus a regenerated data file, not new engine code.

Rendering the ClientHello

A template holds everything fixed about a browser’s hello. The builder fills in what must be fresh on every connection: the client random, the session id, the key share public keys, and the GREASE values, drawn the way BoringSSL draws them. Chrome also permutes its extension order on each connection, and the builder does the same. That is why JA3, which keeps extension order, changes from one Chrome connection to the next, while JA4, which sorts extensions, stays stable. Padding follows BoringSSL’s rule, so the hello has the length the browser’s would have.

The TLS 1.3 engine

In TLS 1.3, everything after the server’s reply (the ServerHello) is encrypted, so a library that writes its own ClientHello has to implement the rest of the handshake too. curl_reap._tls is a TLS 1.3 client built on primitives from the cryptography package. It handles HelloRetryRequest, certificate compression, Application-Layer Protocol Settings (ALPS), Encrypted Client Hello (ECH) GREASE, KeyUpdate, session tickets and pre-shared key (PSK) resumption. Certificates are checked with the cryptography package’s verifier, which follows Request for Comments (RFC) 5280. The key exchange includes the X25519MLKEM768 post-quantum hybrid, with a pure-Python implementation of the Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM).

HTTP/2 and headers

curl_reap._h2 sends the captured SETTINGS, WINDOW_UPDATE, PRIORITY frames, HEADERS flags and pseudo header order for the target, and compresses each header with the same representation the captured browser used. Headers come from the target’s captured set for the protocol in use: the HTTP/1.1 set with browser casing, or the lowercase HTTP/2 set. Your own headers replace values in place, so the order stays the browser’s.

Fallbacks, and how to tell which path ran

The engine implements TLS 1.3 only. Connections that cannot use it go through the standard library’s OpenSSL binding instead:

Situation	Backend	What still matches the browser
Built-in target, server speaks TLS 1.3	native	handshake, HTTP/2 framing, headers
Server picks TLS 1.2	openssl	HTTP/2 framing and headers
Request sends a client certificate (cert=)	openssl	HTTP/2 framing and headers
Custom Profile without a template	openssl	headers only
Plain http:// URL	plain	headers only

Response.infos["tls_backend"] says which backend a response came over: "native", "openssl" or "plain".

Trade-offs

• Speed. On the project’s reference machine, a request on a fresh HTTPS connection (a full handshake each time) has a median time of about 0.96 milliseconds against 0.45 milliseconds for curl_cffi, and HTTP/2 runs at 0.75x curl_cffi’s throughput. On reused HTTP/1.1 and HTTPS connections curl_reap is faster (1.16x to 1.78x). PSK resumption shortens repeat handshakes where servers issue tickets. The numbers are in Transport and impersonation profiles.
• One request per connection. HTTP/2 connections carry one request at a time. Concurrency comes from more connections, not multiplexing.
• Snapshots age. A capture is one browser version. Browsers change their hello over time, and a target stays accurate only for the version it recorded.
• Not implemented: HTTP/3 (QUIC), a native TLS 1.2 handshake, and cleartext HTTP/2.

Failure modes

• The fingerprint matches and the site still blocks. The handshake is one signal among several. IP reputation, request rate, missing cookies and JavaScript challenges are separate. Response.block_reason and Response.looks_like_challenge tell you what came back. curl_reap does not solve challenges; see Responsible use.
• You break the agreement yourself. Setting another browser’s User-Agent, or a ja3= string without matching headers, creates the contradictions the profiles exist to avoid.
• The path changed under you. A server that only speaks TLS 1.2, a client certificate, or a custom profile moves the connection to OpenSSL. Check infos["tls_backend"]. Through a proxy, read the known issue with TLS 1.2 servers.
• Something between you and the server re-originates TLS. A proxy that terminates TLS sends its own ClientHello, not yours.
• Legacy names are approximations. The 22 legacy names reuse the nearest capture’s fingerprint. See Impersonation targets.

Verify it yourself

reap doctor fetches a fingerprint echo service as a target and compares what the service saw with the capture:

reap doctor chrome

  ja4        MATCH
  ja4_r      MATCH
  ja3        n/a  (not compared: this browser permutes its extensions per connection)
  akamai     MATCH
  headers    MATCH

[reap] chrome: every comparable fingerprint matches the capture

It exits with 0 when every comparable row matches and 1 when one differs. See Command line.

The parsing and crawling layers

The same reasoning shapes the other two layers.

• Selectors. A Cascading Style Sheets (CSS) selector encodes assumptions about class names and structure. A saved signature records several independent signals instead: tag, classes, id, attribute names, text and tag path. When a site renames a class, the other signals can still score the element above the threshold. See Selectors and self-healing.
• Extraction. Many sites already publish their data in machine-readable form, as JavaScript Object Notation for Linked Data (JSON-LD) for search engines and as Open Graph meta tags for link previews. Reading those avoids selectors entirely. See Structured extraction.
• Crawling. A crawl is a queue of pending requests, a set of seen requests, a page budget and a politeness clock per host. Each one is bounded: the budget is reserved before the network, max_pending bounds the queue, per_domain_concurrency bounds requests in flight per host, and AutoThrottle hands out start slots per domain. The queue lives in memory, so a crash loses pending work. See Crawling.