Source: https://velofy.co/curl_reap/how-it-works/ [curl\_reap](https://velofy.co/curl_reap/) / Get started # How it works Why curl\_reap is built the way it is: what a server sees of a client, how captured browser bytes are replayed, the trade-offs and the failure modes. This page explains the design of curl\_reap from first principles: the problem, the constraints, the mechanism, what it costs and how it fails. The guides cover how to use each part; this page covers why each part exists. ## The problem A scraper does three jobs, and each one is harder than it looks: 1. **Get the page.** The server decides what to send based on how the request looks, not only on what it asks for. A request that looks like a script can get a block page while a browser gets the content. 2. **Find the data.** Selectors are written against today’s markup. When the site renames a class, the selector quietly finds nothing. 3. **Do it many times.** Thousands of requests to one host are load on someone else’s server. A crawl needs limits, or it hurts the site and gets itself blocked. curl\_reap has one layer for each job: a transport, a parser with self-healing selectors and extractors, and a crawl engine. Most of the design work is in the transport, so most of this page is about it. ## What a server can see A request over Hypertext Transfer Protocol Secure (HTTPS) reaches the server in layers. Each layer carries details the client chose, and each one can be compared with what a real browser sends. * **The TLS ClientHello.** Transport Layer Security (TLS) starts with the client’s ClientHello, which is sent before any encryption is set up, so every field in it is readable on the wire: the cipher suites in order, the extensions in order, the supported groups and key shares, the signature algorithms, the Application-Layer Protocol Negotiation (ALPN) list, and Generate Random Extensions And Sustain Extensibility (GREASE) placeholder values. Fingerprint formats such as JA3 and JA4 reduce these fields to a short string. Different TLS libraries produce different hellos, so the fingerprint names the library: Python’s `ssl` module (OpenSSL) and Chrome (BoringSSL) look nothing alike. * **The HTTP/2 connection preface.** HTTP/2 is version 2 of the Hypertext Transfer Protocol (HTTP). Before the first request, an HTTP/2 client sends a SETTINGS frame, a WINDOW\_UPDATE and sometimes PRIORITY frames. The values, their order, the order of the pseudo headers (`:method`, `:authority`, `:scheme`, `:path`) and the header compression choices differ from browser to browser. The Akamai HTTP/2 fingerprint reduces these to a short string. * **The headers.** Which headers are sent, in which order, with which casing. A browser sends a different set over HTTP/2 than over HTTP/1.1: Chrome 124 and newer and Safari 18.0 and newer add `priority: u=0, i` on HTTP/2. * **Behavior.** Request rate, cookies, whether JavaScript runs, and the reputation of the Internet Protocol (IP) address the request comes from. The constraint that follows: **every layer must agree.** A Chrome User-Agent header on an OpenSSL handshake is a contradiction that a server can detect with one comparison. Copying the headers is not enough, so a curl\_reap profile covers the handshake, the HTTP/2 framing and both header sets together. ## Why not the standard library or a curl binding Python’s `ssl` module wraps OpenSSL. It lets you choose ciphers and ALPN protocols, but it has no controls for extension order, GREASE, the set of key shares, certificate compression or padding. It cannot produce a browser’s ClientHello. curl-impersonate and curl\_cffi solve this by compiling curl against the browser’s own TLS library. That works, and curl\_reap used curl\_cffi before 1.0.0, but it ties every user to a compiled binary. Since 1.0.0 the transport is written in Python. The project README gives the reason: no compiled curl binding, no runtime downloads, and a readable engine a maintainer can adapt the day a browser changes its hello. ## The mechanism ### Captured templates A browser’s fingerprint cannot be derived from a specification; it can only be recorded. Each of the 38 concrete targets is a recording: * `tools/capture_clienthello.py` opens a local listener, lets the reference client (curl\_cffi 0.16.3) connect as one target, and records the raw ClientHello. Repeated runs classify every field as fixed or variable. * `tools/capture_h2.py` runs a local HTTP/2 server and records every frame the client sends: SETTINGS pairs in wire order, the WINDOW\_UPDATE increment, PRIORITY frames, HEADERS flags, and how each header was compressed. It also records the HTTP/1.1 header order. * Generator scripts turn the captures into plain Python data modules (`curl_reap/_tls/templates_data.py`, `curl_reap/_h2/templates_data.py` and `curl_reap/_profiles_data.py`). Adding or refreshing a target is therefore a new capture plus a regenerated data file, not new engine code. ### Rendering the ClientHello A template holds everything fixed about a browser’s hello. The builder fills in what must be fresh on every connection: the client random, the session id, the key share public keys, and the GREASE values, drawn the way BoringSSL draws them. Chrome also permutes its extension order on each connection, and the builder does the same. That is why JA3, which keeps extension order, changes from one Chrome connection to the next, while JA4, which sorts extensions, stays stable. Padding follows BoringSSL’s rule, so the hello has the length the browser’s would have. ### The TLS 1.3 engine In TLS 1.3, everything after the server’s reply (the ServerHello) is encrypted, so a library that writes its own ClientHello has to implement the rest of the handshake too. `curl_reap._tls` is a TLS 1.3 client built on primitives from the `cryptography` package. It handles HelloRetryRequest, certificate compression, Application-Layer Protocol Settings (ALPS), Encrypted Client Hello (ECH) GREASE, KeyUpdate, session tickets and pre-shared key (PSK) resumption. Certificates are checked with the `cryptography` package’s verifier, which follows Request for Comments (RFC) 5280. The key exchange includes the X25519MLKEM768 post-quantum hybrid, with a pure-Python implementation of the Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM). ### HTTP/2 and headers `curl_reap._h2` sends the captured SETTINGS, WINDOW\_UPDATE, PRIORITY frames, HEADERS flags and pseudo header order for the target, and compresses each header with the same representation the captured browser used. Headers come from the target’s captured set for the protocol in use: the HTTP/1.1 set with browser casing, or the lowercase HTTP/2 set. Your own headers replace values in place, so the order stays the browser’s. ### Fallbacks, and how to tell which path ran The engine implements TLS 1.3 only. Connections that cannot use it go through the standard library’s OpenSSL binding instead: | Situation | Backend | What still matches the browser | | --- | --- | --- | | Built-in target, server speaks TLS 1.3 | `native` | handshake, HTTP/2 framing, headers | | Server picks TLS 1.2 | `openssl` | HTTP/2 framing and headers | | Request sends a client certificate (`cert=`) | `openssl` | HTTP/2 framing and headers | | Custom `Profile` without a template | `openssl` | headers only | | Plain `http://` URL | `plain` | headers only | `Response.infos["tls_backend"]` says which backend a response came over: `"native"`, `"openssl"` or `"plain"`. ## Trade-offs * **Speed.** On the project’s reference machine, a request on a fresh HTTPS connection (a full handshake each time) has a median time of about 0.96 milliseconds against 0.45 milliseconds for curl\_cffi, and HTTP/2 runs at 0.75x curl\_cffi’s throughput. On reused HTTP/1.1 and HTTPS connections curl\_reap is faster (1.16x to 1.78x). PSK resumption shortens repeat handshakes where servers issue tickets. The numbers are in [Transport and impersonation profiles](https://velofy.co/curl_reap/transport-and-profiles/#performance). * **One request per connection.** HTTP/2 connections carry one request at a time. Concurrency comes from more connections, not multiplexing. * **Snapshots age.** A capture is one browser version. Browsers change their hello over time, and a target stays accurate only for the version it recorded. * **Not implemented:** HTTP/3 (QUIC), a native TLS 1.2 handshake, and cleartext HTTP/2. ## Failure modes * **The fingerprint matches and the site still blocks.** The handshake is one signal among several. IP reputation, request rate, missing cookies and JavaScript challenges are separate. `Response.block_reason` and `Response.looks_like_challenge` tell you what came back. curl\_reap does not solve challenges; see [Responsible use](https://velofy.co/curl_reap/responsible-use/). * **You break the agreement yourself.** Setting another browser’s User-Agent, or a `ja3=` string without matching headers, creates the contradictions the profiles exist to avoid. * **The path changed under you.** A server that only speaks TLS 1.2, a client certificate, or a custom profile moves the connection to OpenSSL. Check `infos["tls_backend"]`. Through a proxy, read the [known issue with TLS 1.2 servers](https://velofy.co/curl_reap/troubleshooting/#tls-12-servers-and-proxies). * **Something between you and the server re-originates TLS.** A proxy that terminates TLS sends its own ClientHello, not yours. * **Legacy names are approximations.** The 22 legacy names reuse the nearest capture’s fingerprint. See [Impersonation targets](https://velofy.co/curl_reap/impersonation-targets/). ## Verify it yourself `reap doctor` fetches a fingerprint echo service as a target and compares what the service saw with the capture: ``` reap doctor chrome ``` ``` ja4 MATCH ja4_r MATCH ja3 n/a (not compared: this browser permutes its extensions per connection) akamai MATCH headers MATCH [reap] chrome: every comparable fingerprint matches the capture ``` It exits with 0 when every comparable row matches and 1 when one differs. See [Command line](https://velofy.co/curl_reap/cli/#check-your-fingerprint-reap-doctor). ## The parsing and crawling layers The same reasoning shapes the other two layers. * **Selectors.** A Cascading Style Sheets (CSS) selector encodes assumptions about class names and structure. A saved signature records several independent signals instead: tag, classes, id, attribute names, text and tag path. When a site renames a class, the other signals can still score the element above the threshold. See [Selectors and self-healing](https://velofy.co/curl_reap/selectors/). * **Extraction.** Many sites already publish their data in machine-readable form, as JavaScript Object Notation for Linked Data (JSON-LD) for search engines and as Open Graph meta tags for link previews. Reading those avoids selectors entirely. See [Structured extraction](https://velofy.co/curl_reap/structured-extraction/). * **Crawling.** A crawl is a queue of pending requests, a set of seen requests, a page budget and a politeness clock per host. Each one is bounded: the budget is reserved before the network, `max_pending` bounds the queue, `per_domain_concurrency` bounds requests in flight per host, and AutoThrottle hands out start slots per domain. The queue lives in memory, so a crash loses pending work. See [Crawling](https://velofy.co/curl_reap/crawling/).