curl_reap / Guides
Transport and impersonation profiles
How the curl_reap 2.0 transport works: native TLS 1.3, OpenSSL fallback, pooling, resumption, impersonate targets and custom fingerprints.
On this page
This page covers the transport: how a Hypertext Transfer Protocol (HTTP) request is put on the wire, which Transport Layer Security (TLS) backend handles it, how connections are reused, and how impersonation profiles shape every layer. How it works explains the reasoning behind the design.
First principles
- A server sees three layers of every request: the TLS ClientHello, the HTTP/2 connection preface and the request headers. Each one can be compared with a real browser’s.
- The layers must agree. A browser User-Agent on a non-browser handshake is easy to spot, so a profile sets all three layers together. Changing one layer by hand (a different User-Agent, a
ja3=string) can break that agreement. - A fingerprint can only be recorded, not derived. Each built-in target replays bytes captured from one browser version. Aliases and legacy names reuse those captures; they add no new fingerprints.
- Reuse must not cross trust settings. Pooled connections and resumption tickets skip work that was done under one
verifyandcertsetting, so they are keyed by that setting.
How the transport works
curl_reap 2.0 has its own transport, from the TLS record layer up. There is no compiled curl binding.
- Native TLS 1.3. A pure-Python TLS 1.3 client (
curl_reap._tls) built on thecryptographypackage. For each of the 38 captured targets it renders the ClientHello captured from that browser byte for byte (aliases use their target’s capture, legacy names the nearest one): cipher order, extension set and order, Generate Random Extensions And Sustain Extensibility (GREASE) values, key shares including the X25519MLKEM768 post-quantum hybrid, certificate compression (zlib, brotli, zstd), Application-Layer Protocol Settings (ALPS), Encrypted Client Hello (ECH) GREASE and the padding rule. It handles HelloRetryRequest, KeyUpdate, session tickets and pre-shared key (PSK) resumption. - OpenSSL fallback. When a server does not pick TLS 1.3, the engine stops after the ServerHello, and the transport reconnects through the standard library’s
sslmodule. That connection sends an OpenSSL-shaped ClientHello; only the HTTP layer is the browser’s. The reconnect takes the same route as the first attempt, through the same proxy, but it cannot run through anhttps://proxy. See TLS 1.2 servers and proxies. - Built-in HTTP/2.
curl_reap._h2sends each target’s SETTINGS, WINDOW_UPDATE, priority and pseudo-header order, with header compression (HPACK) from thehpackpackage. HTTP/2 is offered through Application-Layer Protocol Negotiation (ALPN) by default and falls back to HTTP/1.1 per connection. See HTTP/2. - HTTP/1.1. The transport writes the request line and every header itself, so header order and casing are exactly what the profile and your code specify. It decodes chunked bodies and
gzip,deflate,brandzstdcontent encodings. - Connections. HTTP CONNECT proxies (always
host:port), Socket Secure (SOCKS) 4, 4a, 5 and 5h proxies, redirects (301, 302, 303, 307, 308), and cookies.
Timeouts are deadlines. A float timeout bounds one exchange as a whole: connect, send and receive. timeout=(connect, read) bounds the connect and each socket read, and sets an overall deadline of their sum. The deadline starts again on each redirect hop and each retry. None disables the clock.
Choosing the TLS backend
Session(tls=...) selects the backend:
| Value | Behavior |
|---|---|
"auto" (default) |
Native engine when the profile has a captured ClientHello template, OpenSSL otherwise. Requests with a client certificate (cert=) use OpenSSL. Falls back to OpenSSL when the server does not speak TLS 1.3, and when the native handshake fails for another non-certificate reason. |
"native" |
Native engine. A native handshake failure is raised instead of retried over OpenSSL. A server that only speaks TLS 1.2 is still reconnected over OpenSSL, because there is no native TLS 1.2. A request with cert= raises ValueError. |
"openssl" |
Always the standard library ssl module. Profiles still set the HTTP headers and HTTP/2 framing. |
import curl_reap as reap
s = reap.Session(impersonate="chrome", tls="auto")
r = s.get("https://example.com")
print(r.infos["tls_backend"], r.http_version, r.infos.get("tls_version"), r.infos.get("group"))
# for example: native HTTP/2 TLSv1.3 X25519MLKEM768
Response.infos["tls_backend"] is what a connection used: "native", "openssl" or "plain" (no TLS). Session.tls_backend is the mode you configured ("auto", "native" or "openssl").
What Response.infos holds
Response.infos is a dict with details of the connection a response came over. The keys depend on the path:
| Key | Native engine | OpenSSL path |
|---|---|---|
tls_backend |
"native" |
"openssl" ("plain" on http://) |
tls_version |
"TLSv1.3" |
the negotiated version, for example "TLSv1.2" |
cipher |
the TLS 1.3 suite | the negotiated suite |
alpn |
"h2", "http/1.1" or None |
same |
group |
the key exchange group, for example "X25519MLKEM768" |
None |
resumed |
True when the handshake resumed a session ticket |
not present |
handshake_seconds |
handshake duration | not present |
peername, sockname |
remote and local address | same |
HTTP/2 responses also carry infos["http_version"] = "HTTP/2".
Client certificates
cert= takes the path of a Privacy-Enhanced Mail (PEM) file holding the certificate and key, or a (cert, key) or (cert, key, password) tuple, on Session or per request. Paths can be strings or os.PathLike objects.
The native engine cannot present a client certificate, so a request with cert= goes through OpenSSL:
- With
tls="auto"(the default) ortls="openssl", the request uses OpenSSL and presents the certificate, both to servers that require one and to servers that only ask.r.infos["tls_backend"]is"openssl". - With
tls="native", the request raisesValueErrorinstead of being sent without the certificate.
s = reap.Session(cert=("client.pem", "client.key"))
s = reap.Session(cert=("client.pem", "client.key.enc", "key-password"))
The handshake for these requests is OpenSSL-shaped; the HTTP/2 framing and headers still follow the profile. The OpenSSL path cannot run through an https:// proxy, so use an http:// or SOCKS proxy with cert=.
Certificate verification
verify is True (the default), False, or the path of a certificate authority (CA) bundle, on Session or per request. On the native path, chain building and policy checks go through the cryptography package’s RFC 5280 web-PKI verifier. Trust anchors come from the interpreter’s OpenSSL default paths, then from certifi when it is installed, then from the usual distribution bundle paths. verify="ca.pem" loads that bundle instead (PEM with one or more certificates, or a single binary DER-encoded certificate).
Connection pooling and resumption
Idle connections are pooled and reused. The pool is bounded (10 idle connections per key, each kept for up to 60 seconds) and checks that a connection is still alive before it reuses it. The pool key is the full route and trust identity:
- scheme, host and port,
- proxy,
- profile name,
- the
verifyvalue, - the client
cert, with any key password replaced by its SHA-256 digest, - the TLS backend,
- the source
interface.
TLS session tickets for PSK resumption are kept in one cache per trust identity, that is per combination of verify and cert (_session_cache_for in curl_reap/transport.py). A resumed handshake skips certificate verification, so a ticket earned under one setting must not resume a connection that asked for another. For example, tickets from requests with verify=False are never used for a request with verify=True, and a connection verified against one CA bundle is never handed to a request that names a different one. A session keeps at most 32 ticket caches and drops the oldest when it needs room.
Profiles
A profile describes how a request looks on the wire. Pass it by name with profile= or impersonate=:
import curl_reap as reap
reap.get("https://example.com", profile="chrome") # the default
reap.get("https://example.com", impersonate="firefox")
reap.get("https://example.com", impersonate="safari")
reap.get("https://example.com", impersonate="chrome124")
s = reap.Session(impersonate="safari17_0")
The short names live in reap.PROFILES (chrome, firefox, safari). They are aliases of the newest capture of each family: chrome is chrome150, firefox is firefox147 and safari is safari2601.
A built-in profile carries:
- The TLS ClientHello template the native engine renders (
Profile.tls_template). - The HTTP/2 spec: SETTINGS, WINDOW_UPDATE, priority and pseudo-header order (
Profile.h2_spec). - Two header sets: the HTTP/1.1 headers in browser order and spelling (
Profile.headers) and the HTTP/2 headers with lowercase names (Profile.h2_headers), as captured for that target. The transport uses the set that matches the connection’s protocol, so Chrome 124 and newer and Safari 18.0 and newer sendpriority: u=0, iover HTTP/2 and not over HTTP/1.1. - OpenSSL fallback settings: a TLS 1.2 cipher order for the family, minimum TLS 1.2 and an
X25519curve hint, used only on the OpenSSL path.
The header sets advertise compression as the browser does. 41 of the 79 names send Accept-Encoding: gzip, deflate, br, zstd. Decoding a zstd body needs the zstd extra on Python 3.13 and older (Python 3.14 has compression.zstd); without it such a response raises ContentDecodingError.
reap.FINGERPRINTS holds 79 names: 38 concrete captures, 19 aliases and 22 legacy 1.x names. The legacy names are approximations: they reuse the TLS template, HTTP/2 spec and headers of the nearest capture and only change the version in the User-Agent and sec-ch-ua. The full list is on the Impersonation targets page, and reap doctor <target> checks a target’s fingerprints from your machine (see Command line).
Lookup is forgiving about spelling: Chrome124 and CHROME124 resolve to chrome124, safari17.0 resolves to safari17_0, and safari26_0 resolves to safari260. An unknown name raises curl_reap.tls.UnknownProfileError, which is both a ValueError and a curl_reap.exceptions.ImpersonateError.
profile= also accepts a dict (or object) shaped like a curl_cffi fingerprint, with client, client_version and os fields. It is mapped to the nearest captured target of that browser family and platform:
reap.get("https://example.com", profile={"client": "chrome", "client_version": "120", "os": "macos"})
# uses chrome120
The rules when both profile and impersonate appear:
- On
Session(...),impersonatewins overprofile. - On a single request, passing both raises
ValueErrorunless they resolve to the same profile. - A per-request
profileorimpersonateoverrides the session’s profile for that request only.
Custom fingerprints per request
Three request keywords build a one-request variant of the current profile. They are accepted by reap.get, Session.get and the other request methods, not by the Session constructor.
ja3="771,4865-4866-...,0-23-65281-...,4588-29-23-24,0": a JA3 string. The native engine builds a ClientHello from it. JA3 strips GREASE, so the result sends none; extension bodies use Chrome’s defaults; key shares are X25519 plus the first post-quantum group listed.akamai="1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p": an Akamai HTTP/2 fingerprint string, used as the HTTP/2 SETTINGS, WINDOW_UPDATE, priority and pseudo-header order.extra_fp={"tls_cipher_suites": [...], "tls_curves": [...]}: cipher order and curve for the OpenSSL fallback path.
r = reap.get(
"https://example.com",
impersonate="chrome",
ja3="771,4865-4866-4867-49195-49199,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17613,4588-29-23-24,0",
akamai="1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p",
)
The header set stays that of the base profile.
Headers
Your headers merge into the profile’s header set for the connection’s protocol (the HTTP/1.1 set, or the captured HTTP/2 set), case-insensitively:
- A header you set that the profile already has replaces the value in the same position, using your casing.
- A new header is appended after the profile’s headers.
- Session headers apply to every request; per-request headers are merged on top of them.
- A per-request header set to
Nonedrops the session header of that name for that request. In 2.0.0 it does not remove a header that comes from the profile. To send fewer headers than a profile has, use a custom profile.
s = reap.Session(headers={"Accept-Language": "de-DE,de;q=0.9", "X-Trace": "1"})
r = s.get("https://example.com", headers={"User-Agent": "my-bot/1.0", "X-Trace": None})
In this example the request keeps the profile’s header order, replaces the profile’s User-Agent and Accept-Language values in place, and does not send X-Trace. Header names containing : or newlines, and values containing newlines, raise ValueError.
Custom profiles
reap.Profile is a frozen dataclass. Pass an instance anywhere a profile name is accepted:
from curl_reap import Profile, Session
mine = Profile(
name="my-bot",
headers=(
("User-Agent", "my-bot/1.0 (+https://example.com/contact)"),
("Accept", "text/html"),
("Accept-Encoding", "gzip, deflate"),
),
)
s = Session(profile=mine)
A hand-built profile has no tls_template, so it always uses the OpenSSL path, and no h2_spec, so the transport offers only http/1.1 in ALPN and the connection speaks HTTP/1.1.
Fields and defaults:
name: required, a non-empty string.family: default"custom".version: default"".headers: ordered(name, value)pairs for HTTP/1.1. Default().h2_headers: HTTP/2 header pairs;None(default) lowercasesheaders.tls_template,h2_spec: names of a captured ClientHello template and HTTP/2 spec. DefaultNone.accept_encoding,form_boundary: derived from the headers and family when empty.ciphers: OpenSSL cipher names for TLS 1.2 and below, in order. Default()keeps OpenSSL’s defaults.alpn_protocols: default("h2", "http/1.1"); must not be empty. It is validated but not read: the transport offersh2andhttp/1.1when the profile has an HTTP/2 spec, andhttp/1.1otherwise.curve: default"X25519";Noneleaves OpenSSL’s default group list in place.minimum_version: default"TLSv1_2".
To change only the User-Agent while keeping a browser’s TLS and HTTP/2 fingerprint, set it as a session or request header on a built-in profile instead.
Rotation
A session can rotate profiles and proxies per request:
s = reap.Session(rotate="random") # chrome, firefox, safari
s = reap.Session(rotate="sequence", profiles=["chrome124", "firefox133", "safari17_0"])
rotateisNone(no rotation),"sequence"or"random".- With
rotateset and noprofiles, the pool is the three entries inreap.PROFILES. profiles=[...]sets your own pool of names orProfileobjects. A pool with more than one entry rotates even withoutrotate, in sequence.- Every attempt, including retries, picks the next profile and the next proxy from the pools.
sticky=Truepins one profile and one proxy for the session’s life.
Proxy pools and block rotation are covered in Sessions, retries, proxies and cookies.
Detecting challenge pages
Response.looks_like_challenge is True when the first 6000 characters of the body contain a common bot-wall marker, such as Cloudflare’s “Just a moment” interstitial. Response.block_reason is None, "rate_limited" (429), "proxy_auth_required" (407), "forbidden" (403) or "challenge". Both are heuristics. See JavaScript rendering for pages that need a browser, and Responsible use before working around a challenge.
Performance
bench/AFTER-2.0.md in the repository compares curl_reap 2.0.0 with curl_cffi 0.16.3 on one machine (Apple M3 Pro, Python 3.12), with every server on 127.0.0.1:
| Scenario | curl_reap / curl_cffi |
|---|---|
| HTTP/1.1 keep-alive, 1 connection | 1.33x |
| HTTP/1.1 keep-alive, 16 threads | 1.78x |
| HTTPS fresh connections (a full handshake each) | 0.48x |
| HTTPS keep-alive, 1 connection | 1.16x |
| HTTP/2 keep-alive, 1 connection | 0.75x |
| 5 MB bodies over HTTP/1.1 | 1.20x |
curl_reap is slower than curl_cffi on fresh TLS handshakes (about 1 ms per handshake in Python, against about 0.45 ms) and on HTTP/2. PSK resumption shortens handshakes where servers issue tickets. These are loopback numbers on one machine; measure on your own workload.
Errors
All transport errors derive from reap.TransportError, which is an OSError:
curl_reap.exceptions.ConnectTimeout,ReadTimeout: a deadline passed.curl_reap.exceptions.SSLError: a TLS handshake failure;CertificateVerifyError(a subclass) for a failed certificate check. Errors raised by the native engine areTransportErrorsubclasses with curl’s error codes (35 for a TLS failure, 60 for verification).curl_reap.exceptions.ProxyError,DNSError,TooManyRedirects,ContentDecodingErrorand others for the matching failures.reap.HTTPStatusError: raised byResponse.raise_for_status(). The response is on.response.ValueError: unknown profile, invalid header, an invalidtlsvalue, orcert=withtls="native".TypeError: an unexpected keyword argument on a request.
Connection, TLS and framing errors are retried by the session’s retry policy before they are raised, for the methods the policy allows (by default GET, HEAD, OPTIONS, PUT and DELETE).