Skip to content
curl_reap

curl_reap / Guides

Transport and impersonation profiles

How the curl_reap 2.0 transport works: native TLS 1.3, OpenSSL fallback, pooling, resumption, impersonate targets and custom fingerprints.

On this page
  1. First principles
  2. How the transport works
  3. Choosing the TLS backend
  4. What Response.infos holds
  5. Client certificates
  6. Certificate verification
  7. Connection pooling and resumption
  8. Profiles
  9. Custom fingerprints per request
  10. Headers
  11. Custom profiles
  12. Rotation
  13. Detecting challenge pages
  14. Performance
  15. Errors

This page covers the transport: how a Hypertext Transfer Protocol (HTTP) request is put on the wire, which Transport Layer Security (TLS) backend handles it, how connections are reused, and how impersonation profiles shape every layer. How it works explains the reasoning behind the design.

First principles

  • A server sees three layers of every request: the TLS ClientHello, the HTTP/2 connection preface and the request headers. Each one can be compared with a real browser’s.
  • The layers must agree. A browser User-Agent on a non-browser handshake is easy to spot, so a profile sets all three layers together. Changing one layer by hand (a different User-Agent, a ja3= string) can break that agreement.
  • A fingerprint can only be recorded, not derived. Each built-in target replays bytes captured from one browser version. Aliases and legacy names reuse those captures; they add no new fingerprints.
  • Reuse must not cross trust settings. Pooled connections and resumption tickets skip work that was done under one verify and cert setting, so they are keyed by that setting.

How the transport works

curl_reap 2.0 has its own transport, from the TLS record layer up. There is no compiled curl binding.

  • Native TLS 1.3. A pure-Python TLS 1.3 client (curl_reap._tls) built on the cryptography package. For each of the 38 captured targets it renders the ClientHello captured from that browser byte for byte (aliases use their target’s capture, legacy names the nearest one): cipher order, extension set and order, Generate Random Extensions And Sustain Extensibility (GREASE) values, key shares including the X25519MLKEM768 post-quantum hybrid, certificate compression (zlib, brotli, zstd), Application-Layer Protocol Settings (ALPS), Encrypted Client Hello (ECH) GREASE and the padding rule. It handles HelloRetryRequest, KeyUpdate, session tickets and pre-shared key (PSK) resumption.
  • OpenSSL fallback. When a server does not pick TLS 1.3, the engine stops after the ServerHello, and the transport reconnects through the standard library’s ssl module. That connection sends an OpenSSL-shaped ClientHello; only the HTTP layer is the browser’s. The reconnect takes the same route as the first attempt, through the same proxy, but it cannot run through an https:// proxy. See TLS 1.2 servers and proxies.
  • Built-in HTTP/2. curl_reap._h2 sends each target’s SETTINGS, WINDOW_UPDATE, priority and pseudo-header order, with header compression (HPACK) from the hpack package. HTTP/2 is offered through Application-Layer Protocol Negotiation (ALPN) by default and falls back to HTTP/1.1 per connection. See HTTP/2.
  • HTTP/1.1. The transport writes the request line and every header itself, so header order and casing are exactly what the profile and your code specify. It decodes chunked bodies and gzip, deflate, br and zstd content encodings.
  • Connections. HTTP CONNECT proxies (always host:port), Socket Secure (SOCKS) 4, 4a, 5 and 5h proxies, redirects (301, 302, 303, 307, 308), and cookies.

Timeouts are deadlines. A float timeout bounds one exchange as a whole: connect, send and receive. timeout=(connect, read) bounds the connect and each socket read, and sets an overall deadline of their sum. The deadline starts again on each redirect hop and each retry. None disables the clock.

Choosing the TLS backend

Session(tls=...) selects the backend:

Value Behavior
"auto" (default) Native engine when the profile has a captured ClientHello template, OpenSSL otherwise. Requests with a client certificate (cert=) use OpenSSL. Falls back to OpenSSL when the server does not speak TLS 1.3, and when the native handshake fails for another non-certificate reason.
"native" Native engine. A native handshake failure is raised instead of retried over OpenSSL. A server that only speaks TLS 1.2 is still reconnected over OpenSSL, because there is no native TLS 1.2. A request with cert= raises ValueError.
"openssl" Always the standard library ssl module. Profiles still set the HTTP headers and HTTP/2 framing.
import curl_reap as reap

s = reap.Session(impersonate="chrome", tls="auto")
r = s.get("https://example.com")
print(r.infos["tls_backend"], r.http_version, r.infos.get("tls_version"), r.infos.get("group"))
# for example: native HTTP/2 TLSv1.3 X25519MLKEM768

Response.infos["tls_backend"] is what a connection used: "native", "openssl" or "plain" (no TLS). Session.tls_backend is the mode you configured ("auto", "native" or "openssl").

What Response.infos holds

Response.infos is a dict with details of the connection a response came over. The keys depend on the path:

Key Native engine OpenSSL path
tls_backend "native" "openssl" ("plain" on http://)
tls_version "TLSv1.3" the negotiated version, for example "TLSv1.2"
cipher the TLS 1.3 suite the negotiated suite
alpn "h2", "http/1.1" or None same
group the key exchange group, for example "X25519MLKEM768" None
resumed True when the handshake resumed a session ticket not present
handshake_seconds handshake duration not present
peername, sockname remote and local address same

HTTP/2 responses also carry infos["http_version"] = "HTTP/2".

Client certificates

cert= takes the path of a Privacy-Enhanced Mail (PEM) file holding the certificate and key, or a (cert, key) or (cert, key, password) tuple, on Session or per request. Paths can be strings or os.PathLike objects.

The native engine cannot present a client certificate, so a request with cert= goes through OpenSSL:

  • With tls="auto" (the default) or tls="openssl", the request uses OpenSSL and presents the certificate, both to servers that require one and to servers that only ask. r.infos["tls_backend"] is "openssl".
  • With tls="native", the request raises ValueError instead of being sent without the certificate.
s = reap.Session(cert=("client.pem", "client.key"))
s = reap.Session(cert=("client.pem", "client.key.enc", "key-password"))

The handshake for these requests is OpenSSL-shaped; the HTTP/2 framing and headers still follow the profile. The OpenSSL path cannot run through an https:// proxy, so use an http:// or SOCKS proxy with cert=.

Certificate verification

verify is True (the default), False, or the path of a certificate authority (CA) bundle, on Session or per request. On the native path, chain building and policy checks go through the cryptography package’s RFC 5280 web-PKI verifier. Trust anchors come from the interpreter’s OpenSSL default paths, then from certifi when it is installed, then from the usual distribution bundle paths. verify="ca.pem" loads that bundle instead (PEM with one or more certificates, or a single binary DER-encoded certificate).

Connection pooling and resumption

Idle connections are pooled and reused. The pool is bounded (10 idle connections per key, each kept for up to 60 seconds) and checks that a connection is still alive before it reuses it. The pool key is the full route and trust identity:

  • scheme, host and port,
  • proxy,
  • profile name,
  • the verify value,
  • the client cert, with any key password replaced by its SHA-256 digest,
  • the TLS backend,
  • the source interface.

TLS session tickets for PSK resumption are kept in one cache per trust identity, that is per combination of verify and cert (_session_cache_for in curl_reap/transport.py). A resumed handshake skips certificate verification, so a ticket earned under one setting must not resume a connection that asked for another. For example, tickets from requests with verify=False are never used for a request with verify=True, and a connection verified against one CA bundle is never handed to a request that names a different one. A session keeps at most 32 ticket caches and drops the oldest when it needs room.

Profiles

A profile describes how a request looks on the wire. Pass it by name with profile= or impersonate=:

import curl_reap as reap

reap.get("https://example.com", profile="chrome")       # the default
reap.get("https://example.com", impersonate="firefox")
reap.get("https://example.com", impersonate="safari")
reap.get("https://example.com", impersonate="chrome124")

s = reap.Session(impersonate="safari17_0")

The short names live in reap.PROFILES (chrome, firefox, safari). They are aliases of the newest capture of each family: chrome is chrome150, firefox is firefox147 and safari is safari2601.

A built-in profile carries:

  • The TLS ClientHello template the native engine renders (Profile.tls_template).
  • The HTTP/2 spec: SETTINGS, WINDOW_UPDATE, priority and pseudo-header order (Profile.h2_spec).
  • Two header sets: the HTTP/1.1 headers in browser order and spelling (Profile.headers) and the HTTP/2 headers with lowercase names (Profile.h2_headers), as captured for that target. The transport uses the set that matches the connection’s protocol, so Chrome 124 and newer and Safari 18.0 and newer send priority: u=0, i over HTTP/2 and not over HTTP/1.1.
  • OpenSSL fallback settings: a TLS 1.2 cipher order for the family, minimum TLS 1.2 and an X25519 curve hint, used only on the OpenSSL path.

The header sets advertise compression as the browser does. 41 of the 79 names send Accept-Encoding: gzip, deflate, br, zstd. Decoding a zstd body needs the zstd extra on Python 3.13 and older (Python 3.14 has compression.zstd); without it such a response raises ContentDecodingError.

reap.FINGERPRINTS holds 79 names: 38 concrete captures, 19 aliases and 22 legacy 1.x names. The legacy names are approximations: they reuse the TLS template, HTTP/2 spec and headers of the nearest capture and only change the version in the User-Agent and sec-ch-ua. The full list is on the Impersonation targets page, and reap doctor <target> checks a target’s fingerprints from your machine (see Command line).

Lookup is forgiving about spelling: Chrome124 and CHROME124 resolve to chrome124, safari17.0 resolves to safari17_0, and safari26_0 resolves to safari260. An unknown name raises curl_reap.tls.UnknownProfileError, which is both a ValueError and a curl_reap.exceptions.ImpersonateError.

profile= also accepts a dict (or object) shaped like a curl_cffi fingerprint, with client, client_version and os fields. It is mapped to the nearest captured target of that browser family and platform:

reap.get("https://example.com", profile={"client": "chrome", "client_version": "120", "os": "macos"})
# uses chrome120

The rules when both profile and impersonate appear:

  • On Session(...), impersonate wins over profile.
  • On a single request, passing both raises ValueError unless they resolve to the same profile.
  • A per-request profile or impersonate overrides the session’s profile for that request only.

Custom fingerprints per request

Three request keywords build a one-request variant of the current profile. They are accepted by reap.get, Session.get and the other request methods, not by the Session constructor.

  • ja3="771,4865-4866-...,0-23-65281-...,4588-29-23-24,0": a JA3 string. The native engine builds a ClientHello from it. JA3 strips GREASE, so the result sends none; extension bodies use Chrome’s defaults; key shares are X25519 plus the first post-quantum group listed.
  • akamai="1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p": an Akamai HTTP/2 fingerprint string, used as the HTTP/2 SETTINGS, WINDOW_UPDATE, priority and pseudo-header order.
  • extra_fp={"tls_cipher_suites": [...], "tls_curves": [...]}: cipher order and curve for the OpenSSL fallback path.
r = reap.get(
    "https://example.com",
    impersonate="chrome",
    ja3="771,4865-4866-4867-49195-49199,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17613,4588-29-23-24,0",
    akamai="1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p",
)

The header set stays that of the base profile.

Headers

Your headers merge into the profile’s header set for the connection’s protocol (the HTTP/1.1 set, or the captured HTTP/2 set), case-insensitively:

  • A header you set that the profile already has replaces the value in the same position, using your casing.
  • A new header is appended after the profile’s headers.
  • Session headers apply to every request; per-request headers are merged on top of them.
  • A per-request header set to None drops the session header of that name for that request. In 2.0.0 it does not remove a header that comes from the profile. To send fewer headers than a profile has, use a custom profile.
s = reap.Session(headers={"Accept-Language": "de-DE,de;q=0.9", "X-Trace": "1"})
r = s.get("https://example.com", headers={"User-Agent": "my-bot/1.0", "X-Trace": None})

In this example the request keeps the profile’s header order, replaces the profile’s User-Agent and Accept-Language values in place, and does not send X-Trace. Header names containing : or newlines, and values containing newlines, raise ValueError.

Custom profiles

reap.Profile is a frozen dataclass. Pass an instance anywhere a profile name is accepted:

from curl_reap import Profile, Session

mine = Profile(
    name="my-bot",
    headers=(
        ("User-Agent", "my-bot/1.0 (+https://example.com/contact)"),
        ("Accept", "text/html"),
        ("Accept-Encoding", "gzip, deflate"),
    ),
)
s = Session(profile=mine)

A hand-built profile has no tls_template, so it always uses the OpenSSL path, and no h2_spec, so the transport offers only http/1.1 in ALPN and the connection speaks HTTP/1.1.

Fields and defaults:

  • name: required, a non-empty string.
  • family: default "custom".
  • version: default "".
  • headers: ordered (name, value) pairs for HTTP/1.1. Default ().
  • h2_headers: HTTP/2 header pairs; None (default) lowercases headers.
  • tls_template, h2_spec: names of a captured ClientHello template and HTTP/2 spec. Default None.
  • accept_encoding, form_boundary: derived from the headers and family when empty.
  • ciphers: OpenSSL cipher names for TLS 1.2 and below, in order. Default () keeps OpenSSL’s defaults.
  • alpn_protocols: default ("h2", "http/1.1"); must not be empty. It is validated but not read: the transport offers h2 and http/1.1 when the profile has an HTTP/2 spec, and http/1.1 otherwise.
  • curve: default "X25519"; None leaves OpenSSL’s default group list in place.
  • minimum_version: default "TLSv1_2".

To change only the User-Agent while keeping a browser’s TLS and HTTP/2 fingerprint, set it as a session or request header on a built-in profile instead.

Rotation

A session can rotate profiles and proxies per request:

s = reap.Session(rotate="random")                          # chrome, firefox, safari
s = reap.Session(rotate="sequence", profiles=["chrome124", "firefox133", "safari17_0"])
  • rotate is None (no rotation), "sequence" or "random".
  • With rotate set and no profiles, the pool is the three entries in reap.PROFILES.
  • profiles=[...] sets your own pool of names or Profile objects. A pool with more than one entry rotates even without rotate, in sequence.
  • Every attempt, including retries, picks the next profile and the next proxy from the pools.
  • sticky=True pins one profile and one proxy for the session’s life.

Proxy pools and block rotation are covered in Sessions, retries, proxies and cookies.

Detecting challenge pages

Response.looks_like_challenge is True when the first 6000 characters of the body contain a common bot-wall marker, such as Cloudflare’s “Just a moment” interstitial. Response.block_reason is None, "rate_limited" (429), "proxy_auth_required" (407), "forbidden" (403) or "challenge". Both are heuristics. See JavaScript rendering for pages that need a browser, and Responsible use before working around a challenge.

Performance

bench/AFTER-2.0.md in the repository compares curl_reap 2.0.0 with curl_cffi 0.16.3 on one machine (Apple M3 Pro, Python 3.12), with every server on 127.0.0.1:

Scenario curl_reap / curl_cffi
HTTP/1.1 keep-alive, 1 connection 1.33x
HTTP/1.1 keep-alive, 16 threads 1.78x
HTTPS fresh connections (a full handshake each) 0.48x
HTTPS keep-alive, 1 connection 1.16x
HTTP/2 keep-alive, 1 connection 0.75x
5 MB bodies over HTTP/1.1 1.20x

curl_reap is slower than curl_cffi on fresh TLS handshakes (about 1 ms per handshake in Python, against about 0.45 ms) and on HTTP/2. PSK resumption shortens handshakes where servers issue tickets. These are loopback numbers on one machine; measure on your own workload.

Errors

All transport errors derive from reap.TransportError, which is an OSError:

  • curl_reap.exceptions.ConnectTimeout, ReadTimeout: a deadline passed.
  • curl_reap.exceptions.SSLError: a TLS handshake failure; CertificateVerifyError (a subclass) for a failed certificate check. Errors raised by the native engine are TransportError subclasses with curl’s error codes (35 for a TLS failure, 60 for verification).
  • curl_reap.exceptions.ProxyError, DNSError, TooManyRedirects, ContentDecodingError and others for the matching failures.
  • reap.HTTPStatusError: raised by Response.raise_for_status(). The response is on .response.
  • ValueError: unknown profile, invalid header, an invalid tls value, or cert= with tls="native".
  • TypeError: an unexpected keyword argument on a request.

Connection, TLS and framing errors are retried by the session’s retry policy before they are raised, for the methods the policy allows (by default GET, HEAD, OPTIONS, PUT and DELETE).