Troubleshooting and limits

This page collects the errors and surprises users run into, with what causes each one and what to do. Every message and limit here comes from the source on main.

First principles

Most problems fall into one of three groups, and each group has a quick check:

• The request took a different path than you expected. Another TLS backend, another protocol, another version of the library. Check r.infos["tls_backend"], r.http_version and reap.__version__.
• The server answered, but not with the page. A block, a challenge or a rate limit is a normal response with a status and a body. Check r.status and r.block_reason.
• You reached a deliberate limit. Body caps, timeouts, the crawl budget and the queue bound are there to keep a long job from running away. The error message names the limit.

Quick checks

import logging
import curl_reap as reap

logging.basicConfig(level=logging.INFO)     # debug=True logs to "curl_reap.transport"

r = reap.get("https://example.com", debug=True)
print(reap.__version__)                     # "2.0.0" for these docs
print(r.status, r.http_version, r.infos["tls_backend"], r.block_reason)

reap doctor chrome        # compare your fingerprint with the capture

debug=True logs every request and response header line at INFO level. Transport Layer Security (TLS) details are in r.infos; see What Response.infos holds.

Installation

• reap.__version__ is not 2.0.0. A plain pip install curl_reap reads the Python Package Index (PyPI), which serves 0.2.2, an older release built on curl_cffi. Install from the main branch as shown in Installation.
• The git install fails. pip needs git on your PATH to install from a git URL.
• ImportError: JS rendering needs Playwright. Install the js extra, then run playwright install chromium.
• reap serve or reap worker exits with status 2. A required variable (REAP_API_KEY, REAP_ALLOWED_ORIGINS) is missing, the server extra is not installed, an origin is invalid, or Redis could not be set up. The message on standard error says which. See Private job service.

Request errors

• ContentDecodingError on a zstd body. 41 of the 79 targets, including chrome (Chrome 123 and newer) and every Firefox target, advertise zstd in Accept-Encoding. Decoding a buffered zstd body needs the zstd extra on Python 3.13 and older. Streamed bodies (stream=True) and downloads cannot decode zstd at all. For those, send headers={"Accept-Encoding": "gzip, deflate, br"}.
• ValueError: tls='native' cannot send a client certificate. The native engine cannot present a client certificate. Use tls="auto" (the default), which sends cert= requests through OpenSSL. See Client certificates.
• NotImplementedError: HTTP/3 (QUIC) is not implemented. http_version="v3" and "v3only" are accepted names, but HTTP/3 does not exist in curl_reap.
• TypeError: unexpected request keyword argument. The keyword is not a request option. The accepted keywords are listed in Per-request options.
• ValueError: profile and impersonate disagree; pass one. A request passed both with names that resolve to different targets. Pass one of them.
• UnknownProfileError. The target name does not exist. reap doctor --list prints all 79 names.
• TransportError: ... exceeds max_body_bytes=67108864. A buffered body is over the 64 mebibyte (MiB) cap. Raise max_body_bytes for HTTP/1.1, or use download() or stream=True. Over HTTP/2 the buffered cap is fixed. This error is a TransportError, so the session retries it like a network error; pass retries=0 when a large body is expected.
• ConnectTimeout or ReadTimeout. A number timeout bounds connect, send and receive together; a (connect, read) tuple bounds the connect and each read. The clock restarts on every redirect hop and retry. See Timeouts.
• TooManyRedirects on a short chain. With allow_redirects="safe", a redirect to a private, loopback or link-local address is refused with this error. The redirect response is on exc.response.
• URLPolicyError. Your url_policy callable returned a falsy value for the request URL or a redirect hop.
• CookieConflict from s.cookies.get(name). Several cookies share the name on different domains or paths. Pass domain= or path=.

Blocks and challenges

• r.block_reason is "forbidden", "proxy_auth_required" or "rate_limited". The server answered 403, 407 or 429. With a proxy pool, the session already switched proxies up to block_rotations times before returning this. A 429 is also retried with its Retry-After value.
• r.block_reason is "challenge". The body looks like a bot-wall interstitial. render_if_empty does not render these, and curl_reap does not solve them. Read Responsible use.
• A crawl callback never sees some pages. Responses with status 400 or higher and challenge-looking responses go to the request’s errback, not to parse. See Crawling.
• r.ok is True for a redirect. In 2.0, ok covers 200 to 399. Use r.is_success for 2xx only.

Fingerprint checks

• reap doctor shows n/a. ja3 is not comparable for browsers that permute their extensions on each connection, and headers is not comparable when the service did not echo them. An n/a row does not fail the check.
• reap doctor shows DIFFERS. Something changed the bytes between curl_reap and the service. Common causes: a proxy or security product that terminates TLS and sends its own handshake, headers you set yourself, or a ja3= or akamai= override. The expected values are printed to standard error.
• r.infos["tls_backend"] is "openssl". The connection did not use the captured handshake. That happens when the server only speaks TLS 1.2, when the request sends cert=, with a custom Profile that has no template, or with tls="openssl". The HTTP layer still follows the profile.

Known issues on main

These are confirmed from the source on main as of 2026-10-25.

TLS 1.2 servers and proxies

When the native handshake finds that the server only speaks TLS 1.2, the transport reconnects for the OpenSSL fallback. That reconnect does not go back through every kind of proxy:

• SOCKS proxies: the fallback connection goes directly to the server, not through the Socket Secure (SOCKS) proxy, so the request leaves from your own address. Only the first, native attempt goes through the proxy.
• HTTP proxies: a direct connection to the server is opened and closed before the request is tunnelled through the proxy.
• HTTPS proxies: the request fails with TransportError: TLS-1.2-only servers cannot be combined with an https proxy.

Until this is fixed, use tls="openssl" on sessions that must never leave the proxy and may reach TLS 1.2 servers. Those connections use OpenSSL’s handshake from the start, through the proxy, at the cost of the captured TLS fingerprint.

Other known issues

• A custom Profile and HTTP/2. A hand-built profile has no HTTP/2 spec, but the transport still offers h2. If the server picks it, the request is written as HTTP/1.1 on that connection. Pass http2=False with custom profiles.
• Removing a profile header. A per-request header set to None removes a session header but not one that comes from the profile. Use a custom profile to send fewer headers.
• stream=True over HTTP/2 buffers the body and replays it. Pass http2=False for a real stream.
• AsyncWebSocket does not accept keyword arguments on send, send_json and close. See WebSocket.
• Cancelling an async request stops the wait but not a socket call already in progress; the request timeout still bounds it.

Limits by design

• No HTTP/3, no native TLS 1.2, no cleartext HTTP/2, and no HTTP/2 multiplexing: one request per connection at a time.
• Buffered bodies: 64 MiB by default over HTTP/1.1 (max_body_bytes), fixed at 64 MiB over HTTP/2. Downloads have no cap.
• The crawl queue lives in memory. A crash loses pending requests; there is no resume.
• Cookies are scoped by domain, path and name, but public-suffix lists are not checked.
• WebSocket: no permessage-deflate compression, and a single incoming frame is capped at 64 MiB.
• Job service: one page per job, results up to 1 MiB, at most 1000 jobs queued and running.