Skip to content
curl_reap

curl_reap / Project

Changelog

Releases and notable changes in curl_reap, from the 0.1.0 PyPI release through the 1.0.0 transport rewrite to the 2.0.0 native TLS 1.3 release.

On this page
  1. First principles
  2. 2.0.0 (2026-10-26, GitHub release and PyPI)
  3. Added
  4. Fixed
  5. Fixed before the release
  6. Changed
  7. Known limitations
  8. 1.1.0 (2026-09-30, GitHub release)
  9. 1.0.0 (2026-08-22, GitHub release)
  10. 0.3.0 (never released)
  11. 0.2.2 (2026-07-15, PyPI)
  12. 0.2.1 (2026-07-15, PyPI)
  13. 0.2.0 (2026-07-08, GitHub release and PyPI)
  14. 0.1.2 (2026-06-30, PyPI)
  15. 0.1.1 (2026-06-30, PyPI)
  16. 0.1.0 (2026-06-30, PyPI)

The repository has a CHANGELOG.md since 2.0.0. It covers 2.0.0 and 1.1.0 in detail and points here for older history. This page also uses git tags, GitHub releases, Python Package Index (PyPI) upload history and commit messages. Dates are Coordinated Universal Time (UTC) upload or release dates; merge dates are named as such.

First principles

A version can live in three places, and they move at different speeds. Code merges to main first. A git tag and GitHub release mark a version as released. PyPI is what a plain pip install reads. They have not always agreed: 1.0.0 and 1.1.0 were released on GitHub only, so PyPI went from 0.2.2 straight to 2.0.0. Check reap.__version__ after installing.

Where things are published:

  • GitHub releases: v0.2.0, v1.0.0, v1.1.0 and v2.0.0. The first three have a wheel and a source distribution (sdist) attached.
  • PyPI: 0.1.0, 0.1.1, 0.1.2, 0.2.0, 0.2.1, 0.2.2 and 2.0.0. PyPI serves 2.0.0.

2.0.0 (2026-10-26, GitHub release and PyPI)

The transport is new from the TLS record layer up. A pure-Python TLS 1.3 client built on cryptography primitives renders each impersonate target’s captured ClientHello byte for byte: cipher order, extension order and GREASE, X25519MLKEM768 post-quantum key shares, certificate compression, ALPS, ECH GREASE and padding. The built-in HTTP/2 framing reproduces each browser’s SETTINGS, WINDOW_UPDATE, priority and HPACK representations. The project reports that the measured JA4, ja4_r, peetprint and Akamai HTTP/2 fingerprints equal the captured browsers’ for all 53 captured target names (38 browser builds plus 15 aliases captured under their own names).

This is the first PyPI release since 0.2.2, which was still built on curl_cffi. 1.0.0 and 1.1.0 were GitHub releases only. Read the Changed list below before upgrading.

The 2.0 code was merged to main on 2026-10-11 in #29. Fixes from #30, #33 and #34, merged on 2026-10-22, and two fixes on the release branch followed before the release.

Added

  • Native TLS 1.3 engine (curl_reap._tls): HelloRetryRequest, KeyUpdate, PSK resumption with a thread-safe session cache (tickets are drained on close, so resumption survives short-lived connections), strict RFC 5280 certificate verification through the cryptography policy API, and an automatic OpenSSL fallback for TLS 1.2-only servers or client certificates. Choose with Session(tls="auto"|"native"|"openssl"). Response.infos["tls_backend"] reports what a connection used.
  • Built-in HTTP/2 (curl_reap._h2): on by default, byte-exact per target, content-decoded bodies, streamed downloads, GOAWAY-aware retry classification and flow control on both windows. The h2 extra is now an empty alias. See HTTP/2.
  • 79 impersonate targets in reap.FINGERPRINTS (38 concrete captures, 19 aliases, 22 legacy names), with the exact per-target HTTP/1.1 and HTTP/2 header sets. See Impersonation targets.
  • reap doctor [target]: fetch a fingerprint echo service as the target and compare ja4, ja4_r, ja3 (where the browser’s JA3 is stable), the Akamai HTTP/2 fingerprint and the header order with the captured values; peetprint is printed. It accepts all 79 names, and reap doctor --list prints them. See Command line.
  • WebSocket client: reap.WebSocket.connect, Session.ws_connect, AsyncSession.ws_connect and reap.AsyncWebSocket. RFC 6455 over the same transport, with masking, fragmentation, automatic pong replies and close codes. See WebSocket.
  • More request keywords: files, cookies, stream, proxies (dict forms), proxy_auth, cert, timeout=(connect, read) tuples, allow_redirects="safe" (refuses private, loopback and link-local redirect targets), interface, debug, and ja3, akamai and extra_fp custom fingerprints. Compatibility keywords that have no effect are accepted with one warning. trust_env honors the *_proxy and NO_PROXY environment variables.
  • Headers, an ordered, case-insensitive multi-dict for requests and responses, and an RFC 6265bis cookie parser and jar (curl_reap.cookies) scoped by domain, path and name.
  • Response additions: lazy text, settable encoding, is_success, redirect_url, history, block_reason, elapsed as a timedelta plus elapsed_seconds, endpoint and TLS details in infos, iter_content, iter_lines and close for streamed bodies, and pickling support.
  • Session controls: url_policy= (checked before every outgoing URL, including each redirect hop), sticky=True (one profile and proxy for the session’s life) and RetryPolicy(allowed_methods=...). See Sessions, retries, proxies and cookies.
  • Crawl engine controls: collect_items=False, per_domain_concurrency, max_pending (a bounded queue), dedup when a request is scheduled, a max_pages budget that counts failed fetches, and Retry-After cooldowns on 429 and 503. See Crawling.
  • reap.BrowserSession (js extra): one browser context reused across goto() calls, the real navigation status, an optional cdp_url attach and url_policy gating. Also extract() for named fields and markdown(main_only=True). See JavaScript rendering.
  • Private job service in a new server extra: reap serve runs a FastAPI job API and reap worker runs a Redis queue worker with leases and fencing tokens, for single-page extraction jobs, plus a non-root Dockerfile. See Private job service.

Fixed

  • Every P1 defect and all but one P2 defect from the 2026-10-10 transport audit. The itemized closure is in docs/STATE.md in the repository; the open one is under Known limitations below. The fixes include Partitioned and unknown cookie attributes, HTTP/2 bodies returned compressed, HTTP/2 request bodies above 16 KB, pooled connections after GOAWAY, download bodies on error statuses, multi-member gzip, UTF-8 redirect targets, torn cache entries, pool liveness and bounds, whole-request deadlines, size caps and CONNECT host:port.
  • HTTP/2 responses are content-decoded, and downloads stream to disk in constant memory on every transport.
  • Retry-After values are clamped to 24 hours, so inf no longer crashes and very large values no longer sleep for real.
  • Redirects that change origin drop Authorization, Cookie, Proxy-Authorization and Host, including an https to http downgrade on the same host.

Fixed before the release

These defects were in 2.0 code on main that no earlier release shipped. They were fixed between the 2.0 merge and the release.

  • TLS trust settings are part of connection reuse (#29). A pooled connection or a TLS 1.3 resumption ticket is never reused across different trust settings: a different verify value (a boolean, a CA bundle path or an SSLContext) or a different client cert. Before the fix, a ticket earned on a verify=False request let a later verified request to the same host resume without certificate verification, because PSK resumption skips the certificate check. Keep-alive connections were also reused across CA bundles, and client certificates were not part of the pool key. Only the 2.0 native engine was affected; 1.1.0 is not. Same-policy pooling and resumption still work.
  • Python 3.14 import deadlock (#29). The native TLS and HTTP/2 engines now load when the package is imported. Before, the first HTTP/2 request imported them from a request thread, which could deadlock on Python 3.14’s per-module import locks when another thread imported hpack at the same time.
  • HTTP/2 GOAWAY after a complete response (#29). When a response and an error GOAWAY arrive in the same read, the fully received response is returned and the GOAWAY only ends the connection. It used to be discarded with an error. An error GOAWAY that arrives before the response still raises.
  • Faster HTTP/2 request header validation (#29). Method, path, header name and header value checks use precompiled regular expressions. The rejected inputs are unchanged.
  • stream=True over HTTP/1.1 works (#30). In the 2.0 merge it returned no body: content was None and iter_content() yielded nothing. Streamed bodies now decode gzip, deflate and br as they arrive, report truncated compressed bodies, keep memory bounded whatever the server’s chunk sizes, and return the connection to the pool when drained. An encoding that cannot be streamed raises ContentDecodingError instead of returning still-encoded bytes. See Streaming responses.
  • Response.infos["tls_backend"] is set (#30) on every connection: "native", "openssl" or "plain". In the 2.0 merge reading it raised KeyError.
  • reap doctor checks the named target (#33). It measures the target’s own traffic (it used Chrome for every target), shows ja3 as n/a instead of failing for Chromium targets, compares the header order, accepts all 79 names including aliases and legacy names, honors --impersonate, names an unknown target plainly, and --list prints every name. See Command line.
  • Each target’s captured HTTP/2 header set is sent (#34). The transport used to send the HTTP/1.1 set lowercased over HTTP/2, so the Chrome (124 and newer) and Safari (18.0 and newer) targets sent no priority header over HTTP/2. See HTTP/2.
  • Client certificates work under tls="auto" (#34). A request with cert= goes through OpenSSL and presents the certificate. Before, the native engine sent an empty certificate. tls="native" with cert= raises ValueError. The (cert, key, password) form works, and paths may be os.PathLike. See Client certificates.
  • The TLS 1.2 fallback stays on the proxy (release branch). When the native handshake found a TLS 1.2-only server, the OpenSSL fallback re-dialed the server directly instead of through the configured proxy. Through a Socket Secure (SOCKS) proxy the request left from the machine’s own address; through an HTTP proxy a direct connection was opened and dropped before the tunnelled one. The fallback now reconnects the same way as the first attempt. With an https:// proxy, which the OpenSSL path cannot run through, it fails before dialing anything. See TLS 1.2 servers and proxies.
  • Custom profiles offer only HTTP/1.1 (release branch). A hand-built Profile has no HTTP/2 spec but offered h2 in Application-Layer Protocol Negotiation (ALPN), so a server that picked HTTP/2 received an HTTP/1.1 request and the request failed. Such profiles now offer http/1.1 only. Built-in targets are unchanged. See Custom profiles.
  • The CLI help for --http2 says it is a no-op, and the Geocoder’s default User-Agent says curl_reap/2.0.

Changed

  • HTTP/2 is negotiated by default. Session(http2=False) opts out. The CLI --http2 flag is accepted but no longer changes anything.
  • Response.ok is True for 2xx and 3xx. Use is_success for 2xx only. The get, meta and links commands use ok for their exit status.
  • RetryPolicy retries only GET, HEAD, OPTIONS, PUT and DELETE by default. POST and PATCH need allowed_methods.
  • max_redirects defaults to 30 (it was 10).
  • The crawl engine sends responses with status 400 or higher, and challenge-looking responses, to the request’s errback instead of parse.
  • cryptography>=42, hpack>=4 and brotli>=1.1 are required. zstandard is an optional zstd extra (pip install "curl-reap[zstd]") for zstd bodies and certificate compression.
  • Exceptions mirror the wider ecosystem’s names and error codes, while TransportError and OSError stay as base classes. HTTPStatusError is now an alias of HTTPError, an OSError subclass (it was a plain Exception).

Known limitations

  • No native TLS 1.2. TLS 1.2-only servers get an OpenSSL-shaped ClientHello. That fallback offers only the X25519 group (Profile.curve), so a TLS 1.2-only server with a P-256 Elliptic Curve Digital Signature Algorithm (ECDSA) certificate fails the handshake; servers with RSA certificates work.
  • An https:// proxy works only on the native TLS path. The OpenSSL path (TLS 1.2-only servers, cert=, custom profiles without a template, tls="openssl") fails through one; use an http:// or SOCKS proxy there.
  • A per-request header set to None removes a session header but not one that comes from the profile. Use a custom Profile to send fewer headers.
  • HTTP/3 (QUIC) is not implemented.

Troubleshooting lists these with the other known issues.

1.1.0 (2026-09-30, GitHub release)

Fixes the reap command line tool and ships the features added since 1.0.0. The release has 67 passing tests.

  • Command line fix. The reap command works again. In 1.0.0 it failed at startup with a SyntaxError in curl_reap/cli.py. Command-line interface (CLI) tests were added, and continuous integration (CI) now runs reap --help.
  • HTTP/2. Optional HTTP/2 transport through hyper-h2: Session(http2=True), reap --http2, and a new h2 extra. Falls back to HTTP/1.1 when the server or the environment does not support it. See HTTP/2.
  • Versioned impersonation targets. impersonate= is back as an alias of profile=, with 38 versioned Chrome, Edge, Firefox and Safari targets plus four aliases in reap.FINGERPRINTS, and a CLI --impersonate option. Conflicting profile and impersonate values on a request raise ValueError. See Impersonation targets.
  • Cookie jar. Session.cookies is a CookieJar that stores cookies from every response and redirect hop and replays them. Exported as reap.CookieJar.
  • Streaming downloads. Session.download(), reap.download() and AsyncSession.download() stream to disk in constant memory with an atomic part-file rename. See Caching and downloads.
  • Challenge detection. Response.looks_like_challenge.
  • JavaScript rendering. reap.render() and reap.render_if_empty() through Playwright, in a new js extra. See JavaScript rendering.
  • Exports. reap.TransportError is exported at the package root.
  • Tests and CI. An adversarial transport test suite, and a CI matrix for Python 3.9 to 3.13.

1.0.0 (2026-08-22, GitHub release)

The from-scratch, breaking release.

  • New transport. curl_cffi is removed. curl_reap now has its own HTTP/1.1 engine on the standard library’s socket and ssl, with byte-level control of the request line and header order and casing, keep-alive pooling, chunked decoding, and gzip and deflate decoding.
  • TLS and header profiles. curl_reap.tls.Profile with curated cipher orderings, ALPN and browser-family headers for chrome, firefox and safari. Custom profiles are supported.
  • Breaking: impersonate= was removed in favor of profile= (reap.get(url, profile="chrome")). It was added back as an alias on main after this release.
  • Retries with exponential backoff and jitter, proxy rotation, and profile rotation across retries.
  • Dependencies are now only lxml and cssselect. Python 3.9+.
  • Selectors, structured extraction, the crawl engine, the disk cache, the async facade, pipelines and the CLI carry over.

0.3.0 (never released)

The version was bumped to 0.3.0 in the repository on 2026-07-19 but was not tagged or uploaded. Its changes shipped in 1.0.0:

  • Encoding detection from the byte order mark, Content-Type and <meta charset>, exposed as Response.encoding.
  • Conditional cache revalidation with If-None-Match and If-Modified-Since.
  • robots.txt Crawl-delay as a per-domain floor, and Reaper.discovered_sitemaps.
  • Gzipped sitemaps in SitemapSpider.
  • Automatic proxy rotation on 403, 407 and 429 with a proxy pool.

0.2.2 (2026-07-15, PyPI)

  • A one-time terminal tip suggesting proxy rotation, and NodeMaven, when a request without a proxy is blocked with 403, 407 or 429. It could be silenced with REAP_NO_TIPS=1. This tip is not part of the 1.0.0 transport.
  • A tracked sponsor link in the PyPI description.

0.2.1 (2026-07-15, PyPI)

  • The README sponsor section with NodeMaven, published to PyPI.

0.2.0 (2026-07-08, GitHub release and PyPI)

A major upgrade into a full-stack scraper; the 0.1 API kept working.

  • Transport: RetryPolicy with backoff, jitter and Retry-After; fingerprint and proxy rotation; DiskCache; AsyncSession, aget and apost; richer Response (status_code, cookies, urljoin(), follow(), raise_for_status()).
  • Parsing: jsonld(), meta_tags(), links(), images(), tables(), markdown(), re_first(), and a find_by_text that returns the deepest match.
  • Crawl engine: a continuous priority-queue scheduler, per-domain AutoThrottle that backs off on 429 and 503, opt-in robots.txt, allowed_domains and max_depth, errbacks, and stats and logging.
  • Spiders and pipelines: Request(priority=, errback=, dont_filter=), SitemapSpider, SqlitePipeline with automatic columns and upsert, and a thread-safe JsonLinesPipeline.
  • CLI: the reap command with get, meta, links and crawl.

0.1.2 (2026-06-30, PyPI)

Uploaded to PyPI. There is no matching tag or commit message in the repository.

0.1.1 (2026-06-30, PyPI)

  • An absolute logo URL so the README renders on PyPI.

0.1.0 (2026-06-30, PyPI)

The first release. Browser TLS and JA3 impersonation through curl_cffi, lxml selectors with self-healing auto_match selectors, and a concurrent crawl engine with AutoThrottle, retries, dedup and pipelines.

Between 0.1.1 and 0.2.0, the repository also gained the Geocoder, the Booking.com example, LEGAL.md and the first documentation site.