curl_reap / Project
Changelog
Releases and notable changes in curl_reap, from the 0.1.0 PyPI release through the 1.0.0 transport rewrite to the 2.0.0 native TLS 1.3 release.
On this page
- First principles
- 2.0.0 (2026-10-26, GitHub release and PyPI)
- Added
- Fixed
- Fixed before the release
- Changed
- Known limitations
- 1.1.0 (2026-09-30, GitHub release)
- 1.0.0 (2026-08-22, GitHub release)
- 0.3.0 (never released)
- 0.2.2 (2026-07-15, PyPI)
- 0.2.1 (2026-07-15, PyPI)
- 0.2.0 (2026-07-08, GitHub release and PyPI)
- 0.1.2 (2026-06-30, PyPI)
- 0.1.1 (2026-06-30, PyPI)
- 0.1.0 (2026-06-30, PyPI)
The repository has a CHANGELOG.md since 2.0.0. It covers 2.0.0 and 1.1.0 in detail and points here for older history. This page also uses git tags, GitHub releases, Python Package Index (PyPI) upload history and commit messages. Dates are Coordinated Universal Time (UTC) upload or release dates; merge dates are named as such.
First principles
A version can live in three places, and they move at different speeds. Code merges to main first. A git tag and GitHub release mark a version as released. PyPI is what a plain pip install reads. They have not always agreed: 1.0.0 and 1.1.0 were released on GitHub only, so PyPI went from 0.2.2 straight to 2.0.0. Check reap.__version__ after installing.
Where things are published:
- GitHub releases: v0.2.0, v1.0.0, v1.1.0 and v2.0.0. The first three have a wheel and a source distribution (sdist) attached.
- PyPI: 0.1.0, 0.1.1, 0.1.2, 0.2.0, 0.2.1, 0.2.2 and 2.0.0. PyPI serves 2.0.0.
2.0.0 (2026-10-26, GitHub release and PyPI)
The transport is new from the TLS record layer up. A pure-Python TLS 1.3 client built on cryptography primitives renders each impersonate target’s captured ClientHello byte for byte: cipher order, extension order and GREASE, X25519MLKEM768 post-quantum key shares, certificate compression, ALPS, ECH GREASE and padding. The built-in HTTP/2 framing reproduces each browser’s SETTINGS, WINDOW_UPDATE, priority and HPACK representations. The project reports that the measured JA4, ja4_r, peetprint and Akamai HTTP/2 fingerprints equal the captured browsers’ for all 53 captured target names (38 browser builds plus 15 aliases captured under their own names).
This is the first PyPI release since 0.2.2, which was still built on curl_cffi. 1.0.0 and 1.1.0 were GitHub releases only. Read the Changed list below before upgrading.
The 2.0 code was merged to main on 2026-10-11 in #29. Fixes from #30, #33 and #34, merged on 2026-10-22, and two fixes on the release branch followed before the release.
Added
- Native TLS 1.3 engine (
curl_reap._tls): HelloRetryRequest, KeyUpdate, PSK resumption with a thread-safe session cache (tickets are drained on close, so resumption survives short-lived connections), strict RFC 5280 certificate verification through thecryptographypolicy API, and an automatic OpenSSL fallback for TLS 1.2-only servers or client certificates. Choose withSession(tls="auto"|"native"|"openssl").Response.infos["tls_backend"]reports what a connection used. - Built-in HTTP/2 (
curl_reap._h2): on by default, byte-exact per target, content-decoded bodies, streamed downloads, GOAWAY-aware retry classification and flow control on both windows. Theh2extra is now an empty alias. See HTTP/2. - 79 impersonate targets in
reap.FINGERPRINTS(38 concrete captures, 19 aliases, 22 legacy names), with the exact per-target HTTP/1.1 and HTTP/2 header sets. See Impersonation targets. reap doctor [target]: fetch a fingerprint echo service as the target and compare ja4, ja4_r, ja3 (where the browser’s JA3 is stable), the Akamai HTTP/2 fingerprint and the header order with the captured values; peetprint is printed. It accepts all 79 names, andreap doctor --listprints them. See Command line.- WebSocket client:
reap.WebSocket.connect,Session.ws_connect,AsyncSession.ws_connectandreap.AsyncWebSocket. RFC 6455 over the same transport, with masking, fragmentation, automatic pong replies and close codes. See WebSocket. - More request keywords:
files,cookies,stream,proxies(dict forms),proxy_auth,cert,timeout=(connect, read)tuples,allow_redirects="safe"(refuses private, loopback and link-local redirect targets),interface,debug, andja3,akamaiandextra_fpcustom fingerprints. Compatibility keywords that have no effect are accepted with one warning.trust_envhonors the*_proxyandNO_PROXYenvironment variables. Headers, an ordered, case-insensitive multi-dict for requests and responses, and an RFC 6265bis cookie parser and jar (curl_reap.cookies) scoped by domain, path and name.- Response additions: lazy
text, settableencoding,is_success,redirect_url,history,block_reason,elapsedas atimedeltapluselapsed_seconds, endpoint and TLS details ininfos,iter_content,iter_linesandclosefor streamed bodies, and pickling support. - Session controls:
url_policy=(checked before every outgoing URL, including each redirect hop),sticky=True(one profile and proxy for the session’s life) andRetryPolicy(allowed_methods=...). See Sessions, retries, proxies and cookies. - Crawl engine controls:
collect_items=False,per_domain_concurrency,max_pending(a bounded queue), dedup when a request is scheduled, amax_pagesbudget that counts failed fetches, andRetry-Aftercooldowns on 429 and 503. See Crawling. reap.BrowserSession(jsextra): one browser context reused acrossgoto()calls, the real navigation status, an optionalcdp_urlattach andurl_policygating. Alsoextract()for named fields andmarkdown(main_only=True). See JavaScript rendering.- Private job service in a new
serverextra:reap serveruns a FastAPI job API andreap workerruns a Redis queue worker with leases and fencing tokens, for single-page extraction jobs, plus a non-rootDockerfile. See Private job service.
Fixed
- Every P1 defect and all but one P2 defect from the 2026-10-10 transport audit. The itemized closure is in
docs/STATE.mdin the repository; the open one is under Known limitations below. The fixes include Partitioned and unknown cookie attributes, HTTP/2 bodies returned compressed, HTTP/2 request bodies above 16 KB, pooled connections after GOAWAY, download bodies on error statuses, multi-member gzip, UTF-8 redirect targets, torn cache entries, pool liveness and bounds, whole-request deadlines, size caps and CONNECThost:port. - HTTP/2 responses are content-decoded, and downloads stream to disk in constant memory on every transport.
Retry-Aftervalues are clamped to 24 hours, soinfno longer crashes and very large values no longer sleep for real.- Redirects that change origin drop
Authorization,Cookie,Proxy-AuthorizationandHost, including anhttpstohttpdowngrade on the same host.
Fixed before the release
These defects were in 2.0 code on main that no earlier release shipped. They were fixed between the 2.0 merge and the release.
- TLS trust settings are part of connection reuse (#29). A pooled connection or a TLS 1.3 resumption ticket is never reused across different trust settings: a different
verifyvalue (a boolean, a CA bundle path or anSSLContext) or a different clientcert. Before the fix, a ticket earned on averify=Falserequest let a later verified request to the same host resume without certificate verification, because PSK resumption skips the certificate check. Keep-alive connections were also reused across CA bundles, and client certificates were not part of the pool key. Only the 2.0 native engine was affected; 1.1.0 is not. Same-policy pooling and resumption still work. - Python 3.14 import deadlock (#29). The native TLS and HTTP/2 engines now load when the package is imported. Before, the first HTTP/2 request imported them from a request thread, which could deadlock on Python 3.14’s per-module import locks when another thread imported
hpackat the same time. - HTTP/2 GOAWAY after a complete response (#29). When a response and an error GOAWAY arrive in the same read, the fully received response is returned and the GOAWAY only ends the connection. It used to be discarded with an error. An error GOAWAY that arrives before the response still raises.
- Faster HTTP/2 request header validation (#29). Method, path, header name and header value checks use precompiled regular expressions. The rejected inputs are unchanged.
stream=Trueover HTTP/1.1 works (#30). In the 2.0 merge it returned no body:contentwasNoneanditer_content()yielded nothing. Streamed bodies now decodegzip,deflateandbras they arrive, report truncated compressed bodies, keep memory bounded whatever the server’s chunk sizes, and return the connection to the pool when drained. An encoding that cannot be streamed raisesContentDecodingErrorinstead of returning still-encoded bytes. See Streaming responses.Response.infos["tls_backend"]is set (#30) on every connection:"native","openssl"or"plain". In the 2.0 merge reading it raisedKeyError.reap doctorchecks the named target (#33). It measures the target’s own traffic (it used Chrome for every target), showsja3asn/ainstead of failing for Chromium targets, compares the header order, accepts all 79 names including aliases and legacy names, honors--impersonate, names an unknown target plainly, and--listprints every name. See Command line.- Each target’s captured HTTP/2 header set is sent (#34). The transport used to send the HTTP/1.1 set lowercased over HTTP/2, so the Chrome (124 and newer) and Safari (18.0 and newer) targets sent no
priorityheader over HTTP/2. See HTTP/2. - Client certificates work under
tls="auto"(#34). A request withcert=goes through OpenSSL and presents the certificate. Before, the native engine sent an empty certificate.tls="native"withcert=raisesValueError. The(cert, key, password)form works, and paths may beos.PathLike. See Client certificates. - The TLS 1.2 fallback stays on the proxy (release branch). When the native handshake found a TLS 1.2-only server, the OpenSSL fallback re-dialed the server directly instead of through the configured proxy. Through a Socket Secure (SOCKS) proxy the request left from the machine’s own address; through an HTTP proxy a direct connection was opened and dropped before the tunnelled one. The fallback now reconnects the same way as the first attempt. With an
https://proxy, which the OpenSSL path cannot run through, it fails before dialing anything. See TLS 1.2 servers and proxies. - Custom profiles offer only HTTP/1.1 (release branch). A hand-built
Profilehas no HTTP/2 spec but offeredh2in Application-Layer Protocol Negotiation (ALPN), so a server that picked HTTP/2 received an HTTP/1.1 request and the request failed. Such profiles now offerhttp/1.1only. Built-in targets are unchanged. See Custom profiles. - The CLI help for
--http2says it is a no-op, and the Geocoder’s default User-Agent sayscurl_reap/2.0.
Changed
- HTTP/2 is negotiated by default.
Session(http2=False)opts out. The CLI--http2flag is accepted but no longer changes anything. Response.okisTruefor 2xx and 3xx. Useis_successfor 2xx only. Theget,metaandlinkscommands useokfor their exit status.RetryPolicyretries onlyGET,HEAD,OPTIONS,PUTandDELETEby default.POSTandPATCHneedallowed_methods.max_redirectsdefaults to 30 (it was 10).- The crawl engine sends responses with status 400 or higher, and challenge-looking responses, to the request’s errback instead of
parse. cryptography>=42,hpack>=4andbrotli>=1.1are required.zstandardis an optionalzstdextra (pip install "curl-reap[zstd]") for zstd bodies and certificate compression.- Exceptions mirror the wider ecosystem’s names and error codes, while
TransportErrorandOSErrorstay as base classes.HTTPStatusErroris now an alias ofHTTPError, anOSErrorsubclass (it was a plainException).
Known limitations
- No native TLS 1.2. TLS 1.2-only servers get an OpenSSL-shaped ClientHello. That fallback offers only the X25519 group (
Profile.curve), so a TLS 1.2-only server with a P-256 Elliptic Curve Digital Signature Algorithm (ECDSA) certificate fails the handshake; servers with RSA certificates work. - An
https://proxy works only on the native TLS path. The OpenSSL path (TLS 1.2-only servers,cert=, custom profiles without a template,tls="openssl") fails through one; use anhttp://or SOCKS proxy there. - A per-request header set to
Noneremoves a session header but not one that comes from the profile. Use a customProfileto send fewer headers. - HTTP/3 (QUIC) is not implemented.
Troubleshooting lists these with the other known issues.
1.1.0 (2026-09-30, GitHub release)
Fixes the reap command line tool and ships the features added since 1.0.0. The release has 67 passing tests.
- Command line fix. The
reapcommand works again. In 1.0.0 it failed at startup with aSyntaxErrorincurl_reap/cli.py. Command-line interface (CLI) tests were added, and continuous integration (CI) now runsreap --help. - HTTP/2. Optional HTTP/2 transport through hyper-h2:
Session(http2=True),reap --http2, and a newh2extra. Falls back to HTTP/1.1 when the server or the environment does not support it. See HTTP/2. - Versioned impersonation targets.
impersonate=is back as an alias ofprofile=, with 38 versioned Chrome, Edge, Firefox and Safari targets plus four aliases inreap.FINGERPRINTS, and a CLI--impersonateoption. Conflictingprofileandimpersonatevalues on a request raiseValueError. See Impersonation targets. - Cookie jar.
Session.cookiesis aCookieJarthat stores cookies from every response and redirect hop and replays them. Exported asreap.CookieJar. - Streaming downloads.
Session.download(),reap.download()andAsyncSession.download()stream to disk in constant memory with an atomic part-file rename. See Caching and downloads. - Challenge detection.
Response.looks_like_challenge. - JavaScript rendering.
reap.render()andreap.render_if_empty()through Playwright, in a newjsextra. See JavaScript rendering. - Exports.
reap.TransportErroris exported at the package root. - Tests and CI. An adversarial transport test suite, and a CI matrix for Python 3.9 to 3.13.
1.0.0 (2026-08-22, GitHub release)
The from-scratch, breaking release.
- New transport. curl_cffi is removed. curl_reap now has its own HTTP/1.1 engine on the standard library’s
socketandssl, with byte-level control of the request line and header order and casing, keep-alive pooling, chunked decoding, and gzip and deflate decoding. - TLS and header profiles.
curl_reap.tls.Profilewith curated cipher orderings, ALPN and browser-family headers forchrome,firefoxandsafari. Custom profiles are supported. - Breaking:
impersonate=was removed in favor ofprofile=(reap.get(url, profile="chrome")). It was added back as an alias on main after this release. - Retries with exponential backoff and jitter, proxy rotation, and profile rotation across retries.
- Dependencies are now only
lxmlandcssselect. Python 3.9+. - Selectors, structured extraction, the crawl engine, the disk cache, the async facade, pipelines and the CLI carry over.
0.3.0 (never released)
The version was bumped to 0.3.0 in the repository on 2026-07-19 but was not tagged or uploaded. Its changes shipped in 1.0.0:
- Encoding detection from the byte order mark,
Content-Typeand<meta charset>, exposed asResponse.encoding. - Conditional cache revalidation with
If-None-MatchandIf-Modified-Since. - robots.txt
Crawl-delayas a per-domain floor, andReaper.discovered_sitemaps. - Gzipped sitemaps in
SitemapSpider. - Automatic proxy rotation on 403, 407 and 429 with a proxy pool.
0.2.2 (2026-07-15, PyPI)
- A one-time terminal tip suggesting proxy rotation, and NodeMaven, when a request without a proxy is blocked with 403, 407 or 429. It could be silenced with
REAP_NO_TIPS=1. This tip is not part of the 1.0.0 transport. - A tracked sponsor link in the PyPI description.
0.2.1 (2026-07-15, PyPI)
- The README sponsor section with NodeMaven, published to PyPI.
0.2.0 (2026-07-08, GitHub release and PyPI)
A major upgrade into a full-stack scraper; the 0.1 API kept working.
- Transport:
RetryPolicywith backoff, jitter andRetry-After; fingerprint and proxy rotation;DiskCache;AsyncSession,agetandapost; richerResponse(status_code,cookies,urljoin(),follow(),raise_for_status()). - Parsing:
jsonld(),meta_tags(),links(),images(),tables(),markdown(),re_first(), and afind_by_textthat returns the deepest match. - Crawl engine: a continuous priority-queue scheduler, per-domain AutoThrottle that backs off on 429 and 503, opt-in robots.txt,
allowed_domainsandmax_depth, errbacks, and stats and logging. - Spiders and pipelines:
Request(priority=, errback=, dont_filter=),SitemapSpider,SqlitePipelinewith automatic columns and upsert, and a thread-safeJsonLinesPipeline. - CLI: the
reapcommand withget,meta,linksandcrawl.
0.1.2 (2026-06-30, PyPI)
Uploaded to PyPI. There is no matching tag or commit message in the repository.
0.1.1 (2026-06-30, PyPI)
- An absolute logo URL so the README renders on PyPI.
0.1.0 (2026-06-30, PyPI)
The first release. Browser TLS and JA3 impersonation through curl_cffi, lxml selectors with self-healing auto_match selectors, and a concurrent crawl engine with AutoThrottle, retries, dedup and pipelines.
Between 0.1.1 and 0.2.0, the repository also gained the Geocoder, the Booking.com example, LEGAL.md and the first documentation site.