Docs / Documentation
curl_reap
Python scraping: browser-exact TLS 1.3 and HTTP/2, self-healing selectors and a crawler
Sponsors
The most efficient proxy provider for web scraping and automation, with the highest quality IPs on the market.
Codes for curl_reap users
CURLREAP3535% off mobile and residential proxiesCURLREAP4040% off ISP (static) proxies
On this page
curl_reap is one Python library for scraping. It combines four jobs that usually need separate tools:
- Transport. Its own Transport Layer Security (TLS) 1.3 client, written in Python on the
cryptographypackage, renders the ClientHello captured from each target browser byte for byte: cipher order, extension order and Generate Random Extensions And Sustain Extensibility (GREASE) values, X25519MLKEM768 post-quantum key shares, certificate compression, Application-Layer Protocol Settings (ALPS), Encrypted Client Hello (ECH) GREASE and padding. Hypertext Transfer Protocol version 2 (HTTP/2) is built in and on by default, with each browser’s SETTINGS, WINDOW_UPDATE, priority and header compression (HPACK) behavior. There are 79 impersonate target names (38 captured browsers, 19 aliases and 22 legacy names). Retries, proxy and profile rotation, cookies, a disk cache, streaming downloads, WebSocket and an async facade are built in. - Parsing. An lxml selector with parsel-style Cascading Style Sheets (CSS) and XML Path Language (XPath) queries,
::textand::attr(name)pseudo elements, finders, and self-healing selectors that re-find an element after a site changes its markup. - Structured extraction. One call each for JavaScript Object Notation for Linked Data (JSON-LD), meta tags, links, images, tables, named fields and readable markdown.
- Crawling. A concurrent crawl engine with a priority scheduler, per-domain AutoThrottle, opt-in robots.txt, depth and domain limits, dedup, sitemap spiders, and pipelines to JSON Lines, comma-separated values (CSV) and SQLite.
It is for Python developers who write scrapers and crawlers and want one library instead of stitching together an HTTP client, a parser and a crawl framework. The required dependencies are lxml, cssselect, cryptography, hpack and brotli, all installed from wheels. There is no compiled curl binding and nothing is downloaded at runtime. Python 3.9 or newer.
First principles
A scrape is three problems, and the four parts above answer them:
- Getting the page. A server sees the TLS handshake, the HTTP/2 framing and the headers before it decides what to send. A request that does not look like a browser’s can get a different answer. curl_reap replays all three layers from captured browsers, so they agree with each other.
- Finding the data. Markup changes without notice, and a selector that finds nothing raises no error. curl_reap reads the structured data sites already publish, and its selectors can re-find an element from a saved signature.
- Doing it many times. Every request is load on someone else’s server. The crawl engine bounds the queue, the page budget and the requests per host, and backs off when a server asks it to.
How it works explains the design, its trade-offs and how it fails.
Minimal example
import curl_reap as reap
page = reap.get("https://quotes.toscrape.com", impersonate="chrome")
print(page.css("span.text::text").getall())
print(page.jsonld())
print(page.http_version) # "HTTP/2" when the server negotiates it
print(page.infos["tls_backend"]) # "native" when the captured handshake was used
reap.get returns a Response. The CSS, XPath and extraction methods work on it directly. impersonate= and profile= accept the same names. See the Quickstart for a crawl in a dozen lines.
What it does
- Native TLS 1.3 with PSK session resumption and RFC 5280 certificate verification, an OpenSSL fallback for servers that only speak TLS 1.2, and connection pooling. See Transport and impersonation profiles.
- 79 impersonate names across Chrome, Edge, Firefox, Tor Browser and Safari, on desktop, Android and iOS. See Impersonation targets.
- HTTP/2 on by default, with content-decoded bodies and streamed downloads. See HTTP/2.
- WebSocket clients (
reap.WebSocket,reap.AsyncWebSocket,Session.ws_connect) on the same transport. See WebSocket. - Sessions with retries, backoff,
Retry-Aftersupport, proxy rotation, cookies, redirects and an async wrapper. See Sessions, retries, proxies and cookies. - A disk response cache with ETag and Last-Modified revalidation, and constant-memory downloads. See Caching and downloads.
- CSS, XPath, regex, finders and self-healing selectors. See Selectors and self-healing.
jsonld(),meta_tags(),links(),images(),tables()andmarkdown(). See Structured extraction.- Spiders,
reap.run()and theReaperengine. See Crawling and Pipelines. extract()for named fields from CSS queries, andmarkdown(main_only=True)for the main content only. See Structured extraction.- Optional JavaScript rendering through Playwright, one page at a time or in a reusable
BrowserSession. See JavaScript rendering. - A
reapcommand for fetching, extracting and crawling from the shell, andreap doctor, which fetches a fingerprint echo service and compares what your machine sends with the captured browser values. See Command line. - A cascading geocoder over OpenStreetMap Nominatim for scraped listings. See Geocoding.
- An optional private job service (
reap serveandreap worker) that runs single-page extraction jobs from a Redis queue. See Private job service.
Common errors and limits are collected in Troubleshooting and limits.
What it does not do
- Servers that only speak TLS 1.2 get the OpenSSL fallback, which sends an OpenSSL-shaped ClientHello. Only the HTTP layer looks like the browser on those connections. There is no native TLS 1.2.
- HTTP/3 (QUIC) is not implemented.
http_version="v3"raisesNotImplementedError. - Custom
Profileobjects without a captured template use the OpenSSL path, so their TLS fingerprint is OpenSSL’s. - It does not solve CAPTCHA challenges (Completely Automated Public Turing tests to tell Computers and Humans Apart), bypass logins or paywalls, or defeat anti-bot services. Read Responsible use and legal notes before you scrape.
How it relates to other tools
curl_reap borrows one idea from each of these projects and puts them behind one API:
- curl_cffi showed how far TLS impersonation can take a scraper. curl_reap versions before 1.0.0 used it. Since 1.0.0 the transport is written in Python with no compiled curl binding. The 2.0 browser templates were captured from curl_cffi 0.16.3, and target names follow its naming (
chrome124,safari17_0), so migrating code keeps working. - Scrapy is the blueprint for the crawl engine: spiders, priorities, AutoThrottle and item pipelines.
- Scrapling and parsel inspired the self-healing selectors and the css/xpath ergonomics. curl_reap does not bundle a browser; rendering is an optional extra.
- curl-impersonate by lwthiker is the original browser-TLS-for-curl project.
Project status
These docs describe curl_reap 2.0.0, released on 2026-10-26 on the Python Package Index (PyPI) and as GitHub release v2.0.0. pip install curl_reap installs it. It is the first PyPI release since 0.2.2, which was still built on curl_cffi; 1.0.0 and 1.1.0 were GitHub releases only. Installation covers the install and the extras, and the Changelog lists what changed.
Sponsor
curl_reap is sponsored by NodeMaven, a proxy provider for web scraping and automation. This is a disclosed commercial relationship. The project README lists NodeMaven’s discount codes for curl_reap users: CURLREAP35 for 35% off mobile and residential proxies, and CURLREAP40 for 40% off ISP (static) proxies.
License
MIT. Source code, issues and releases are at github.com/anishfyi/curl_reap.
