curl_reap / Get started
Installation
Install curl_reap 2.0.0 from PyPI, what the required dependencies are, the zstd, js and server extras, and how to check which version you have.
On this page
This page covers what curl_reap needs, how to install the version these docs describe, the optional extras, and a development setup.
First principles
- What you install is Python plus five prebuilt dependencies. All five ship wheels for common platforms, nothing is compiled on your machine, and nothing is downloaded while the library runs.
- The version decides the transport. A plain
pip installreads the Python Package Index (PyPI), which serves 2.0.0. An older pinned version is a different library: 0.2.2, the previous PyPI release, is built on curl_cffi. - Optional features stay optional. Rendering,
zstddecoding and the job service import their dependencies only when you use them, so the core install stays small.
Requirements
- Python 3.9 or newer (
requires-python = ">=3.9"). - Five runtime dependencies, installed automatically:
| Package | Version | Used for |
|---|---|---|
lxml |
>=5.0 |
HTML parsing and selectors |
cssselect |
>=1.2 |
CSS to XPath translation |
cryptography |
>=42 |
the native TLS 1.3 engine: key exchange, AEAD ciphers, certificate verification |
hpack |
>=4.0 |
HTTP/2 header compression |
brotli |
>=1.1 |
Brotli response bodies and certificate compression |
All five ship wheels for common platforms. There is no compiled curl binding and nothing is downloaded at runtime.
Install
pip install curl_reap
This installs 2.0.0, released on 2026-10-26. To upgrade an existing install, run pip install --upgrade curl_reap. The distribution name is curl_reap; pip treats curl_reap and curl-reap as the same package, and PyPI lists it as curl-reap. The import name is curl_reap:
import curl_reap as reap
print(reap.__version__) # 2.0.0
Installing the package also installs the reap command line tool.
Versions
| Version | Where | Notes |
|---|---|---|
| 2.0.0 | PyPI and GitHub release v2.0.0 |
What these docs describe: the native TLS 1.3 and HTTP/2 transport. |
| 1.1.0 | GitHub release v1.1.0 only |
Standard-library socket and ssl transport, HTTP/2 through the h2 extra. |
| 1.0.0 | GitHub release v1.0.0 only |
The rewrite that removed curl_cffi. |
| 0.2.2 | PyPI | The last release built on curl_cffi. It does not have the API these docs describe. |
PyPI went from 0.2.2 to 2.0.0. Check reap.__version__ after installing. If it does not say 2.0.0, you have an older release; read the Changed list of the 2.0.0 entry in the Changelog before upgrading.
Optional extras
Extras are imported lazily, so curl_reap works without them. Put the extra in brackets, and quote the argument so the shell leaves the brackets alone:
pip install "curl-reap[zstd]"
| Extra | Installs | Adds |
|---|---|---|
zstd |
zstandard>=0.23 |
Decoding of Content-Encoding: zstd bodies, and zstd certificate compression in the TLS engine. On Python 3.14 and newer the standard library’s compression.zstd module covers both without the extra. |
js |
playwright>=1.51,<1.52 |
reap.render(), reap.render_if_empty() and reap.BrowserSession. See JavaScript rendering. |
server |
fastapi>=0.115,<0.116, uvicorn>=0.30,<0.31, redis>=5.2,<5.3 |
The private job API and worker behind reap serve and reap worker. |
h2 |
nothing | Kept as an empty alias for one release. HTTP/2 is built in since 2.0.0. |
dev |
pytest>=7.0, build>=1.0, twine>=5.0, httpx>=0.27,<0.28, zstandard>=0.23 |
Running the test suite and building the package. |
After installing the js extra, download the browser Playwright drives:
playwright install chromium
Without it, reap.render() cannot launch a browser.
Brotli no longer needs anything extra: brotli is a required dependency in 2.0.0, so br response bodies are always decoded.
Development install
The repository’s CI installs the package in editable mode with the dev and server extras and runs the test suite:
git clone https://github.com/anishfyi/curl_reap
cd curl_reap
python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev,server]"
python -m pytest tests/ -q
A few HTTP/2 tests use the hyper-h2 package (h2) as a test server and are skipped when it is not installed. The h2 extra no longer installs it, so run pip install h2 if you want those tests.
Files curl_reap writes
Some features write to the current working directory by default. Add them to .gitignore in your own projects:
.reap_selectors.json: saved selector signatures for self-healing selectors..reap_cache/: the default DiskCache directory..reap_geocode.json: the default Geocoder cache.