Skip to content
curl_reap

curl_reap / Get started

Installation

Install curl_reap 2.0.0 from PyPI, what the required dependencies are, the zstd, js and server extras, and how to check which version you have.

On this page
  1. First principles
  2. Requirements
  3. Install
  4. Versions
  5. Optional extras
  6. Development install
  7. Files curl_reap writes

This page covers what curl_reap needs, how to install the version these docs describe, the optional extras, and a development setup.

First principles

  • What you install is Python plus five prebuilt dependencies. All five ship wheels for common platforms, nothing is compiled on your machine, and nothing is downloaded while the library runs.
  • The version decides the transport. A plain pip install reads the Python Package Index (PyPI), which serves 2.0.0. An older pinned version is a different library: 0.2.2, the previous PyPI release, is built on curl_cffi.
  • Optional features stay optional. Rendering, zstd decoding and the job service import their dependencies only when you use them, so the core install stays small.

Requirements

  • Python 3.9 or newer (requires-python = ">=3.9").
  • Five runtime dependencies, installed automatically:
Package Version Used for
lxml >=5.0 HTML parsing and selectors
cssselect >=1.2 CSS to XPath translation
cryptography >=42 the native TLS 1.3 engine: key exchange, AEAD ciphers, certificate verification
hpack >=4.0 HTTP/2 header compression
brotli >=1.1 Brotli response bodies and certificate compression

All five ship wheels for common platforms. There is no compiled curl binding and nothing is downloaded at runtime.

Install

pip install curl_reap

This installs 2.0.0, released on 2026-10-26. To upgrade an existing install, run pip install --upgrade curl_reap. The distribution name is curl_reap; pip treats curl_reap and curl-reap as the same package, and PyPI lists it as curl-reap. The import name is curl_reap:

import curl_reap as reap
print(reap.__version__)    # 2.0.0

Installing the package also installs the reap command line tool.

Versions

Version Where Notes
2.0.0 PyPI and GitHub release v2.0.0 What these docs describe: the native TLS 1.3 and HTTP/2 transport.
1.1.0 GitHub release v1.1.0 only Standard-library socket and ssl transport, HTTP/2 through the h2 extra.
1.0.0 GitHub release v1.0.0 only The rewrite that removed curl_cffi.
0.2.2 PyPI The last release built on curl_cffi. It does not have the API these docs describe.

PyPI went from 0.2.2 to 2.0.0. Check reap.__version__ after installing. If it does not say 2.0.0, you have an older release; read the Changed list of the 2.0.0 entry in the Changelog before upgrading.

Optional extras

Extras are imported lazily, so curl_reap works without them. Put the extra in brackets, and quote the argument so the shell leaves the brackets alone:

pip install "curl-reap[zstd]"
Extra Installs Adds
zstd zstandard>=0.23 Decoding of Content-Encoding: zstd bodies, and zstd certificate compression in the TLS engine. On Python 3.14 and newer the standard library’s compression.zstd module covers both without the extra.
js playwright>=1.51,<1.52 reap.render(), reap.render_if_empty() and reap.BrowserSession. See JavaScript rendering.
server fastapi>=0.115,<0.116, uvicorn>=0.30,<0.31, redis>=5.2,<5.3 The private job API and worker behind reap serve and reap worker.
h2 nothing Kept as an empty alias for one release. HTTP/2 is built in since 2.0.0.
dev pytest>=7.0, build>=1.0, twine>=5.0, httpx>=0.27,<0.28, zstandard>=0.23 Running the test suite and building the package.

After installing the js extra, download the browser Playwright drives:

playwright install chromium

Without it, reap.render() cannot launch a browser.

Brotli no longer needs anything extra: brotli is a required dependency in 2.0.0, so br response bodies are always decoded.

Development install

The repository’s CI installs the package in editable mode with the dev and server extras and runs the test suite:

git clone https://github.com/anishfyi/curl_reap
cd curl_reap
python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev,server]"
python -m pytest tests/ -q

A few HTTP/2 tests use the hyper-h2 package (h2) as a test server and are skipped when it is not installed. The h2 extra no longer installs it, so run pip install h2 if you want those tests.

Files curl_reap writes

Some features write to the current working directory by default. Add them to .gitignore in your own projects: