Skip to content
curl_reap

curl_reap / Reference

Python API reference

Every public name in curl_reap 2.0 with signatures and defaults: Session, Response, Headers, cookies, WebSocket and the exception hierarchy.

On this page
  1. First principles
  2. Breaking change in 2.0: Response.ok
  3. __all__
  4. Transport
  5. Module functions (exported)
  6. Session (exported)
  7. AsyncSession (exported)
  8. Response (exported)
  9. Headers (in curl_reap.headers)
  10. RetryPolicy (exported)
  11. CookieJar (exported) and curl_reap.cookies
  12. DiskCache (exported)
  13. WebSocket (exported)
  14. Profiles (exported)
  15. Rendering (optional js extra, exported)
  16. Exceptions
  17. Parsing
  18. Selector (exported)
  19. SelectorList (exported)
  20. Self-healing functions (exported)
  21. Crawling
  22. Spider and SitemapSpider (exported)
  23. Request (exported)
  24. run and Reaper (exported)
  25. AutoThrottle (exported)
  26. Pipelines (exported)
  27. Extras
  28. Fingerprint check (in curl_reap.fingerprint)

This is the Python application programming interface (API) of curl_reap. Everything in the sections below marked “exported” is importable from the package root (import curl_reap as reap) and listed in curl_reap.__all__. A few documented names live in submodules and say so. Signatures and defaults are copied from the 2.0.0 source. Guides explain usage in more depth.

First principles

The public names fall into three layers:

  • Transport (get, Session, Response, WebSocket): puts requests on the wire and returns responses.
  • Parsing (Selector, the extraction methods, the self-healing functions): works on any HyperText Markup Language (HTML) document, fetched or not. A Response passes these calls to its Selector.
  • Crawling (Spider, Request, Reaper, run, pipelines): schedules many requests through one Session and feeds the responses to your callbacks.

The layers are loosely coupled, so each piece also works on its own: a Selector on saved HTML, a Session without a crawl, or a Reaper with a session you configured yourself. Names that live only in submodules (curl_reap.headers, curl_reap.cookies, curl_reap.exceptions, curl_reap.fingerprint) are listed below with the module to import them from.

Breaking change in 2.0: Response.ok

Response.ok is now True for status 200 to 399, not only 2xx. Code that used if r.ok: to mean “the request succeeded” should switch to r.is_success, which is True for 2xx only. raise_for_status() raises for every status outside 2xx, including 3xx.

r = s.get(url, allow_redirects=False)
r.ok           # True for a 302
r.is_success   # False for a 302

Other changes that can affect existing code: HTTPStatusError is now an alias of HTTPError, which subclasses RequestException and OSError (it was a plain Exception); elapsed is a timedelta (use elapsed_seconds for the float); max_redirects defaults to 30; HTTP/2 is negotiated by default; and RetryPolicy no longer retries POST or PATCH unless allowed_methods includes them.

__all__

get, post, fetch, download, Session, Response, RetryPolicy,
HTTPStatusError, TransportError, URLPolicyError, CookieJar,
PROFILES, FINGERPRINTS, Profile, DiskCache, detect_encoding,
AsyncSession, aget, apost, WebSocket, AsyncWebSocket,
render, render_if_empty, BrowserSession,
Selector, SelectorList, signature, similarity, save, relocate,
Spider, SitemapSpider, Request, Reaper, run,
Pipeline, DedupPipeline, JsonLinesPipeline, CsvPipeline, SqlitePipeline,
pipelines, AutoThrottle, Geocoder, geocode, __version__

Transport

Module functions (exported)

  • reap.get(url, **kw) -> Response: GET through a shared module-level Session.
  • reap.post(url, **kw) -> Response: POST through the shared session.
  • reap.fetch(url, **kw) -> Response: an alias of get.
  • reap.download(url, path, **kw) -> Response: stream to a file through the shared session.
  • async reap.aget(url, **kw) and async reap.apost(url, **kw): one request on a temporary AsyncSession.
  • reap.detect_encoding(content, headers=None): charset from a byte order mark, Content-Type, or HTML meta tags, or None.

The keywords **kw are the per-request options listed in Sessions.

Session (exported)

Session(profile="chrome", headers=None, timeout=30, retries=2, proxy=None,
        rotate=None, profiles=None, retry_policy=None, cache=None,
        on_response=None, block_rotations=None, verify=True,
        allow_redirects=True, max_redirects=30, impersonate=None,
        http2=True, sticky=False, url_policy=None, tls="auto",
        max_body_bytes=67108864, interface=None, debug=False,
        trust_env=True, cert=None, proxies=None, http_version=None)
  • Methods: request(method, url, **kw), get, post, head, put, patch, delete, options, download(url, path, **kw), ws_connect(url, **kw), close(). Context manager.
  • Attributes: profile (the resolved Profile), profiles (the rotation pool), cookies (a CookieJar), retry_policy, timeout, cache, verify, cert, http2, http_version, sticky, url_policy, max_body_bytes, debug, trust_env, block_rotations. Properties: retries, tls_backend (the configured tls= mode).
  • rotate must be None, "sequence" or "random"; tls must be "auto", "native" or "openssl"; http_version must be one of "v1", "v2", "v2tls", "v2_prior_knowledge", "v3", "v3only", and the last two raise NotImplementedError.

Options are described in Sessions, retries, proxies and cookies.

AsyncSession (exported)

AsyncSession(profile="chrome", headers=None, timeout=30, retries=2, proxy=None,
             rotate=None, profiles=None, retry_policy=None, cache=None,
             on_response=None, block_rotations=None, verify=True,
             allow_redirects=True, max_redirects=30, impersonate=None,
             http2=True, sticky=False, url_policy=None, max_clients=10,
             tls="auto", **kwargs)

Extra **kwargs go to Session (for example trust_env, proxies, cert, http_version). max_clients bounds concurrent requests with a semaphore. Awaitable methods: request, get, post, head, put, patch, delete, options, download, ws_connect, close. Properties: profile, retries, cookies. Async context manager. Runs the synchronous session on worker threads.

Response (exported)

Response(raw=None, meta=None, elapsed=0.0, *, status=None, url=None, headers=None, content=None, text=None, from_cache=False). Sessions create responses for you; the keyword form is handy for tests and for parsing saved HTML.

Status:

  • status, status_code (alias), reason.
  • ok: True for 200 to 399. is_success: True for 2xx only.
  • is_redirect: 3xx with a Location header. redirect_url: Location resolved against url, or None. redirect_count: len(history).
  • raise_for_status(): raises HTTPError unless is_success; returns the response otherwise.

Body:

  • content: the decoded body bytes.
  • text: decoded lazily on first access with encoding, using replacement characters for invalid bytes. Settable.
  • encoding: detected from a byte order mark, Content-Type or HTML meta tags, else default_encoding ("utf-8"). Setting it clears the cached text, so the next access re-decodes.
  • charset (alias charset_encoding): the charset the body declares, or None.
  • json(**kw): parses the body with json.loads (a bad body raises the standard library’s json.JSONDecodeError).
  • iter_content(chunk_size=65536), iter_lines(chunk_size=65536, delimiter=None), close(). See Caching and downloads.

Metadata:

  • url, headers (a Headers), cookies (dict of cookies set by this response), history (redirect responses), meta, from_cache, raw.
  • elapsed: a datetime.timedelta. elapsed_seconds: the same as a float.
  • http_version ("HTTP/1.1" or "HTTP/2") and http_version_int (libcurl’s numbering: 2 for HTTP/1.1, 3 for HTTP/2).
  • infos: connection details. On every response that came over the network it holds tls_backend: "native", "openssl" or "plain". A cache hit or a rendered response has an empty infos. On TLS connections it also holds alpn, cipher, group and tls_version, plus resumed and handshake_seconds on the native engine; peername and sockname when known; http_version on HTTP/2.
  • primary_ip, primary_port, local_ip, local_port, request_size, response_size.
  • request: a dict snapshot of what was sent (method, url, headers).
  • looks_like_challenge: True when the body looks like a bot wall. block_reason: "forbidden", "proxy_auth_required", "rate_limited", "challenge" or None.

Parsing:

  • urljoin(url), follow(target, callback=None, **kw), selector().
  • Selector passthroughs: css, css_first, xpath, re, re_first, find_by_text, find_similar, save, extract(fields, **kw).
  • Extraction: jsonld(), meta_tags(), links(**kw), images(**kw), tables(), markdown(max_len=None, main_only=False).

Pickling: a Response can be pickled. The pickled state drops raw, any open stream and the cached selector; status, URL, headers, content, text, encoding, cookies, history, timing and meta survive.

Headers (in curl_reap.headers)

Not exported at the package root: from curl_reap.headers import Headers. Response.headers is an instance. CaseInsensitiveHeaders in curl_reap.http is the same class, kept as the 1.x name.

Headers(headers=None, encoding=None) accepts a mapping, (name, value) pairs, a raw header block, or another Headers. It is a dict subclass, ordered, case-insensitive and multi-value:

  • h[name] joins duplicate values with ", "; get(name, default=None).
  • get_list(name, split_commas=False): every value in order.
  • multi_items(): every (name, value) pair in wire order. raw(): the same as latin-1 bytes.
  • h[name] = value replaces every occurrence; add(name, value) appends.
  • del, pop, popitem, setdefault, update, clear, copy, merge(*sources), keys, values, items.

RetryPolicy (exported)

RetryPolicy(retries=2, backoff=0.5, max_backoff=30.0, statuses=RETRY_STATUSES, respect_retry_after=True, allowed_methods=None)

RETRY_STATUSES (in curl_reap.http) is 408, 425, 429, 500, 502, 503, 504, 520, 521, 522, 523 and 524. allowed_methods=None means GET, HEAD, OPTIONS, PUT and DELETE. Methods: should_retry_status(status), allows_method(method), delay(attempt, retry_after=None). Retry-After is clamped to 24 hours and is not capped by max_backoff. Negative values raise ValueError.

CookieJar (exported) and curl_reap.cookies

curl_reap.cookies exports CookieJar, Cookies, CookieConflict and parse_set_cookie.

  • CookieJar(): thread-safe, scoped by domain, path and name. Methods: absorb(url, specs), store(url, pairs), set(name, value, domain=None, path="/", secure=False), get(name, default=None, domain=None, path=None), delete(name, domain=None, path=None), update(cookies), header_for(url, extra=None), get_dict(), items(), save(path), load(path), to_cookiejar(), clear(), len() and iteration over cookie dicts.
  • parse_set_cookie(value): one Set-Cookie value to a dict (name, value and the known attributes), or None.
  • Cookies(cookies=None): a mutable name -> value mapping over a jar, with set, get, delete and update. It raises CookieConflict when several cookies share a name.

See Cookies for the scoping rules.

DiskCache (exported)

DiskCache(directory=".reap_cache", ttl=3600.0, methods=("GET",)). Methods: accepts(method), get(method, url, params=None), get_stale(method, url, params=None), touch(method, url, params=None), set(method, url, params, resp), clear(). See Caching and downloads.

WebSocket (exported)

  • WebSocket.connect(url, *, session=None, subprotocols=(), headers=None, timeout=None, profile=None, proxy=None, verify=True)
  • Session.ws_connect(url, **kw) returns a WebSocket bound to the session.
  • await AsyncSession.ws_connect(url, **kw) returns an AsyncWebSocket.

See WebSocket.

Profiles (exported)

  • Profile(name, family="custom", version="", headers=(), h2_headers=None, tls_template=None, h2_spec=None, accept_encoding="", form_boundary="", ciphers=(), alpn_protocols=("h2", "http/1.1"), curve="X25519", minimum_version="TLSv1_2"): a frozen dataclass.
  • PROFILES: dict of chrome, firefox, safari.
  • FINGERPRINTS: dict of all 79 target names (38 captured targets, 19 aliases, 22 legacy names). See Impersonation targets.

The module curl_reap.tls also defines resolve_profile(profile) and create_ssl_context(profile="chrome", verify=True, alpn=("http/1.1",)). They are not exported at the package root.

Rendering (optional js extra, exported)

  • render(url, *, timeout=30, wait_until="networkidle", proxy=None, user_agent=None, browser_channel="chromium", headless=True, script=None) -> Response
  • render_if_empty(response, *, min_bytes=2000, **kw) -> (Response, Response | None)
  • BrowserSession(*, timeout=30, wait_until="domcontentloaded", proxy=None, user_agent=None, browser_channel="chromium", headless=True, cdp_url=None, url_policy=None) with goto(url, *, wait_for=None, script=None, timeout=None, wait_until=None), follow(target, **kw), close(), and the page and url properties.

See JavaScript rendering.

Exceptions

The package root exports TransportError, HTTPStatusError and URLPolicyError. The full set lives in curl_reap.exceptions. Names follow the wider Python HTTP ecosystem, and each class carries a libcurl-style error number in .code (shown below), plus .response when a response is involved. Every class except URLPolicyError is an OSError.

OSError
└── RequestException (alias RequestsError)            code 0
    ├── TransportError                                 code 0
    │   ├── ConnectionError                            code 7
    │   │   ├── DNSError                               code 6
    │   │   ├── ProxyError                             code 5
    │   │   ├── SSLError                               code 35
    │   │   │   └── CertificateVerifyError             code 60
    │   │   └── InterfaceError                         code 45
    │   ├── Timeout                                    code 28
    │   │   ├── ConnectTimeout (also ConnectionError)  code 28
    │   │   └── ReadTimeout                            code 28
    │   ├── TooManyRedirects                           code 47
    │   ├── IncompleteRead                             code 18
    │   ├── ChunkedEncodingError                       code 56
    │   ├── ContentDecodingError                       code 61
    │   └── curl_reap._h2.H2Error                      code 16
    ├── InvalidURL (also ValueError)                   code 3
    ├── InvalidSchema (also ValueError)                code 1
    ├── InvalidHeader (also ValueError)                code 43
    ├── ImpersonateError                               code 0
    ├── SessionClosed                                  code 0
    ├── StreamConsumedError                            code 0
    ├── UnrewindableBodyError                          code 0
    ├── HTTPError (alias HTTPStatusError)              code 22
    ├── CookieConflict                                 code 0
    ├── InvalidJSONError                               code 0
    │   └── JSONDecodeError (also ValueError)          code 0
    └── WebSocketError                                 code 0
        ├── WebSocketClosed                            code 0
        └── WebSocketTimeout (also Timeout)            code 28

ValueError
└── URLPolicyError
  • HTTPError is raised by raise_for_status(); the response is on .response. It is not a TransportError, so except TransportError does not catch it.
  • TooManyRedirects is also raised for a redirect refused by allow_redirects="safe"; the redirect response is on .response.
  • H2Error adds error_code, error_name, stream_id and retryable. The session reports HTTP/2 failures as TransportError with the H2Error as __cause__. See HTTP/2.
  • WebSocketClosed has close_code and reason.
  • curl_reap.exceptions also exports CurlECode (an IntEnum of libcurl error numbers) and CODE2ERROR, which maps a code to the class above.

Parsing

Selector (exported)

Selector(content=None, element=None, url=None, status=None, headers=None)

  • css(query, auto_match=False, identifier=None, storage=None) -> SelectorList
  • css_first(query, default=None, **kw)
  • xpath(query) -> SelectorList
  • find_by_text(text, partial=True, first=False, deepest=True) -> SelectorList
  • find_similar(sample, threshold=0.6, limit=None) -> SelectorList
  • save(identifier, storage=None): returns the selector.
  • re(pattern, flags=0) -> SelectorList, re_first(pattern, default=None, flags=0)
  • extract(fields, *, many=False): fields maps names to CSS queries; returns each first match (or all matches with many=True).
  • Properties: text, attrib, html, base_url. Method: attr(name, default=None).
  • Extraction: jsonld(), meta_tags(), links(absolute=True, unique=True, internal_only=False), images(absolute=True), tables(), markdown(max_len=None, main_only=False).

SelectorList (exported)

A list subclass with get(default=None), getall(), text(), attr(name, default=None) and css(query, **kw).

Self-healing functions (exported)

  • signature(el) -> dict
  • similarity(a, b) -> float (0 to 1)
  • save(identifier, el, storage=None): default store .reap_selectors.json.
  • relocate(identifier, tree, storage=None, threshold=0.6): an lxml element or None.

See Selectors and self-healing.

Crawling

Spider and SitemapSpider (exported)

  • Spider: class attributes name="reap", start_urls=[], allowed_domains=None, max_depth=None, custom_headers=None; methods start() and parse(page) (must be implemented).
  • SitemapSpider(Spider): class attributes sitemap_urls=[], url_pattern=None; method parse_sitemap(page).

Request (exported)

Request(url, callback=None, method="GET", meta=None, priority=0, errback=None, dont_filter=False, **kw). Extra keywords go to Session.request. Method: fingerprint().

run and Reaper (exported)

  • run(spider, **kw) -> list: accepts a class or instance.
  • Reaper(spider, concurrency=8, retries=2, throttle=True, delay=0.0, profile="chrome", pipelines=None, dedup=True, on_item=None, max_pages=None, max_depth=None, respect_robots=False, session=None, cache=None, rotate=None, proxy=None, collect_items=True, per_domain_concurrency=None, max_pending=None)
  • Reaper attributes: run(), stats, items, session, pipelines, discovered_sitemaps.

See Crawling.

AutoThrottle (exported)

AutoThrottle(base_delay=0.0, target_concurrency=8, max_delay=30.0, enabled=True) with observe(latency, status=None), wait() and the current delay.

Pipelines (exported)

  • Pipeline: base class with open(), process(item), close().
  • DedupPipeline(key=None)
  • JsonLinesPipeline(path)
  • CsvPipeline(path)
  • SqlitePipeline(path, table="items", key=None)
  • pipelines: the curl_reap.pipelines module.

See Pipelines.

Extras

  • Geocoder(endpoint="https://nominatim.openstreetmap.org/search", user_agent=DEFAULT_UA, cache=".reap_geocode.json", min_interval=1.1, profile="chrome", min_importance=0.0) with geocode(name=None, area=None, city=None, country=None) and clean_name(name).
  • geocode(name=None, area=None, city=None, country=None): uses a shared default Geocoder.
  • __version__: the package version string ("2.0.0").

Fingerprint check (in curl_reap.fingerprint)

Not exported at the package root: from curl_reap.fingerprint import check, EXPECTED.

  • check(target="chrome", url="https://tls.peet.ws/api/all", session=None) -> dict: fetches the echo service as target and compares the result with the capture. Without session it creates a Session(impersonate=target) and closes it afterwards. Any object with get(url) returning something with json() works as session. Raises KeyError for an unknown target.
  • The report has target, capture (the captured target whose values apply; it differs from target for aliases and legacy names), the observed ja3, ja4, peetprint and akamai, expected, matches, headers (the header names in the order the service saw them), expected_headers and http_version.
  • matches holds ja4, ja4_r, akamai, ja3 and headers, each True, False, or None when it cannot be compared.
  • EXPECTED: the captured values per target (ja3, ja3_fixed, ja4, ja4_r, akamai).

reap doctor is the command-line front end. See Command line.

See Geocoding.