curl_reap / Reference
Python API reference
Every public name in curl_reap 2.0 with signatures and defaults: Session, Response, Headers, cookies, WebSocket and the exception hierarchy.
On this page
- First principles
- Breaking change in 2.0: Response.ok
- __all__
- Transport
- Module functions (exported)
- Session (exported)
- AsyncSession (exported)
- Response (exported)
- Headers (in curl_reap.headers)
- RetryPolicy (exported)
- CookieJar (exported) and curl_reap.cookies
- DiskCache (exported)
- WebSocket (exported)
- Profiles (exported)
- Rendering (optional js extra, exported)
- Exceptions
- Parsing
- Selector (exported)
- SelectorList (exported)
- Self-healing functions (exported)
- Crawling
- Spider and SitemapSpider (exported)
- Request (exported)
- run and Reaper (exported)
- AutoThrottle (exported)
- Pipelines (exported)
- Extras
- Fingerprint check (in curl_reap.fingerprint)
This is the Python application programming interface (API) of curl_reap. Everything in the sections below marked “exported” is importable from the package root (import curl_reap as reap) and listed in curl_reap.__all__. A few documented names live in submodules and say so. Signatures and defaults are copied from the 2.0.0 source. Guides explain usage in more depth.
First principles
The public names fall into three layers:
- Transport (
get,Session,Response,WebSocket): puts requests on the wire and returns responses. - Parsing (
Selector, the extraction methods, the self-healing functions): works on any HyperText Markup Language (HTML) document, fetched or not. AResponsepasses these calls to itsSelector. - Crawling (
Spider,Request,Reaper,run, pipelines): schedules many requests through oneSessionand feeds the responses to your callbacks.
The layers are loosely coupled, so each piece also works on its own: a Selector on saved HTML, a Session without a crawl, or a Reaper with a session you configured yourself. Names that live only in submodules (curl_reap.headers, curl_reap.cookies, curl_reap.exceptions, curl_reap.fingerprint) are listed below with the module to import them from.
Breaking change in 2.0: Response.ok
Response.ok is now True for status 200 to 399, not only 2xx. Code that used if r.ok: to mean “the request succeeded” should switch to r.is_success, which is True for 2xx only. raise_for_status() raises for every status outside 2xx, including 3xx.
r = s.get(url, allow_redirects=False)
r.ok # True for a 302
r.is_success # False for a 302
Other changes that can affect existing code: HTTPStatusError is now an alias of HTTPError, which subclasses RequestException and OSError (it was a plain Exception); elapsed is a timedelta (use elapsed_seconds for the float); max_redirects defaults to 30; HTTP/2 is negotiated by default; and RetryPolicy no longer retries POST or PATCH unless allowed_methods includes them.
__all__
get, post, fetch, download, Session, Response, RetryPolicy,
HTTPStatusError, TransportError, URLPolicyError, CookieJar,
PROFILES, FINGERPRINTS, Profile, DiskCache, detect_encoding,
AsyncSession, aget, apost, WebSocket, AsyncWebSocket,
render, render_if_empty, BrowserSession,
Selector, SelectorList, signature, similarity, save, relocate,
Spider, SitemapSpider, Request, Reaper, run,
Pipeline, DedupPipeline, JsonLinesPipeline, CsvPipeline, SqlitePipeline,
pipelines, AutoThrottle, Geocoder, geocode, __version__
Transport
Module functions (exported)
reap.get(url, **kw) -> Response: GET through a shared module-levelSession.reap.post(url, **kw) -> Response: POST through the shared session.reap.fetch(url, **kw) -> Response: an alias ofget.reap.download(url, path, **kw) -> Response: stream to a file through the shared session.async reap.aget(url, **kw)andasync reap.apost(url, **kw): one request on a temporaryAsyncSession.reap.detect_encoding(content, headers=None): charset from a byte order mark,Content-Type, or HTML meta tags, orNone.
The keywords **kw are the per-request options listed in Sessions.
Session (exported)
Session(profile="chrome", headers=None, timeout=30, retries=2, proxy=None,
rotate=None, profiles=None, retry_policy=None, cache=None,
on_response=None, block_rotations=None, verify=True,
allow_redirects=True, max_redirects=30, impersonate=None,
http2=True, sticky=False, url_policy=None, tls="auto",
max_body_bytes=67108864, interface=None, debug=False,
trust_env=True, cert=None, proxies=None, http_version=None)
- Methods:
request(method, url, **kw),get,post,head,put,patch,delete,options,download(url, path, **kw),ws_connect(url, **kw),close(). Context manager. - Attributes:
profile(the resolvedProfile),profiles(the rotation pool),cookies(aCookieJar),retry_policy,timeout,cache,verify,cert,http2,http_version,sticky,url_policy,max_body_bytes,debug,trust_env,block_rotations. Properties:retries,tls_backend(the configuredtls=mode). rotatemust beNone,"sequence"or"random";tlsmust be"auto","native"or"openssl";http_versionmust be one of"v1","v2","v2tls","v2_prior_knowledge","v3","v3only", and the last two raiseNotImplementedError.
Options are described in Sessions, retries, proxies and cookies.
AsyncSession (exported)
AsyncSession(profile="chrome", headers=None, timeout=30, retries=2, proxy=None,
rotate=None, profiles=None, retry_policy=None, cache=None,
on_response=None, block_rotations=None, verify=True,
allow_redirects=True, max_redirects=30, impersonate=None,
http2=True, sticky=False, url_policy=None, max_clients=10,
tls="auto", **kwargs)
Extra **kwargs go to Session (for example trust_env, proxies, cert, http_version). max_clients bounds concurrent requests with a semaphore. Awaitable methods: request, get, post, head, put, patch, delete, options, download, ws_connect, close. Properties: profile, retries, cookies. Async context manager. Runs the synchronous session on worker threads.
Response (exported)
Response(raw=None, meta=None, elapsed=0.0, *, status=None, url=None, headers=None, content=None, text=None, from_cache=False). Sessions create responses for you; the keyword form is handy for tests and for parsing saved HTML.
Status:
status,status_code(alias),reason.ok:Truefor 200 to 399.is_success:Truefor 2xx only.is_redirect: 3xx with aLocationheader.redirect_url:Locationresolved againsturl, orNone.redirect_count:len(history).raise_for_status(): raisesHTTPErrorunlessis_success; returns the response otherwise.
Body:
content: the decoded body bytes.text: decoded lazily on first access withencoding, using replacement characters for invalid bytes. Settable.encoding: detected from a byte order mark,Content-Typeor HTML meta tags, elsedefault_encoding("utf-8"). Setting it clears the cachedtext, so the next access re-decodes.charset(aliascharset_encoding): the charset the body declares, orNone.json(**kw): parses the body withjson.loads(a bad body raises the standard library’sjson.JSONDecodeError).iter_content(chunk_size=65536),iter_lines(chunk_size=65536, delimiter=None),close(). See Caching and downloads.
Metadata:
url,headers(aHeaders),cookies(dict of cookies set by this response),history(redirect responses),meta,from_cache,raw.elapsed: adatetime.timedelta.elapsed_seconds: the same as a float.http_version("HTTP/1.1"or"HTTP/2") andhttp_version_int(libcurl’s numbering: 2 for HTTP/1.1, 3 for HTTP/2).infos: connection details. On every response that came over the network it holdstls_backend:"native","openssl"or"plain". A cache hit or a rendered response has an emptyinfos. On TLS connections it also holdsalpn,cipher,groupandtls_version, plusresumedandhandshake_secondson the native engine;peernameandsocknamewhen known;http_versionon HTTP/2.primary_ip,primary_port,local_ip,local_port,request_size,response_size.request: a dict snapshot of what was sent (method,url,headers).looks_like_challenge:Truewhen the body looks like a bot wall.block_reason:"forbidden","proxy_auth_required","rate_limited","challenge"orNone.
Parsing:
urljoin(url),follow(target, callback=None, **kw),selector().- Selector passthroughs:
css,css_first,xpath,re,re_first,find_by_text,find_similar,save,extract(fields, **kw). - Extraction:
jsonld(),meta_tags(),links(**kw),images(**kw),tables(),markdown(max_len=None, main_only=False).
Pickling: a Response can be pickled. The pickled state drops raw, any open stream and the cached selector; status, URL, headers, content, text, encoding, cookies, history, timing and meta survive.
Headers (in curl_reap.headers)
Not exported at the package root: from curl_reap.headers import Headers. Response.headers is an instance. CaseInsensitiveHeaders in curl_reap.http is the same class, kept as the 1.x name.
Headers(headers=None, encoding=None) accepts a mapping, (name, value) pairs, a raw header block, or another Headers. It is a dict subclass, ordered, case-insensitive and multi-value:
h[name]joins duplicate values with", ";get(name, default=None).get_list(name, split_commas=False): every value in order.multi_items(): every(name, value)pair in wire order.raw(): the same as latin-1 bytes.h[name] = valuereplaces every occurrence;add(name, value)appends.del,pop,popitem,setdefault,update,clear,copy,merge(*sources),keys,values,items.
RetryPolicy (exported)
RetryPolicy(retries=2, backoff=0.5, max_backoff=30.0, statuses=RETRY_STATUSES, respect_retry_after=True, allowed_methods=None)
RETRY_STATUSES (in curl_reap.http) is 408, 425, 429, 500, 502, 503, 504, 520, 521, 522, 523 and 524. allowed_methods=None means GET, HEAD, OPTIONS, PUT and DELETE. Methods: should_retry_status(status), allows_method(method), delay(attempt, retry_after=None). Retry-After is clamped to 24 hours and is not capped by max_backoff. Negative values raise ValueError.
CookieJar (exported) and curl_reap.cookies
curl_reap.cookies exports CookieJar, Cookies, CookieConflict and parse_set_cookie.
CookieJar(): thread-safe, scoped by domain, path and name. Methods:absorb(url, specs),store(url, pairs),set(name, value, domain=None, path="/", secure=False),get(name, default=None, domain=None, path=None),delete(name, domain=None, path=None),update(cookies),header_for(url, extra=None),get_dict(),items(),save(path),load(path),to_cookiejar(),clear(),len()and iteration over cookie dicts.parse_set_cookie(value): oneSet-Cookievalue to a dict (name,valueand the known attributes), orNone.Cookies(cookies=None): a mutablename -> valuemapping over a jar, withset,get,deleteandupdate. It raisesCookieConflictwhen several cookies share a name.
See Cookies for the scoping rules.
DiskCache (exported)
DiskCache(directory=".reap_cache", ttl=3600.0, methods=("GET",)). Methods: accepts(method), get(method, url, params=None), get_stale(method, url, params=None), touch(method, url, params=None), set(method, url, params, resp), clear(). See Caching and downloads.
WebSocket (exported)
WebSocket.connect(url, *, session=None, subprotocols=(), headers=None, timeout=None, profile=None, proxy=None, verify=True)Session.ws_connect(url, **kw)returns aWebSocketbound to the session.await AsyncSession.ws_connect(url, **kw)returns anAsyncWebSocket.
See WebSocket.
Profiles (exported)
Profile(name, family="custom", version="", headers=(), h2_headers=None, tls_template=None, h2_spec=None, accept_encoding="", form_boundary="", ciphers=(), alpn_protocols=("h2", "http/1.1"), curve="X25519", minimum_version="TLSv1_2"): a frozen dataclass.PROFILES: dict ofchrome,firefox,safari.FINGERPRINTS: dict of all 79 target names (38 captured targets, 19 aliases, 22 legacy names). See Impersonation targets.
The module curl_reap.tls also defines resolve_profile(profile) and create_ssl_context(profile="chrome", verify=True, alpn=("http/1.1",)). They are not exported at the package root.
Rendering (optional js extra, exported)
render(url, *, timeout=30, wait_until="networkidle", proxy=None, user_agent=None, browser_channel="chromium", headless=True, script=None) -> Responserender_if_empty(response, *, min_bytes=2000, **kw) -> (Response, Response | None)BrowserSession(*, timeout=30, wait_until="domcontentloaded", proxy=None, user_agent=None, browser_channel="chromium", headless=True, cdp_url=None, url_policy=None)withgoto(url, *, wait_for=None, script=None, timeout=None, wait_until=None),follow(target, **kw),close(), and thepageandurlproperties.
See JavaScript rendering.
Exceptions
The package root exports TransportError, HTTPStatusError and URLPolicyError. The full set lives in curl_reap.exceptions. Names follow the wider Python HTTP ecosystem, and each class carries a libcurl-style error number in .code (shown below), plus .response when a response is involved. Every class except URLPolicyError is an OSError.
OSError
└── RequestException (alias RequestsError) code 0
├── TransportError code 0
│ ├── ConnectionError code 7
│ │ ├── DNSError code 6
│ │ ├── ProxyError code 5
│ │ ├── SSLError code 35
│ │ │ └── CertificateVerifyError code 60
│ │ └── InterfaceError code 45
│ ├── Timeout code 28
│ │ ├── ConnectTimeout (also ConnectionError) code 28
│ │ └── ReadTimeout code 28
│ ├── TooManyRedirects code 47
│ ├── IncompleteRead code 18
│ ├── ChunkedEncodingError code 56
│ ├── ContentDecodingError code 61
│ └── curl_reap._h2.H2Error code 16
├── InvalidURL (also ValueError) code 3
├── InvalidSchema (also ValueError) code 1
├── InvalidHeader (also ValueError) code 43
├── ImpersonateError code 0
├── SessionClosed code 0
├── StreamConsumedError code 0
├── UnrewindableBodyError code 0
├── HTTPError (alias HTTPStatusError) code 22
├── CookieConflict code 0
├── InvalidJSONError code 0
│ └── JSONDecodeError (also ValueError) code 0
└── WebSocketError code 0
├── WebSocketClosed code 0
└── WebSocketTimeout (also Timeout) code 28
ValueError
└── URLPolicyError
HTTPErroris raised byraise_for_status(); the response is on.response. It is not aTransportError, soexcept TransportErrordoes not catch it.TooManyRedirectsis also raised for a redirect refused byallow_redirects="safe"; the redirect response is on.response.H2Erroraddserror_code,error_name,stream_idandretryable. The session reports HTTP/2 failures asTransportErrorwith theH2Erroras__cause__. See HTTP/2.WebSocketClosedhasclose_codeandreason.curl_reap.exceptionsalso exportsCurlECode(anIntEnumof libcurl error numbers) andCODE2ERROR, which maps a code to the class above.
Parsing
Selector (exported)
Selector(content=None, element=None, url=None, status=None, headers=None)
css(query, auto_match=False, identifier=None, storage=None) -> SelectorListcss_first(query, default=None, **kw)xpath(query) -> SelectorListfind_by_text(text, partial=True, first=False, deepest=True) -> SelectorListfind_similar(sample, threshold=0.6, limit=None) -> SelectorListsave(identifier, storage=None): returns the selector.re(pattern, flags=0) -> SelectorList,re_first(pattern, default=None, flags=0)extract(fields, *, many=False):fieldsmaps names to CSS queries; returns each first match (or all matches withmany=True).- Properties:
text,attrib,html,base_url. Method:attr(name, default=None). - Extraction:
jsonld(),meta_tags(),links(absolute=True, unique=True, internal_only=False),images(absolute=True),tables(),markdown(max_len=None, main_only=False).
SelectorList (exported)
A list subclass with get(default=None), getall(), text(), attr(name, default=None) and css(query, **kw).
Self-healing functions (exported)
signature(el) -> dictsimilarity(a, b) -> float(0 to 1)save(identifier, el, storage=None): default store.reap_selectors.json.relocate(identifier, tree, storage=None, threshold=0.6): an lxml element orNone.
See Selectors and self-healing.
Crawling
Spider and SitemapSpider (exported)
Spider: class attributesname="reap",start_urls=[],allowed_domains=None,max_depth=None,custom_headers=None; methodsstart()andparse(page)(must be implemented).SitemapSpider(Spider): class attributessitemap_urls=[],url_pattern=None; methodparse_sitemap(page).
Request (exported)
Request(url, callback=None, method="GET", meta=None, priority=0, errback=None, dont_filter=False, **kw). Extra keywords go to Session.request. Method: fingerprint().
run and Reaper (exported)
run(spider, **kw) -> list: accepts a class or instance.Reaper(spider, concurrency=8, retries=2, throttle=True, delay=0.0, profile="chrome", pipelines=None, dedup=True, on_item=None, max_pages=None, max_depth=None, respect_robots=False, session=None, cache=None, rotate=None, proxy=None, collect_items=True, per_domain_concurrency=None, max_pending=None)Reaperattributes:run(),stats,items,session,pipelines,discovered_sitemaps.
See Crawling.
AutoThrottle (exported)
AutoThrottle(base_delay=0.0, target_concurrency=8, max_delay=30.0, enabled=True) with observe(latency, status=None), wait() and the current delay.
Pipelines (exported)
Pipeline: base class withopen(),process(item),close().DedupPipeline(key=None)JsonLinesPipeline(path)CsvPipeline(path)SqlitePipeline(path, table="items", key=None)pipelines: thecurl_reap.pipelinesmodule.
See Pipelines.
Extras
Geocoder(endpoint="https://nominatim.openstreetmap.org/search", user_agent=DEFAULT_UA, cache=".reap_geocode.json", min_interval=1.1, profile="chrome", min_importance=0.0)withgeocode(name=None, area=None, city=None, country=None)andclean_name(name).geocode(name=None, area=None, city=None, country=None): uses a shared defaultGeocoder.__version__: the package version string ("2.0.0").
Fingerprint check (in curl_reap.fingerprint)
Not exported at the package root: from curl_reap.fingerprint import check, EXPECTED.
check(target="chrome", url="https://tls.peet.ws/api/all", session=None) -> dict: fetches the echo service astargetand compares the result with the capture. Withoutsessionit creates aSession(impersonate=target)and closes it afterwards. Any object withget(url)returning something withjson()works assession. RaisesKeyErrorfor an unknown target.- The report has
target,capture(the captured target whose values apply; it differs fromtargetfor aliases and legacy names), the observedja3,ja4,peetprintandakamai,expected,matches,headers(the header names in the order the service saw them),expected_headersandhttp_version. matchesholdsja4,ja4_r,akamai,ja3andheaders, eachTrue,False, orNonewhen it cannot be compared.EXPECTED: the captured values per target (ja3,ja3_fixed,ja4,ja4_r,akamai).
reap doctor is the command-line front end. See Command line.
See Geocoding.