curl_reap / Guides
Sessions, retries, proxies and cookies
Configure a curl_reap 2.0 Session: request keywords, timeouts, retries, proxies, the cookie jar, headers, redirects and TLS backends.
On this page
A Session holds everything that should carry over between Hypertext Transfer Protocol (HTTP) requests. This page lists its options and the per-request keywords, then covers timeouts, retries, proxies, headers, cookies, redirects, Transport Layer Security (TLS) verification, streaming and the async wrapper.
First principles
- A session is state that outlives one request: a connection pool, a cookie jar, default options and a rotation counter. Reusing one session reuses all of it. A new session starts cold, with new handshakes and no cookies.
- A retry sends the request again. Repeating a
GETis harmless; repeating aPOSTcan place a second order. So only methods that are safe to repeat are retried unless you opt in. - Credentials belong to an origin. When a redirect leaves the origin (scheme, host and port), the session drops
Authorization,CookieandProxy-Authorizationso they never reach a host you did not choose. - A proxy changes where a connection comes from, not what it looks like. The profile still sets the handshake and headers; rotation advances the profile and the proxy together on each attempt.
One-shot calls and sessions
reap.get, reap.post, reap.fetch (an alias of get) and reap.download use one shared module-level Session with default settings. Every keyword they accept is a per-request option, listed below.
For anything beyond a quick fetch, create your own session. It pools connections, keeps a cookie jar, and holds defaults:
import curl_reap as reap
with reap.Session(profile="chrome", timeout=20, retries=3) as s:
r = s.get("https://example.com/search", params={"q": "lamps"})
r.raise_for_status()
print(r.status, r.headers.get("content-type"), len(r.content))
Methods: request(method, url, **kw), get, post, head, put, patch, delete, options, download(url, path, **kw), ws_connect(url, **kw) (see WebSocket) and close(). A session is a context manager and is safe to share between threads.
Session options
Session(...) takes these keyword arguments:
| Option | Default | Meaning |
|---|---|---|
profile |
"chrome" |
Profile name, impersonate target or Profile. See Transport and impersonation profiles. |
impersonate |
None |
Impersonate target; overrides profile when set. |
headers |
None |
Mapping, (name, value) pairs or Headers added to every request. |
timeout |
30 |
Seconds, a (connect, read) tuple, or None. See Timeouts. |
retries |
2 |
Retry count for the default retry policy. |
retry_policy |
None |
A RetryPolicy; replaces retries when given. |
proxy |
None |
One proxy URL, or a list of them for rotation. |
proxies |
None |
A proxy URL or a dict such as {"https": ..., "http": ..., "all://": ...}. See Proxies. |
trust_env |
True |
Use HTTPS_PROXY, HTTP_PROXY, ALL_PROXY and NO_PROXY when no proxy is configured. |
rotate |
None |
"sequence" or "random" profile and proxy rotation. |
profiles |
None |
A custom rotation pool of profiles. |
sticky |
False |
Pin the first profile and first proxy for the session’s life. |
block_rotations |
None |
How many times to switch proxy on a block status. Defaults to the proxy pool size minus one. |
cache |
None |
A DiskCache. See Caching and downloads. |
on_response |
None |
A callable that receives every final Response, including cache hits. |
verify |
True |
True for the system trust store, False to skip certificate checks, or a CA bundle path. |
cert |
None |
Client certificate: a path, a (cert, key) pair or a (cert, key, password) tuple. Such requests use OpenSSL. See Client certificates. |
tls |
"auto" |
TLS backend: "auto", "native" or "openssl". See TLS. |
allow_redirects |
True |
True, False or "safe". |
max_redirects |
30 |
-1 means no limit. |
http2 |
True |
Offer HTTP/2 through ALPN. See HTTP/2. |
http_version |
None |
"v1", "v2", "v2tls" or "v2_prior_knowledge"; overrides http2. |
max_body_bytes |
67108864 |
Cap for a buffered HTTP/1.1 body (64 MiB). |
url_policy |
None |
A callable checked against every outgoing URL, including redirect hops. A falsy result raises URLPolicyError. |
interface |
None |
Source IP address (or (host, port)) for every socket. |
debug |
False |
Log request and response headers to the curl_reap.transport logger at INFO level. |
Per-request options
Session.request and every shortcut accept:
params: query parameters, appended to any query already in the URL.data: bytes, str, a mapping or a list of pairs. Mappings and pairs are form-encoded withContent-Type: application/x-www-form-urlencoded.json: a value serialized as compact JSON withContent-Type: application/json. Cannot be combined withdata.files: a mapping or list of(name, spec)pairs sent asmultipart/form-data. A spec is a file object, bytes, str, or a tuple(filename, content[, content_type[, headers]]).datafields go first. Cannot be combined withjson. The boundary is Chrome-shaped (----WebKitFormBoundaryplus 16 characters).cookies: a dict, a list of pairs, or"name=value"strings. See Cookies.headers,timeout,verify,cert,allow_redirects,max_redirects,max_body_bytes,debug,http_version: override the session value.retriesorretry_policy: override the retry policy for this request.proxy: one proxy URL for this request. It turns off proxy rotation for the request.proxies: a proxy URL or dict for this request.proxy_auth:"user:password"for the proxy, used when the proxy URL has no credentials of its own.profileorimpersonate: a profile for this request. Passing both with different values raisesValueError.ja3,akamai,extra_fp: custom TLS and HTTP/2 fingerprints for this request (a JA3 string, an Akamai HTTP/2 string, and a dict withtls_cipher_suitesortls_curves).auth: a(username, password)pair for Basic auth.stream:Truereads the body as it arrives instead of buffering it, over HTTP/1.1. See Streaming responses.meta: a dict copied ontoResponse.meta.no_cache:Trueskips the session cache.interface: accepted, but the connection pool uses the value set on theSession.
Any other keyword raises TypeError, with two exceptions kept for compatibility with code written for libcurl-based clients: curl_options, curl_infos, thread, curl, use_thread_local_curl, async_curl and loop are accepted and ignored with one warning per process, and perk is accepted and ignored with a warning (the post-quantum key share is negotiated automatically).
Timeouts
- A number bounds one request attempt as a whole: connecting, sending and receiving must all finish within that many seconds.
- A
(connect, read)tuple bounds the connect step and each socket read separately, and sets an overall deadline ofconnect + readfor the attempt. Either value can beNone. Nonedisables timeouts.
The deadline starts again on each redirect hop and on each retry. A timeout raises ConnectTimeout or ReadTimeout, both subclasses of Timeout and TransportError.
s.get("https://example.com", timeout=(3.05, 27))
Retries and backoff
The default RetryPolicy(retries=2, backoff=0.5, max_backoff=30.0, respect_retry_after=True) retries:
- transport failures (connection, TLS, proxy, framing and timeout errors), and
- responses with status 408, 425, 429, 500, 502, 503, 504, 520, 521, 522, 523 or 524,
but only for methods in allowed_methods, which defaults to GET, HEAD, OPTIONS, PUT and DELETE. A POST or PATCH is not retried unless you opt in:
from curl_reap import RetryPolicy, Session
policy = RetryPolicy(retries=5, backoff=1.0, max_backoff=60,
statuses={429, 503}, allowed_methods={"GET", "POST"})
s = Session(retry_policy=policy)
The wait before attempt n (starting at 0) is backoff * 2**n multiplied by a random factor between 0.5 and 1.5, capped at max_backoff. When a retryable response carries Retry-After (seconds or an HTTP date) and respect_retry_after is on, that value is used instead. It is clamped to 24 hours; max_backoff does not cap it.
After the last retry, a transport failure raises (a TransportError subclass) and a retryable status is returned as a normal Response. Pass retries=0 on requests that must never repeat.
Proxies
Supported proxy schemes are http, https, socks4, socks4a, socks5 and socks5h. HTTPS targets go through an HTTP or HTTPS proxy with CONNECT host:port. Credentials in an HTTP proxy URL are sent as Proxy-Authorization: Basic. A proxy URL without a scheme is treated as http://. Default ports are 8080 for HTTP and HTTPS proxies and 1080 for Socket Secure (SOCKS) proxies. An https:// proxy works only on the native TLS path. If a target may only speak TLS 1.2, or a request uses the OpenSSL path for another reason, read TLS 1.2 servers and proxies first.
s = reap.Session(proxy="http://user:pass@proxy.example:8000")
s = reap.Session(proxy="socks5h://proxy.example:1080")
s.get("https://example.com", proxy="http://proxy.example:8000", proxy_auth="user:pass")
A list of proxies makes a pool. Each attempt takes the next proxy, in sequence or at random depending on rotate:
s = reap.Session(proxy=["http://p1:8080", "http://p2:8080", "http://p3:8080"],
rotate="random")
The proxies dict is read in the key order https, http, all://, all. curl_reap does not pick a proxy by URL scheme: on a Session, every distinct value joins the rotation pool (after any proxy= entries); on a single request, the first value found in that order is used.
Environment proxies
With trust_env=True (the default) and no proxy from proxy, proxies or the session pool, the session reads HTTPS_PROXY, HTTP_PROXY and ALL_PROXY (upper or lower case) for https:// and http:// URLs. The variables are read once per process, on first use. NO_PROXY is a comma-separated list of host names: a host matches when it equals an entry or ends with . plus the entry. A leading dot on an entry is ignored, and * is not treated as a wildcard. Pass trust_env=False to ignore the environment.
Rotating off a block
With a proxy pool, a response with status 403, 407 or 429 makes the session switch to the next proxy and try again, up to block_rotations times (by default, pool size minus one). This happens before and in addition to normal retries. It is skipped when the request passes its own proxy=, when the session is sticky, and for methods outside allowed_methods.
Headers
Request and response headers use Headers (in curl_reap.headers): an ordered, case-insensitive, multi-value mapping that is also a dict subclass.
from curl_reap.headers import Headers
h = Headers([("Set-Cookie", "a=1"), ("set-cookie", "b=2")])
h["set-cookie"] # "a=1, b=2"
h.get_list("Set-Cookie") # ["a=1", "b=2"]
h.add("X-Trace", "1") # append a value
h["X-Trace"] = "2" # replace every value
h.multi_items() # every (name, value) pair in wire order
Session headers and request headers merge over the profile’s headers, matching names case-insensitively: a name you set replaces the profile’s value in its position, and new names are appended. Request headers win over session headers.
s = reap.Session(headers={"Accept-Language": "de-DE,de;q=0.9"})
s.get("https://example.com/file.bin", headers={"Accept-Encoding": "gzip, deflate, br"})
Cookies
Every session has a CookieJar at s.cookies (from curl_reap.cookies). Set-Cookie headers are parsed the RFC 6265bis way: the first name=value pair is the cookie, attributes are matched case-insensitively, and unknown attributes are ignored (a Partitioned cookie is stored, an unknown key=value never becomes a cookie). Cookies are stored automatically, including on each redirect hop, and replayed on matching requests.
- Cookies are scoped by domain, path and name. Same-named cookies on different hosts or paths coexist.
- A cookie without a
Domainattribute is host-only and never reaches subdomains. ADomainattribute is accepted only when it names the host or a parent of it; it then matches that domain and its subdomains. Public-suffix lists are not checked. - Path matching respects segment boundaries (
/appdoes not match/apple), and longer paths are sent first. Securecookies are only accepted fromhttps://responses and only sent tohttps://URLs. There is no exception forlocalhost.- Expired cookies, and cookies with
Max-Age=0, are dropped.
s = reap.Session()
s.get("https://example.com/login")
print(s.cookies.get_dict()) # {"sessionid": "..."}
s.cookies.get("sessionid", domain="example.com")
s.cookies.set("consent", "yes", domain="example.com")
s.cookies.save("cookies.json") # and s.cookies.load("cookies.json")
std = s.cookies.to_cookiejar() # an http.cookiejar.CookieJar
Jar methods: absorb(url, specs), store(url, pairs), set(name, value, domain=None, path="/", secure=False), get(name, default=None, domain=None, path=None), delete(name, domain=None, path=None), update(cookies), header_for(url), get_dict(), items(), save(path), load(path), to_cookiejar(), clear(), plus len() and iteration over cookie dicts. get raises CookieConflict when several scopes hold the same name; pass domain or path to pick one. set without domain stores an unscoped cookie that is sent to every host. get_dict() is a flattened view: same-named cookies collapse to the most recent one.
Per-request cookies and your own Cookie header:
cookies={...}on a request sends exactly those cookies. The jar’s cookies are not added to that request.- A
Cookieheader set on the session or the request also stops the jar from adding cookies, until a redirect moves to another origin. Response.cookiesis a dict of the cookies set by that response.
Redirects
Redirects with status 301, 302, 303, 307 and 308 are followed up to max_redirects (default 30); one more raises TooManyRedirects, with the last response on .response.
- A 303, or a 301 or 302 answer to a
POST, switches toGETand drops the body and its content headers. AHEADstays aHEAD. A 307 or 308 keeps the method and body. - When a redirect moves to a different origin (scheme, host or port),
auth=credentials and anyAuthorization,Cookie,Proxy-AuthorizationandHostheaders you set are dropped. This also covers anhttpstohttpdowngrade on the same host. - A UTF-8
Locationis not double-encoded. allow_redirects="safe"follows redirects but refuses one whose target resolves to a private, loopback, link-local, multicast, reserved or unspecified address, raisingTooManyRedirects.- With
allow_redirects=False, the 3xx response is returned;r.redirect_urlis itsLocationresolved against the URL. Response.urlis the final URL andResponse.historyholds the redirect responses.
Authentication
s.get("https://api.example.com/me", auth=("user", "secret"))
s.get("https://user:secret@api.example.com/me") # same, from the URL
TLS verification and backends
s.get("https://self-signed.example", verify=False)
s.get("https://internal.example", verify="/etc/ssl/internal-ca.pem")
tls= picks the TLS implementation:
"auto"(default): the native TLS 1.3 engine for profiles with a captured ClientHello (all built-in targets). It uses the standard-library OpenSSL stack for requests withcert=, when the server only speaks TLS 1.2, and when the native handshake fails."native": the native engine. A TLS-1.2-only server still uses OpenSSL; other native failures raise, andcert=raisesValueError."openssl": always the standard-librarysslmodule.
A handshake failure raises SSLError, and a certificate verification failure raises CertificateVerifyError. Both are TransportError subclasses. r.infos["tls_backend"] tells you which backend a response used.
Streaming responses
stream=True skips the cache and lets r.iter_content() and r.iter_lines() read the body as it arrives:
import curl_reap as reap
with reap.Session(http2=False) as s:
r = s.get("https://example.com/export.csv", stream=True)
for line in r.iter_lines():
print(line.decode())
- Over HTTP/1.1 the body is not buffered:
r.contentisNone, and memory stays bounded by the chunk size. Agzip,deflateorbrbody is decoded as it streams. Any otherContent-Encoding, includingzstd, raisesContentDecodingErrorwhen the request is made, and a truncated compressed body raises it while you read. When the body has been read to the end, the connection goes back to the pool.r.close()abandons the rest of the body and closes the connection. - Over HTTP/2 the body is still buffered (up to 64 MiB) and
iter_content()replays it. Passhttp2=Falsewhen you need a true stream from a server that offers HTTP/2.
Many built-in targets (41 of 79, including chrome and firefox) advertise zstd in Accept-Encoding, so for streamed reads send your own, for example headers={"Accept-Encoding": "gzip, deflate, br"}.
To write a large body to disk, download() is simpler. See Caching and downloads. iter_content() and iter_lines() also work on any normal (non-streamed) response, replaying the buffered body in chunks.
The Response object
The Python API reference lists every attribute. The main points for 2.0:
okisTruefor any status from 200 to 399. Useis_successfor 2xx only.raise_for_status()raisesHTTPError(aliasHTTPStatusError) for anything that is not 2xx.textis decoded lazily on first access.encodingis detected from a byte order mark,Content-Typeor HTML meta tags (falling back to UTF-8), and you can set it to re-decode.elapsedis adatetime.timedelta;elapsed_secondsis the float.headersis aHeadersmulti-dict.
Async
AsyncSession takes the same arguments as Session plus max_clients=10, which bounds concurrent in-flight requests with a semaphore. It has awaitable request, get, post, head, put, patch, delete, options, download, ws_connect and close, a cookies property, and is an async context manager. reap.aget and reap.apost run one request on a temporary session.
import asyncio
import curl_reap as reap
async def main():
async with reap.AsyncSession(profile="chrome") as s:
pages = await asyncio.gather(*(s.get(u) for u in [
"https://example.com/a", "https://example.com/b"]))
print([p.status for p in pages])
asyncio.run(main())
The async facade runs each request on a worker thread with asyncio.to_thread, using the same synchronous transport. Cancelling a task stops the wait but cannot interrupt a socket call already in progress; the request timeout still bounds it.