Skip to content
curl_reap

curl_reap / Guides

Sessions, retries, proxies and cookies

Configure a curl_reap 2.0 Session: request keywords, timeouts, retries, proxies, the cookie jar, headers, redirects and TLS backends.

On this page
  1. First principles
  2. One-shot calls and sessions
  3. Session options
  4. Per-request options
  5. Timeouts
  6. Retries and backoff
  7. Proxies
  8. Environment proxies
  9. Rotating off a block
  10. Headers
  11. Cookies
  12. Redirects
  13. Authentication
  14. TLS verification and backends
  15. Streaming responses
  16. The Response object
  17. Async

A Session holds everything that should carry over between Hypertext Transfer Protocol (HTTP) requests. This page lists its options and the per-request keywords, then covers timeouts, retries, proxies, headers, cookies, redirects, Transport Layer Security (TLS) verification, streaming and the async wrapper.

First principles

  • A session is state that outlives one request: a connection pool, a cookie jar, default options and a rotation counter. Reusing one session reuses all of it. A new session starts cold, with new handshakes and no cookies.
  • A retry sends the request again. Repeating a GET is harmless; repeating a POST can place a second order. So only methods that are safe to repeat are retried unless you opt in.
  • Credentials belong to an origin. When a redirect leaves the origin (scheme, host and port), the session drops Authorization, Cookie and Proxy-Authorization so they never reach a host you did not choose.
  • A proxy changes where a connection comes from, not what it looks like. The profile still sets the handshake and headers; rotation advances the profile and the proxy together on each attempt.

One-shot calls and sessions

reap.get, reap.post, reap.fetch (an alias of get) and reap.download use one shared module-level Session with default settings. Every keyword they accept is a per-request option, listed below.

For anything beyond a quick fetch, create your own session. It pools connections, keeps a cookie jar, and holds defaults:

import curl_reap as reap

with reap.Session(profile="chrome", timeout=20, retries=3) as s:
    r = s.get("https://example.com/search", params={"q": "lamps"})
    r.raise_for_status()
    print(r.status, r.headers.get("content-type"), len(r.content))

Methods: request(method, url, **kw), get, post, head, put, patch, delete, options, download(url, path, **kw), ws_connect(url, **kw) (see WebSocket) and close(). A session is a context manager and is safe to share between threads.

Session options

Session(...) takes these keyword arguments:

Option Default Meaning
profile "chrome" Profile name, impersonate target or Profile. See Transport and impersonation profiles.
impersonate None Impersonate target; overrides profile when set.
headers None Mapping, (name, value) pairs or Headers added to every request.
timeout 30 Seconds, a (connect, read) tuple, or None. See Timeouts.
retries 2 Retry count for the default retry policy.
retry_policy None A RetryPolicy; replaces retries when given.
proxy None One proxy URL, or a list of them for rotation.
proxies None A proxy URL or a dict such as {"https": ..., "http": ..., "all://": ...}. See Proxies.
trust_env True Use HTTPS_PROXY, HTTP_PROXY, ALL_PROXY and NO_PROXY when no proxy is configured.
rotate None "sequence" or "random" profile and proxy rotation.
profiles None A custom rotation pool of profiles.
sticky False Pin the first profile and first proxy for the session’s life.
block_rotations None How many times to switch proxy on a block status. Defaults to the proxy pool size minus one.
cache None A DiskCache. See Caching and downloads.
on_response None A callable that receives every final Response, including cache hits.
verify True True for the system trust store, False to skip certificate checks, or a CA bundle path.
cert None Client certificate: a path, a (cert, key) pair or a (cert, key, password) tuple. Such requests use OpenSSL. See Client certificates.
tls "auto" TLS backend: "auto", "native" or "openssl". See TLS.
allow_redirects True True, False or "safe".
max_redirects 30 -1 means no limit.
http2 True Offer HTTP/2 through ALPN. See HTTP/2.
http_version None "v1", "v2", "v2tls" or "v2_prior_knowledge"; overrides http2.
max_body_bytes 67108864 Cap for a buffered HTTP/1.1 body (64 MiB).
url_policy None A callable checked against every outgoing URL, including redirect hops. A falsy result raises URLPolicyError.
interface None Source IP address (or (host, port)) for every socket.
debug False Log request and response headers to the curl_reap.transport logger at INFO level.

Per-request options

Session.request and every shortcut accept:

  • params: query parameters, appended to any query already in the URL.
  • data: bytes, str, a mapping or a list of pairs. Mappings and pairs are form-encoded with Content-Type: application/x-www-form-urlencoded.
  • json: a value serialized as compact JSON with Content-Type: application/json. Cannot be combined with data.
  • files: a mapping or list of (name, spec) pairs sent as multipart/form-data. A spec is a file object, bytes, str, or a tuple (filename, content[, content_type[, headers]]). data fields go first. Cannot be combined with json. The boundary is Chrome-shaped (----WebKitFormBoundary plus 16 characters).
  • cookies: a dict, a list of pairs, or "name=value" strings. See Cookies.
  • headers, timeout, verify, cert, allow_redirects, max_redirects, max_body_bytes, debug, http_version: override the session value.
  • retries or retry_policy: override the retry policy for this request.
  • proxy: one proxy URL for this request. It turns off proxy rotation for the request.
  • proxies: a proxy URL or dict for this request.
  • proxy_auth: "user:password" for the proxy, used when the proxy URL has no credentials of its own.
  • profile or impersonate: a profile for this request. Passing both with different values raises ValueError.
  • ja3, akamai, extra_fp: custom TLS and HTTP/2 fingerprints for this request (a JA3 string, an Akamai HTTP/2 string, and a dict with tls_cipher_suites or tls_curves).
  • auth: a (username, password) pair for Basic auth.
  • stream: True reads the body as it arrives instead of buffering it, over HTTP/1.1. See Streaming responses.
  • meta: a dict copied onto Response.meta.
  • no_cache: True skips the session cache.
  • interface: accepted, but the connection pool uses the value set on the Session.

Any other keyword raises TypeError, with two exceptions kept for compatibility with code written for libcurl-based clients: curl_options, curl_infos, thread, curl, use_thread_local_curl, async_curl and loop are accepted and ignored with one warning per process, and perk is accepted and ignored with a warning (the post-quantum key share is negotiated automatically).

Timeouts

  • A number bounds one request attempt as a whole: connecting, sending and receiving must all finish within that many seconds.
  • A (connect, read) tuple bounds the connect step and each socket read separately, and sets an overall deadline of connect + read for the attempt. Either value can be None.
  • None disables timeouts.

The deadline starts again on each redirect hop and on each retry. A timeout raises ConnectTimeout or ReadTimeout, both subclasses of Timeout and TransportError.

s.get("https://example.com", timeout=(3.05, 27))

Retries and backoff

The default RetryPolicy(retries=2, backoff=0.5, max_backoff=30.0, respect_retry_after=True) retries:

  • transport failures (connection, TLS, proxy, framing and timeout errors), and
  • responses with status 408, 425, 429, 500, 502, 503, 504, 520, 521, 522, 523 or 524,

but only for methods in allowed_methods, which defaults to GET, HEAD, OPTIONS, PUT and DELETE. A POST or PATCH is not retried unless you opt in:

from curl_reap import RetryPolicy, Session

policy = RetryPolicy(retries=5, backoff=1.0, max_backoff=60,
                     statuses={429, 503}, allowed_methods={"GET", "POST"})
s = Session(retry_policy=policy)

The wait before attempt n (starting at 0) is backoff * 2**n multiplied by a random factor between 0.5 and 1.5, capped at max_backoff. When a retryable response carries Retry-After (seconds or an HTTP date) and respect_retry_after is on, that value is used instead. It is clamped to 24 hours; max_backoff does not cap it.

After the last retry, a transport failure raises (a TransportError subclass) and a retryable status is returned as a normal Response. Pass retries=0 on requests that must never repeat.

Proxies

Supported proxy schemes are http, https, socks4, socks4a, socks5 and socks5h. HTTPS targets go through an HTTP or HTTPS proxy with CONNECT host:port. Credentials in an HTTP proxy URL are sent as Proxy-Authorization: Basic. A proxy URL without a scheme is treated as http://. Default ports are 8080 for HTTP and HTTPS proxies and 1080 for Socket Secure (SOCKS) proxies. An https:// proxy works only on the native TLS path. If a target may only speak TLS 1.2, or a request uses the OpenSSL path for another reason, read TLS 1.2 servers and proxies first.

s = reap.Session(proxy="http://user:pass@proxy.example:8000")
s = reap.Session(proxy="socks5h://proxy.example:1080")
s.get("https://example.com", proxy="http://proxy.example:8000", proxy_auth="user:pass")

A list of proxies makes a pool. Each attempt takes the next proxy, in sequence or at random depending on rotate:

s = reap.Session(proxy=["http://p1:8080", "http://p2:8080", "http://p3:8080"],
                 rotate="random")

The proxies dict is read in the key order https, http, all://, all. curl_reap does not pick a proxy by URL scheme: on a Session, every distinct value joins the rotation pool (after any proxy= entries); on a single request, the first value found in that order is used.

Environment proxies

With trust_env=True (the default) and no proxy from proxy, proxies or the session pool, the session reads HTTPS_PROXY, HTTP_PROXY and ALL_PROXY (upper or lower case) for https:// and http:// URLs. The variables are read once per process, on first use. NO_PROXY is a comma-separated list of host names: a host matches when it equals an entry or ends with . plus the entry. A leading dot on an entry is ignored, and * is not treated as a wildcard. Pass trust_env=False to ignore the environment.

Rotating off a block

With a proxy pool, a response with status 403, 407 or 429 makes the session switch to the next proxy and try again, up to block_rotations times (by default, pool size minus one). This happens before and in addition to normal retries. It is skipped when the request passes its own proxy=, when the session is sticky, and for methods outside allowed_methods.

Headers

Request and response headers use Headers (in curl_reap.headers): an ordered, case-insensitive, multi-value mapping that is also a dict subclass.

from curl_reap.headers import Headers

h = Headers([("Set-Cookie", "a=1"), ("set-cookie", "b=2")])
h["set-cookie"]               # "a=1, b=2"
h.get_list("Set-Cookie")      # ["a=1", "b=2"]
h.add("X-Trace", "1")         # append a value
h["X-Trace"] = "2"            # replace every value
h.multi_items()               # every (name, value) pair in wire order

Session headers and request headers merge over the profile’s headers, matching names case-insensitively: a name you set replaces the profile’s value in its position, and new names are appended. Request headers win over session headers.

s = reap.Session(headers={"Accept-Language": "de-DE,de;q=0.9"})
s.get("https://example.com/file.bin", headers={"Accept-Encoding": "gzip, deflate, br"})

Cookies

Every session has a CookieJar at s.cookies (from curl_reap.cookies). Set-Cookie headers are parsed the RFC 6265bis way: the first name=value pair is the cookie, attributes are matched case-insensitively, and unknown attributes are ignored (a Partitioned cookie is stored, an unknown key=value never becomes a cookie). Cookies are stored automatically, including on each redirect hop, and replayed on matching requests.

  • Cookies are scoped by domain, path and name. Same-named cookies on different hosts or paths coexist.
  • A cookie without a Domain attribute is host-only and never reaches subdomains. A Domain attribute is accepted only when it names the host or a parent of it; it then matches that domain and its subdomains. Public-suffix lists are not checked.
  • Path matching respects segment boundaries (/app does not match /apple), and longer paths are sent first.
  • Secure cookies are only accepted from https:// responses and only sent to https:// URLs. There is no exception for localhost.
  • Expired cookies, and cookies with Max-Age=0, are dropped.
s = reap.Session()
s.get("https://example.com/login")
print(s.cookies.get_dict())                      # {"sessionid": "..."}
s.cookies.get("sessionid", domain="example.com")
s.cookies.set("consent", "yes", domain="example.com")
s.cookies.save("cookies.json")                   # and s.cookies.load("cookies.json")
std = s.cookies.to_cookiejar()                   # an http.cookiejar.CookieJar

Jar methods: absorb(url, specs), store(url, pairs), set(name, value, domain=None, path="/", secure=False), get(name, default=None, domain=None, path=None), delete(name, domain=None, path=None), update(cookies), header_for(url), get_dict(), items(), save(path), load(path), to_cookiejar(), clear(), plus len() and iteration over cookie dicts. get raises CookieConflict when several scopes hold the same name; pass domain or path to pick one. set without domain stores an unscoped cookie that is sent to every host. get_dict() is a flattened view: same-named cookies collapse to the most recent one.

Per-request cookies and your own Cookie header:

  • cookies={...} on a request sends exactly those cookies. The jar’s cookies are not added to that request.
  • A Cookie header set on the session or the request also stops the jar from adding cookies, until a redirect moves to another origin.
  • Response.cookies is a dict of the cookies set by that response.

Redirects

Redirects with status 301, 302, 303, 307 and 308 are followed up to max_redirects (default 30); one more raises TooManyRedirects, with the last response on .response.

  • A 303, or a 301 or 302 answer to a POST, switches to GET and drops the body and its content headers. A HEAD stays a HEAD. A 307 or 308 keeps the method and body.
  • When a redirect moves to a different origin (scheme, host or port), auth= credentials and any Authorization, Cookie, Proxy-Authorization and Host headers you set are dropped. This also covers an https to http downgrade on the same host.
  • A UTF-8 Location is not double-encoded.
  • allow_redirects="safe" follows redirects but refuses one whose target resolves to a private, loopback, link-local, multicast, reserved or unspecified address, raising TooManyRedirects.
  • With allow_redirects=False, the 3xx response is returned; r.redirect_url is its Location resolved against the URL.
  • Response.url is the final URL and Response.history holds the redirect responses.

Authentication

s.get("https://api.example.com/me", auth=("user", "secret"))
s.get("https://user:secret@api.example.com/me")              # same, from the URL

TLS verification and backends

s.get("https://self-signed.example", verify=False)
s.get("https://internal.example", verify="/etc/ssl/internal-ca.pem")

tls= picks the TLS implementation:

  • "auto" (default): the native TLS 1.3 engine for profiles with a captured ClientHello (all built-in targets). It uses the standard-library OpenSSL stack for requests with cert=, when the server only speaks TLS 1.2, and when the native handshake fails.
  • "native": the native engine. A TLS-1.2-only server still uses OpenSSL; other native failures raise, and cert= raises ValueError.
  • "openssl": always the standard-library ssl module.

A handshake failure raises SSLError, and a certificate verification failure raises CertificateVerifyError. Both are TransportError subclasses. r.infos["tls_backend"] tells you which backend a response used.

Streaming responses

stream=True skips the cache and lets r.iter_content() and r.iter_lines() read the body as it arrives:

import curl_reap as reap

with reap.Session(http2=False) as s:
    r = s.get("https://example.com/export.csv", stream=True)
    for line in r.iter_lines():
        print(line.decode())
  • Over HTTP/1.1 the body is not buffered: r.content is None, and memory stays bounded by the chunk size. A gzip, deflate or br body is decoded as it streams. Any other Content-Encoding, including zstd, raises ContentDecodingError when the request is made, and a truncated compressed body raises it while you read. When the body has been read to the end, the connection goes back to the pool. r.close() abandons the rest of the body and closes the connection.
  • Over HTTP/2 the body is still buffered (up to 64 MiB) and iter_content() replays it. Pass http2=False when you need a true stream from a server that offers HTTP/2.

Many built-in targets (41 of 79, including chrome and firefox) advertise zstd in Accept-Encoding, so for streamed reads send your own, for example headers={"Accept-Encoding": "gzip, deflate, br"}.

To write a large body to disk, download() is simpler. See Caching and downloads. iter_content() and iter_lines() also work on any normal (non-streamed) response, replaying the buffered body in chunks.

The Response object

The Python API reference lists every attribute. The main points for 2.0:

  • ok is True for any status from 200 to 399. Use is_success for 2xx only. raise_for_status() raises HTTPError (alias HTTPStatusError) for anything that is not 2xx.
  • text is decoded lazily on first access. encoding is detected from a byte order mark, Content-Type or HTML meta tags (falling back to UTF-8), and you can set it to re-decode.
  • elapsed is a datetime.timedelta; elapsed_seconds is the float.
  • headers is a Headers multi-dict.

Async

AsyncSession takes the same arguments as Session plus max_clients=10, which bounds concurrent in-flight requests with a semaphore. It has awaitable request, get, post, head, put, patch, delete, options, download, ws_connect and close, a cookies property, and is an async context manager. reap.aget and reap.apost run one request on a temporary session.

import asyncio
import curl_reap as reap


async def main():
    async with reap.AsyncSession(profile="chrome") as s:
        pages = await asyncio.gather(*(s.get(u) for u in [
            "https://example.com/a", "https://example.com/b"]))
    print([p.status for p in pages])

asyncio.run(main())

The async facade runs each request on a worker thread with asyncio.to_thread, using the same synchronous transport. Cancelling a task stops the wait but cannot interrupt a socket call already in progress; the request timeout still bounds it.