Skip to content
curl_reap

curl_reap / Guides

Caching and downloads

Cache GET responses on disk with atomic entries and ETag revalidation, and stream large downloads to disk in constant memory with curl_reap.

On this page
  1. First principles
  2. Disk cache
  3. Atomic writes
  4. Revalidation
  5. Caching in a crawl
  6. Streaming downloads
  7. Over HTTP/2
  8. iter_content, iter_lines and stream=True

This page covers two ways to avoid holding or refetching whole bodies: a disk cache for repeated GET requests, and downloads that stream to a file.

First principles

  • A cache trades freshness for speed. An entry younger than ttl is served without asking the server. An older one is checked with the server’s validators, an entity tag (ETag) or a Last-Modified date, which costs a request but not the body.
  • An entry is only useful if it is whole. Metadata and body live in one file that replaces the old one in a single atomic step, so a reader never sees half an entry.
  • A download’s memory use should not depend on the file’s size. Bytes go to disk as they arrive, and the file appears under its final name only when the response is complete and successful.

Disk cache

A DiskCache stores responses on disk so repeated GETs during development come from disk instead of the network.

import curl_reap as reap

s = reap.Session(cache=reap.DiskCache(ttl=3600))
r1 = s.get("https://example.com")    # network
r2 = s.get("https://example.com")    # disk
print(r2.from_cache)                   # True

DiskCache(directory=".reap_cache", ttl=3600.0, methods=("GET",)):

  • directory: where entries are written. Each entry is one .entry file named by a SHA-256 of the method, URL and params. The file holds a JSON metadata line (status, URL, headers, save time, ETag, Last-Modified) followed by the decoded body.
  • ttl: seconds an entry counts as fresh. 0 or None means entries never expire.
  • methods: methods the cache accepts. The session only consults the cache for GET.

Responses with Response.ok set are stored, which in 2.0 means status 200 to 399. With redirects followed (the default) the response you get back is normally 2xx; with allow_redirects=False a 3xx answer is stored too. Requests with no_cache=True, stream=True and downloads bypass the cache. cache.clear() deletes every entry (and any .json and .body files left by the 1.x layout) and returns how many files it removed.

Atomic writes

An entry is written to a temporary file in the cache directory, flushed with fsync, then moved into place with os.replace. Because metadata and body live in one file and the move is atomic, a reader running at the same time sees either the old entry or the new one, never fresh metadata next to a truncated body. The 1.x layout, with separate .json and .body files, could produce such torn entries. Old .json/.body entries are not read by 2.0.

Revalidation

When an entry is older than ttl, the session does not throw it away. If the stored response had an ETag or Last-Modified header, the next request sends If-None-Match or If-Modified-Since. A 304 Not Modified answer refreshes the entry’s age and returns the stored response with from_cache=True. Any other answer is returned normally and, if ok, replaces the entry.

Caching in a crawl

reap.run and Reaper accept cache= and pass it to the session they create:

items = reap.run(MySpider, cache=reap.DiskCache(ttl=86400))

Streaming downloads

download streams a response body straight to a file. Memory use stays constant whatever the size, over HTTP/1.1 and over HTTP/2:

r = reap.download("https://example.com/big.zip", "big.zip")
print(r.meta["saved_to"], r.meta["bytes"])

with reap.Session() as s:
    s.download("https://example.com/data.csv.gz", "data.csv.gz", timeout=120)

Session.download(url, path, **kw) sends a GET and accepts the same keywords as any request. reap.download uses the shared session and AsyncSession.download does the same work on a worker thread.

  • The body is written to <path>.reap-part and moved to <path> with os.replace only when complete. A failed download removes the part file; a retry starts the file again.

  • The file is written only for a 2xx response. Redirect bodies and error bodies (a 404 or 503 page, for example) never touch it, and an existing file at path is left as it was. Check r.status or r.is_success.

  • When bytes were written, meta["saved_to"] is the path and meta["bytes"] is the number of bytes written. Response.content is empty; the data is in the file.

  • A gzip, deflate or Brotli Content-Encoding is decoded as the bytes arrive. A zstd-encoded body cannot be streamed and raises ContentDecodingError. Many built-in targets (41 of 79, including chrome, Chrome 123 and newer, and every Firefox target) send Accept-Encoding with zstd, so for downloads from servers that may answer with zstd, send your own Accept-Encoding:

    s.download(url, "out.bin", headers={"Accept-Encoding": "gzip, deflate, br"})
  • Over HTTP/1.1, a response without a body (such as 204) does not create the file. Over HTTP/2, a 2xx response with an empty body produces an empty file.

  • Redirects and retries apply as for any request. Downloads always bypass the cache.

Over HTTP/2

Over HTTP/2 the received bytes go to a spooled temporary file (held in memory up to 8 MiB, on disk beyond that) and are then decoded into the part file, so memory stays bounded and the 64 MiB cap on buffered HTTP/2 bodies does not apply. In 1.1.0, HTTP/2 downloads were buffered in memory and failed above 64 MiB. See HTTP/2.

iter_content, iter_lines and stream=True

Response.iter_content(chunk_size=65536) yields the body in chunks and Response.iter_lines(chunk_size=65536, delimiter=None) yields lines without their line endings (\n or \r\n, or split on delimiter when given). On a normal response they replay the buffered body:

r = s.get("https://example.com/data.csv")
for line in r.iter_lines():
    print(line.decode())

With stream=True, iter_content and iter_lines read the body as it arrives instead of replaying a buffer. This works over HTTP/1.1; over HTTP/2 the body is still buffered (up to 64 MiB) and replayed. Streamed bodies decode gzip, deflate and br on the fly; other encodings, including zstd, raise ContentDecodingError. Response.close() abandons the rest of a streamed body and releases its connection; on a buffered response it does nothing. Details are in Streaming responses. To write a body to disk, download() is simpler and works the same over both protocols.