Private job service

The optional server extra turns curl_reap into a small private service. reap serve runs a web application programming interface (API) built on FastAPI that accepts single-page extraction jobs, and reap worker runs processes that fetch the pages and store the results. Jobs wait in a Redis queue between the two. The code is in curl_reap/service.py and curl_reap/jobs.py.

The service handles one page per job. Large multi-page crawls stay in the library, through Reaper. There is no crawl endpoint.

First principles

• Whoever can submit a Uniform Resource Locator (URL) can make your worker fetch it. A shared fetch service is a door into the network it runs on. So jobs may only target origins on an exact allowlist, each redirect hop is checked again, and clients cannot supply proxies, headers, credentials or scripts.
• Workers crash. A job a worker claimed must come back if that worker dies, so a claim is a lease that expires unless the worker renews it. The price is that a job can run more than once: delivery is at-least-once.
• Politeness has to be shared. Ten workers hitting one site must space their requests as if they were one. The spacing and any server-requested cooldown live in Redis, not in each worker.
• Errors must not leak secrets. A proxy URL or a Redis password can end up in an exception message. Failed jobs report a reason, a status and a URL, never a raw exception.

Install and configure

pip install "curl_reap[server] @ git+https://github.com/anishfyi/curl_reap@main"

The extra adds FastAPI, Uvicorn and redis-py. Both commands read their settings from environment variables:

Variable	Used by	Meaning
REAP_API_KEY	serve, worker	Required. The shared secret clients send in the x-reap-key header.
REAP_ALLOWED_ORIGINS	serve, worker	Required. Comma-separated origins jobs may fetch, as scheme://host:port.
REAP_REDIS_URL	serve, worker	Default redis://127.0.0.1:6379/0. Never printed.
REAP_PROXY_URL	worker	Optional. One proxy for every worker request.

An origin is http or https, a host and a port. A missing port is filled in (443 or 80). An origin with a path, query, fragment or credentials is rejected at start-up.

Keep secrets out of shell history and configuration files. The project README stores them with smbk and injects them at run time; any secret manager or orchestrator that sets environment variables works the same way:

smbk add REAP_API_KEY
smbk add REAP_REDIS_URL      # only when the URL carries a password
export REAP_ALLOWED_ORIGINS="https://example.com:443"

smbk run -k REAP_API_KEY -- reap serve --host 127.0.0.1 --port 8000
smbk run -k REAP_API_KEY -k REAP_REDIS_URL -- reap worker

serve and worker exit with status 2 when a required variable is missing, the extra is not installed, an origin is invalid, or the Redis client cannot be created.

Submit a job and read the result

curl -sS -X POST http://127.0.0.1:8000/jobs \
    -H "x-reap-key: $REAP_API_KEY" -H 'content-type: application/json' \
    -d '{"url": "https://example.com/", "css": {"heading": "h1::text"}}'
# {"id": "3f2a...", "status": "queued"}

curl -sS http://127.0.0.1:8000/jobs/3f2a... -H "x-reap-key: $REAP_API_KEY"

A finished job has this shape:

{"id": "3f2a...", "status": "succeeded",
 "result": {"url": "https://example.com:443/", "status": 200,
            "fields": {"heading": "..."},
            "markdown": "...", "meta": {"title": "..."}, "jsonld": []}}

The job’s URL is stored in canonical form, with the port written out, so the result URL shows it too.

The result holds the final URL, its status, the css fields (the first match for each name, as extract() returns it), the main content as markdown (the <main> or <article> element, or the whole page when there is none, cut to max_chars), the meta tags and the JavaScript Object Notation for Linked Data (JSON-LD) blocks.

API reference

Every route except /healthz needs the x-reap-key header; a wrong or missing key gets 401. The interactive API docs pages are turned off.

• GET /healthz: {"status": "ok"}, or 503 with {"status": "unavailable"} when Redis does not answer.
• POST /jobs: queue a job and return 202 with {"id": ..., "status": "queued"}.
• GET /jobs/{id}: {"id", "status"}, plus result when the status is succeeded or error when it is failed. 404 for an unknown or expired id.

The POST /jobs body is a JavaScript Object Notation (JSON) object with these fields, and no others:

• url (required): an http or https URL without credentials, backslashes or control characters, whose origin is on the allowlist.
• css (optional): up to 64 names, each at most 128 characters, mapped to Cascading Style Sheets (CSS) queries of at most 1024 characters.
• max_chars (optional): an integer from 1 to 100000. Default 20000.

Rejections: 413 for a body over 64 kibibytes (KiB), 422 for an invalid body or field (including an origin not on the allowlist), 429 when the queue is full, and 503 when Redis is unavailable. The queue holds at most 1000 jobs, queued and running together.

A job’s status moves from queued to running to succeeded or failed.

What a worker does

Each worker process runs one job at a time:

1. It claims the next job with a 60-second lease and renews the lease every 20 seconds while it works.
2. It fetches through a Session(sticky=True, retries=0, timeout=30) with no proxy rotation, confined to the allowlist by url_policy, through REAP_PROXY_URL when set.
3. Before each request it checks robots.txt for that origin (cached for 5 minutes) and reserves a slot in the origin’s shared schedule: at least 1 second apart, or the robots.txt Crawl-delay when that is longer.
4. It follows redirects itself, at most 10, and checks every hop against the allowlist, robots.txt and the schedule before sending it.
5. It extracts the result and finishes the job. A result larger than 1 mebibyte (MiB) as JSON fails the job with result_too_large.

robots.txt fails closed. A 404 or 410 means everything is allowed and a 2xx is parsed; anything else (a redirect, 403, 429, 5xx or a challenge page) fails the job, and the origin’s robots.txt is fetched again after the 5-minute cache expires.

A 429 or 503 with Retry-After sets a cooldown for the origin that every worker sharing the Redis respects.

To scale, run more reap worker processes with the same environment against the same Redis.

Delivery and failures

• At-least-once. When a worker dies, its lease expires and the next claim puts the job back in the queue. After 3 deliveries the job fails with lease_expired. A worker whose lease lapsed cannot overwrite the job: its result is discarded.
• Queued jobs never expire. Finished and failed jobs expire 24 hours after they end. Configure Redis for durability: append-only file persistence (appendonly yes, appendfsync everysec) and the noeviction policy.
• Failure reasons. A failed job’s error is an object with a reason, and a status and url where they apply:

reason	Meaning
blocked	Status 400 or higher, or a challenge page. Includes block_reason.
robots	robots.txt could not be read; the job fails closed.
robots_disallow	robots.txt disallows the URL.
disallowed_origin	The URL or a redirect hop left the allowlist.
too_many_redirects	More than 10 redirects.
fetch_error	The request raised (kind is the exception class name), a redirect had no Location, or the status was not usable.
extraction_error	Extraction raised. Includes kind.
lease_lost	The worker lost its lease while waiting.
result_too_large	The result was over 1 MiB.
result_unserializable	The result could not be written as JSON.

A job that ran out of deliveries has the plain string lease_expired as its error.

Run it in a container

The repository’s Dockerfile builds a python:3.12-slim image with the server extra, running as a non-root user. The entry point is reap, and the default command is serve --host 0.0.0.0 --port 8000:

docker build -t curl-reap .
docker run --rm -p 8000:8000 \
    -e REAP_API_KEY -e REAP_ALLOWED_ORIGINS -e REAP_REDIS_URL \
    curl-reap serve --host 0.0.0.0 --port 8000
docker run --rm -e REAP_API_KEY -e REAP_ALLOWED_ORIGINS -e REAP_REDIS_URL \
    curl-reap worker

Inside a container, 127.0.0.1 is the container itself, so set REAP_REDIS_URL to an address the container can reach. Secrets come from the environment; nothing is baked into the image.

Deployment limits

• Keep the API on a private network behind a Transport Layer Security (TLS) ingress. The x-reap-key shared secret is not an identity system.
• The allowlist is checked per URL, but it is not an egress firewall: an allowed host name can still resolve to another address, which is how Domain Name System (DNS) rebinding works. Add egress rules at the network layer.
• Response bodies are buffered in memory, up to the session’s 64 MiB default. Set container or process resource limits.
• There is no provider-specific proxy integration. REAP_PROXY_URL takes one generic proxy URL.