Source: https://velofy.co/curl_reap/job-service/

[curl\_reap](https://velofy.co/curl_reap/) / Guides

# Private job service

Run curl\_reap as a private single-page extraction service: reap serve, reap worker, the Redis job queue, API rules, failure reasons and deployment limits.

The optional `server` extra turns curl\_reap into a small private service. `reap serve` runs a web application programming interface (API) built on FastAPI that accepts single-page extraction jobs, and `reap worker` runs processes that fetch the pages and store the results. Jobs wait in a Redis queue between the two. The code is in `curl_reap/service.py` and `curl_reap/jobs.py`.

The service handles one page per job. Large multi-page crawls stay in the library, through [`Reaper`](https://velofy.co/curl_reap/crawling/). There is no crawl endpoint.

## First principles

*   **Whoever can submit a Uniform Resource Locator (URL) can make your worker fetch it.** A shared fetch service is a door into the network it runs on. So jobs may only target origins on an exact allowlist, each redirect hop is checked again, and clients cannot supply proxies, headers, credentials or scripts.
*   **Workers crash.** A job a worker claimed must come back if that worker dies, so a claim is a lease that expires unless the worker renews it. The price is that a job can run more than once: delivery is at-least-once.
*   **Politeness has to be shared.** Ten workers hitting one site must space their requests as if they were one. The spacing and any server-requested cooldown live in Redis, not in each worker.
*   **Errors must not leak secrets.** A proxy URL or a Redis password can end up in an exception message. Failed jobs report a reason, a status and a URL, never a raw exception.

## Install and configure

```
pip install "curl_reap[server] @ git+https://github.com/anishfyi/curl_reap@main"
```

The extra adds FastAPI, Uvicorn and redis-py. Both commands read their settings from environment variables:

| Variable | Used by | Meaning |
| --- | --- | --- |
| `REAP_API_KEY` | serve, worker | Required. The shared secret clients send in the `x-reap-key` header. |
| `REAP_ALLOWED_ORIGINS` | serve, worker | Required. Comma-separated origins jobs may fetch, as `scheme://host:port`. |
| `REAP_REDIS_URL` | serve, worker | Default `redis://127.0.0.1:6379/0`. Never printed. |
| `REAP_PROXY_URL` | worker | Optional. One proxy for every worker request. |

An origin is `http` or `https`, a host and a port. A missing port is filled in (443 or 80). An origin with a path, query, fragment or credentials is rejected at start-up.

Keep secrets out of shell history and configuration files. The project README stores them with [smbk](https://github.com/anishfyi/smbk) and injects them at run time; any secret manager or orchestrator that sets environment variables works the same way:

```
smbk add REAP_API_KEY
smbk add REAP_REDIS_URL      # only when the URL carries a password
export REAP_ALLOWED_ORIGINS="https://example.com:443"

smbk run -k REAP_API_KEY -- reap serve --host 127.0.0.1 --port 8000
smbk run -k REAP_API_KEY -k REAP_REDIS_URL -- reap worker
```

`serve` and `worker` exit with status 2 when a required variable is missing, the extra is not installed, an origin is invalid, or the Redis client cannot be created.

## Submit a job and read the result

```
curl -sS -X POST http://127.0.0.1:8000/jobs \
    -H "x-reap-key: $REAP_API_KEY" -H 'content-type: application/json' \
    -d '{"url": "https://example.com/", "css": {"heading": "h1::text"}}'
# {"id": "3f2a...", "status": "queued"}

curl -sS http://127.0.0.1:8000/jobs/3f2a... -H "x-reap-key: $REAP_API_KEY"
```

A finished job has this shape:

```
{"id": "3f2a...", "status": "succeeded",
 "result": {"url": "https://example.com:443/", "status": 200,
            "fields": {"heading": "..."},
            "markdown": "...", "meta": {"title": "..."}, "jsonld": []}}
```

The job’s URL is stored in canonical form, with the port written out, so the result URL shows it too.

The `result` holds the final URL, its status, the `css` fields (the first match for each name, as [`extract()`](https://velofy.co/curl_reap/structured-extraction/#extract) returns it), the main content as markdown (the `<main>` or `<article>` element, or the whole page when there is none, cut to `max_chars`), the meta tags and the JavaScript Object Notation for Linked Data (JSON-LD) blocks.

## API reference

Every route except `/healthz` needs the `x-reap-key` header; a wrong or missing key gets 401. The interactive API docs pages are turned off.

*   `GET /healthz`: `{"status": "ok"}`, or 503 with `{"status": "unavailable"}` when Redis does not answer.
*   `POST /jobs`: queue a job and return 202 with `{"id": ..., "status": "queued"}`.
*   `GET /jobs/{id}`: `{"id", "status"}`, plus `result` when the status is `succeeded` or `error` when it is `failed`. 404 for an unknown or expired id.

The `POST /jobs` body is a JavaScript Object Notation (JSON) object with these fields, and no others:

*   `url` (required): an `http` or `https` URL without credentials, backslashes or control characters, whose origin is on the allowlist.
*   `css` (optional): up to 64 names, each at most 128 characters, mapped to Cascading Style Sheets (CSS) queries of at most 1024 characters.
*   `max_chars` (optional): an integer from 1 to 100000. Default 20000.

Rejections: 413 for a body over 64 kibibytes (KiB), 422 for an invalid body or field (including an origin not on the allowlist), 429 when the queue is full, and 503 when Redis is unavailable. The queue holds at most 1000 jobs, queued and running together.

A job’s status moves from `queued` to `running` to `succeeded` or `failed`.

## What a worker does

Each worker process runs one job at a time:

1.  It claims the next job with a 60-second lease and renews the lease every 20 seconds while it works.
2.  It fetches through a `Session(sticky=True, retries=0, timeout=30)` with no proxy rotation, confined to the allowlist by `url_policy`, through `REAP_PROXY_URL` when set.
3.  Before each request it checks robots.txt for that origin (cached for 5 minutes) and reserves a slot in the origin’s shared schedule: at least 1 second apart, or the robots.txt `Crawl-delay` when that is longer.
4.  It follows redirects itself, at most 10, and checks every hop against the allowlist, robots.txt and the schedule before sending it.
5.  It extracts the result and finishes the job. A result larger than 1 mebibyte (MiB) as JSON fails the job with `result_too_large`.

robots.txt fails closed. A 404 or 410 means everything is allowed and a 2xx is parsed; anything else (a redirect, 403, 429, 5xx or a challenge page) fails the job, and the origin’s robots.txt is fetched again after the 5-minute cache expires.

A 429 or 503 with `Retry-After` sets a cooldown for the origin that every worker sharing the Redis respects.

To scale, run more `reap worker` processes with the same environment against the same Redis.

## Delivery and failures

*   **At-least-once.** When a worker dies, its lease expires and the next claim puts the job back in the queue. After 3 deliveries the job fails with `lease_expired`. A worker whose lease lapsed cannot overwrite the job: its result is discarded.
*   **Queued jobs never expire.** Finished and failed jobs expire 24 hours after they end. Configure Redis for durability: append-only file persistence (`appendonly yes`, `appendfsync everysec`) and the `noeviction` policy.
*   **Failure reasons.** A failed job’s `error` is an object with a `reason`, and a `status` and `url` where they apply:

| `reason` | Meaning |
| --- | --- |
| `blocked` | Status 400 or higher, or a challenge page. Includes `block_reason`. |
| `robots` | robots.txt could not be read; the job fails closed. |
| `robots_disallow` | robots.txt disallows the URL. |
| `disallowed_origin` | The URL or a redirect hop left the allowlist. |
| `too_many_redirects` | More than 10 redirects. |
| `fetch_error` | The request raised (`kind` is the exception class name), a redirect had no `Location`, or the status was not usable. |
| `extraction_error` | Extraction raised. Includes `kind`. |
| `lease_lost` | The worker lost its lease while waiting. |
| `result_too_large` | The result was over 1 MiB. |
| `result_unserializable` | The result could not be written as JSON. |

A job that ran out of deliveries has the plain string `lease_expired` as its `error`.

## Run it in a container

The repository’s `Dockerfile` builds a `python:3.12-slim` image with the `server` extra, running as a non-root user. The entry point is `reap`, and the default command is `serve --host 0.0.0.0 --port 8000`:

```
docker build -t curl-reap .
docker run --rm -p 8000:8000 \
    -e REAP_API_KEY -e REAP_ALLOWED_ORIGINS -e REAP_REDIS_URL \
    curl-reap serve --host 0.0.0.0 --port 8000
docker run --rm -e REAP_API_KEY -e REAP_ALLOWED_ORIGINS -e REAP_REDIS_URL \
    curl-reap worker
```

Inside a container, `127.0.0.1` is the container itself, so set `REAP_REDIS_URL` to an address the container can reach. Secrets come from the environment; nothing is baked into the image.

## Deployment limits

*   Keep the API on a private network behind a Transport Layer Security (TLS) ingress. The `x-reap-key` shared secret is not an identity system.
*   The allowlist is checked per URL, but it is not an egress firewall: an allowed host name can still resolve to another address, which is how Domain Name System (DNS) rebinding works. Add egress rules at the network layer.
*   Response bodies are buffered in memory, up to the session’s 64 MiB default. Set container or process resource limits.
*   There is no provider-specific proxy integration. `REAP_PROXY_URL` takes one generic proxy URL.