curl_reap / Guides
Command line
Use the reap command to fetch pages, extract with CSS or XPath, crawl to JSONL, CSV or SQLite, check fingerprints with reap doctor, and more.
On this page
Installing curl_reap also installs a reap command. It covers the common jobs without writing a script: fetch a page, pull values out of it, list its links, crawl a site, check your fingerprint, and run the private job service.
First principles
- Each subcommand is one library call.
reap getisSession.getplus a printer,reap crawlisreap.runwith a generated spider, andreap doctoriscurl_reap.fingerprint.check. Anything the command line cannot express, the library can. - Options that build the session come before the subcommand. The profile, impersonation target and rotation decide how the request looks on the wire, so they belong to
reapitself, not to one subcommand. - The exit status reports the outcome. Scripts can branch on it without parsing the output: success, a bad response or fingerprint, or a usage or configuration error.
Global options go first
--profile, --impersonate, --rotate and --http2 belong to reap itself, so they must come before the subcommand:
reap --profile firefox get https://example.com
reap --impersonate chrome124 get https://example.com
Written after the subcommand (reap get URL --profile firefox), they are rejected as unrecognized arguments.
--impersonatetakes any name inreap.FINGERPRINTS(79 in 2.0.0) and overrides--profile. See Impersonation targets. An unknown name stops the command with anUnknownProfileErrortraceback.--rotaterotates requests across thechrome,firefoxandsafariprofiles. When it is set,--impersonateand--profilehave no effect on the request.- HTTP/2 is negotiated by default in 2.0.0.
--http2is still accepted so old scripts keep working, but it no longer changes anything. There is no CLI option to turn HTTP/2 off.
Fetch one page
reap get https://example.com # readable markdown (the default)
reap get https://example.com --max-len 2000 # markdown, truncated
reap get https://example.com --html # raw HTML
reap get https://example.com --css "h1::text" # one line per match
reap get https://example.com --xpath "//a/@href"
reap get https://api.example.com/items --json # pretty-printed JSON
Only one output mode is used. If several are given, the first of --css, --xpath, --json, --html wins, and markdown is the fallback. When a CSS or XPath match is an element rather than a string, its text is printed.
Metadata and links
reap meta https://example.com # {"meta": {...}, "jsonld": [...]} as JSON
reap links https://example.com # one "url<TAB>text" line per link
reap links https://example.com --internal # same-origin links only
meta prints the output of meta_tags() and jsonld() described in Structured extraction.
Crawl
reap crawl https://quotes.toscrape.com --css "span.text::text" --max-pages 20 -o out.jsonl
reap crawl https://example.com --follow --max-depth 2 --robots -o pages.csv
reap crawl https://example.com --follow --allowed example.com docs.example.com -o site.db
- With
--css(a Cascading Style Sheets selector), each page yields one item per match:{"url": ..., "value": ...}. - Without
--css, each page yields{"url": ..., "title": ...}from the page’s<title>. --followqueues every link on each page with the page’s origin (scheme, host and port).-opicks the output format by extension:.jsonl(JsonLinesPipeline),.csv(CsvPipeline) or.db(SqlitePipeline). Any other extension writes no file.- Without
-o, items are printed as JSON lines on standard output. With-o, a count is printed to standard error.
The crawl uses the global --profile and --rotate options. --impersonate applies to get, meta, links and doctor, not to crawl.
Check your fingerprint: reap doctor
reap doctor makes one request to a fingerprint echo service as a target, then compares what the service saw with the values captured from the real browser. It is new in 2.0.0.
reap doctor # the --impersonate target, else "chrome"
reap doctor firefox147
reap --impersonate safari doctor # same as: reap doctor safari
reap doctor --list # all 79 names; aliases show their target
reap doctor chrome100 --url https://my-echo.example/api/all
The default service is https://tls.peet.ws/api/all. --url points at another service that returns the same JSON shape. The request is made with a session that impersonates the target, so you measure that target’s real traffic. The global --profile and --rotate options do not affect doctor.
Every name in reap.FINGERPRINTS works. An alias or a legacy name is compared with the capture it renders. When the name has no captured values of its own (four aliases and all 22 legacy names), the first line says so: target edge_99 (captured as edge99).
The output lists what the service observed, then one row per comparison. A real run against the default service, with the long ja3 and peetprint values shortened:
target chrome
url https://tls.peet.ws/api/all
http h2
ja3 771,4865-4866-4867-49195-...
ja4 t13d1516h2_8daaf6152771_806a8c22fdea
peetprint GREASE-772-771|2-1.1|GREASE-4588-29-23-24|...
akamai h2 1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p
ja4 MATCH
ja4_r MATCH
ja3 n/a (not compared: this browser permutes its extensions per connection)
akamai MATCH
headers MATCH
[reap] chrome: every comparable fingerprint matches the capture
| Row | Compared with the capture |
|---|---|
ja4 |
JA4 |
ja4_r |
the raw (unhashed) JA4 |
ja3 |
JA3, only for targets whose JA3 is stable; n/a for browsers that permute their extensions on each connection |
akamai |
the Akamai HTTP/2 fingerprint |
headers |
the order of the regular request headers, against the target’s set for the protocol in use; n/a when the service did not echo the header block |
peetprint is printed but not compared. A row marked n/a does not fail the check. When a row differs, the command prints [reap] <target> differs from the capture (...) and the expected values of the differing rows to standard error.
Exit status:
0: every comparable row matches, or--list.1: at least one row differs.2: the target is unknown (the message starts with[reap] unknown fingerprint target), or the service could not be reached ([reap] doctor could not reach).
The same check is available from Python:
from curl_reap.fingerprint import check
report = check("firefox147")
print(report["capture"], report["matches"])
# for example: firefox147 {"ja4": True, "ja4_r": True, "akamai": True, "ja3": True, "headers": True}
check(target="chrome", url="https://tls.peet.ws/api/all", session=None) creates and closes a session that impersonates target when you pass none. The report also holds the observed ja3, ja4, peetprint and akamai values, expected, headers, expected_headers and http_version. A None in matches means not comparable.
Job service: reap serve and reap worker
reap serve runs a private Hypertext Transfer Protocol (HTTP) application programming interface (API) that queues single-page extraction jobs in Redis, and reap worker runs a process that executes them. Both need the server extra (pip install "curl-reap[server]", which adds FastAPI, Uvicorn and redis-py) and read their settings from environment variables:
| Variable | Used by | Meaning |
|---|---|---|
REAP_API_KEY |
serve, worker | Required. Clients send it in the X-Reap-Key header. |
REAP_ALLOWED_ORIGINS |
serve, worker | Required. Comma-separated scheme://host:port origins that jobs may fetch. |
REAP_REDIS_URL |
serve, worker | Redis URL. Default redis://127.0.0.1:6379/0. It is never printed. |
REAP_PROXY_URL |
worker | Optional proxy for the worker’s requests. |
export REAP_API_KEY=change-me
export REAP_ALLOWED_ORIGINS=https://example.com:443
reap serve --host 127.0.0.1 --port 8000
reap worker
The API has three routes. Every route except /healthz requires the X-Reap-Key header.
GET /healthz:{"status": "ok"}, or 503 when the queue is unavailable.POST /jobswith a JSON body{"url": ..., "css": {name: query}, "max_chars": 20000}: returns 202 with{"id": ..., "status": "queued"}. Onlyurlis required. The URL’s origin must be inREAP_ALLOWED_ORIGINS. Bodies are limited to 64 KB.GET /jobs/{id}: the job status, withresultwhen it succeeded (url,status,fields,markdown,meta,jsonld) orerrorwhen it failed.
The worker makes requests without retries or proxy rotation, respects robots.txt, and spaces requests to the same origin (at least one second apart by default). serve and worker exit with 2 when a required variable is missing, the extra is not installed, or the Redis client cannot be created from REAP_REDIS_URL. Delivery guarantees, error reasons, deployment limits and a Docker image are covered in Private job service.
Exit status
get,metaandlinksexit with 0 when the final response status is 200 to 399, and 1 otherwise. Redirects are followed, so a 3xx is usually not the final status. For other statuses,getalso prints[reap] HTTP <status> for <url>to standard error.- A network or Transport Layer Security (TLS) error is not caught: Python prints a traceback and the exit status is 1.
crawlexits with 0 when the crawl finishes.doctorexits with 0, 1 or 2 as described above.serveandworkerexit with 2 on a configuration error.- Invalid arguments exit with 2. Ctrl-C exits with 130.
Every flag is listed in the CLI reference.