curl_reap / Reference
CLI reference
Every option of the reap command and its get, meta, links, crawl, doctor, serve and worker subcommands, with defaults and exit codes.
On this page
This page covers the reap command-line interface (CLI) in curl_reap 2.0.0. It is taken from build_parser() in curl_reap/cli.py.
First principles
The parser has two levels, and each option sits at the level its effect belongs to. Options that decide how a request looks on the wire (profile, impersonation target, rotation) build the Session, so they belong to reap itself and come before the subcommand. Options that decide what to do with the response (a selector, an output file, a page budget) belong to the subcommand. Every subcommand returns an exit code, so scripts can act on the outcome without parsing output.
Synopsis:
reap [-h] [--http2] [--impersonate IMPERSONATE]
[--profile {chrome,firefox,safari}] [--rotate {random,sequence}]
{get,meta,links,crawl,serve,worker,doctor} ...
The command is installed as a console script pointing at curl_reap.cli:main. python -m curl_reap.cli runs the same parser. reap -h and reap <subcommand> -h print help. Usage examples are in the Command line guide.
Global options
Place these before the subcommand.
| Option | Default | Meaning |
|---|---|---|
--profile {chrome,firefox,safari} |
chrome |
TLS and header profile. Used by get, meta, links and crawl. |
--impersonate TARGET |
none | Any name in reap.FINGERPRINTS, such as chrome124, firefox133 or safari17_0. Overrides --profile. Used by get, meta, links and, as the default target, doctor. See Impersonation targets. |
--rotate {random,sequence} |
none | Rotate across the chrome, firefox and safari profiles. When set, --profile and --impersonate do not affect the request. Used by get, meta, links and crawl. |
--http2 |
off | Accepted for compatibility and ignored. HTTP/2 is negotiated by default since 2.0.0. |
-h, --help |
Show help. |
doctor uses --impersonate only. serve and worker do not use any global option.
Changed since 1.1.0: --http2 no longer has an effect (in 1.1.0 it turned on HTTP/2 through the h2 extra), and the serve, worker and doctor subcommands are new. No option was removed.
reap get URL
Fetch one page and print it.
| Option | Default | Meaning |
|---|---|---|
--css QUERY |
Print each match of a CSS selector, one per line. | |
--xpath QUERY |
Print each match of an XPath expression, one per line. | |
--json |
off | Parse the body as JSON and pretty-print it. |
--html |
off | Print the raw HTML. |
--markdown |
off | Print readable markdown. This is already the default, so the flag has no additional effect. |
--max-len N |
no limit | Truncate the markdown output to N characters. |
Precedence when several are given: --css, then --xpath, then --json, then --html, then markdown. A status outside 200 to 399 also prints [reap] HTTP <status> for <url> to standard error.
reap meta URL
Print a JSON object with two keys: meta (the meta_tags() dict) and jsonld (the jsonld() list). No options.
reap links URL
Print one line per link: the absolute URL, a tab, and the link text.
| Option | Default | Meaning |
|---|---|---|
--internal |
off | Only links with the page’s origin (scheme, host and port). |
reap crawl URL
Crawl starting from URL.
| Option | Default | Meaning |
|---|---|---|
--css QUERY |
CSS selector applied to each page; one item per match. | |
--follow |
off | Follow same-origin links found on each page. |
--allowed [DOMAIN ...] |
no restriction | Allowed domains. |
--concurrency N |
8 |
Worker threads. |
--max-pages N |
50 |
Page budget. |
--max-depth N |
3 |
Maximum link depth. |
--robots |
off | Respect robots.txt. |
-o PATH, --out PATH |
standard output | Write items to .jsonl, .csv or .db (SQLite), chosen by extension. |
reap doctor [target]
Fetch a fingerprint echo service as target and compare the observed fingerprints with the values captured for it.
| Argument or option | Default | Meaning |
|---|---|---|
target |
--impersonate, else chrome |
Any of the 79 names. Aliases and legacy names are compared with the capture they render. |
--url URL |
https://tls.peet.ws/api/all |
The fingerprint echo service. |
--list |
off | Print every name, one per line (aliases as name -> target), and exit. |
Compared rows: ja4, ja4_r, ja3 (only for targets whose JA3 is stable), akamai (the Akamai HTTP/2 fingerprint) and headers (the request header order). A row that cannot be compared prints n/a and does not fail the check. peetprint is printed but not compared. The Command line guide shows the output.
reap serve
Run the private job API with Uvicorn. Needs the server extra (pip install "curl-reap[server]").
| Option | Default | Meaning |
|---|---|---|
--host HOST |
127.0.0.1 |
Address to listen on. |
--port PORT |
8000 |
Port to listen on. |
Environment: REAP_API_KEY (required), REAP_ALLOWED_ORIGINS (required, comma-separated scheme://host:port list) and REAP_REDIS_URL (default redis://127.0.0.1:6379/0).
reap worker
Run one job worker that claims jobs from Redis and executes them until stopped. No options. Needs the server extra and the same environment as serve, plus the optional REAP_PROXY_URL for the worker’s requests.
Exit codes
| Code | Meaning |
|---|---|
0 |
get, meta, links: final status 200 to 399. crawl: the crawl finished. doctor: every comparable row matched, or --list. |
1 |
get, meta, links: any other status. doctor: at least one row differs. Also any uncaught error, such as a connection failure or an unknown --impersonate name, which prints a traceback. |
2 |
Invalid arguments (reported by argparse). doctor: an unknown target, or the echo service could not be reached. serve, worker: a missing variable, a missing server extra, or an unusable REAP_REDIS_URL. |
130 |
Interrupted with Ctrl-C. |