Skip to content
curl_reap

curl_reap / Reference

CLI reference

Every option of the reap command and its get, meta, links, crawl, doctor, serve and worker subcommands, with defaults and exit codes.

On this page
  1. First principles
  2. Global options
  3. reap get URL
  4. reap meta URL
  5. reap links URL
  6. reap crawl URL
  7. reap doctor [target]
  8. reap serve
  9. reap worker
  10. Exit codes

This page covers the reap command-line interface (CLI) in curl_reap 2.0.0. It is taken from build_parser() in curl_reap/cli.py.

First principles

The parser has two levels, and each option sits at the level its effect belongs to. Options that decide how a request looks on the wire (profile, impersonation target, rotation) build the Session, so they belong to reap itself and come before the subcommand. Options that decide what to do with the response (a selector, an output file, a page budget) belong to the subcommand. Every subcommand returns an exit code, so scripts can act on the outcome without parsing output.

Synopsis:

reap [-h] [--http2] [--impersonate IMPERSONATE]
     [--profile {chrome,firefox,safari}] [--rotate {random,sequence}]
     {get,meta,links,crawl,serve,worker,doctor} ...

The command is installed as a console script pointing at curl_reap.cli:main. python -m curl_reap.cli runs the same parser. reap -h and reap <subcommand> -h print help. Usage examples are in the Command line guide.

Global options

Place these before the subcommand.

Option Default Meaning
--profile {chrome,firefox,safari} chrome TLS and header profile. Used by get, meta, links and crawl.
--impersonate TARGET none Any name in reap.FINGERPRINTS, such as chrome124, firefox133 or safari17_0. Overrides --profile. Used by get, meta, links and, as the default target, doctor. See Impersonation targets.
--rotate {random,sequence} none Rotate across the chrome, firefox and safari profiles. When set, --profile and --impersonate do not affect the request. Used by get, meta, links and crawl.
--http2 off Accepted for compatibility and ignored. HTTP/2 is negotiated by default since 2.0.0.
-h, --help Show help.

doctor uses --impersonate only. serve and worker do not use any global option.

Changed since 1.1.0: --http2 no longer has an effect (in 1.1.0 it turned on HTTP/2 through the h2 extra), and the serve, worker and doctor subcommands are new. No option was removed.

reap get URL

Fetch one page and print it.

Option Default Meaning
--css QUERY Print each match of a CSS selector, one per line.
--xpath QUERY Print each match of an XPath expression, one per line.
--json off Parse the body as JSON and pretty-print it.
--html off Print the raw HTML.
--markdown off Print readable markdown. This is already the default, so the flag has no additional effect.
--max-len N no limit Truncate the markdown output to N characters.

Precedence when several are given: --css, then --xpath, then --json, then --html, then markdown. A status outside 200 to 399 also prints [reap] HTTP <status> for <url> to standard error.

reap meta URL

Print a JSON object with two keys: meta (the meta_tags() dict) and jsonld (the jsonld() list). No options.

Print one line per link: the absolute URL, a tab, and the link text.

Option Default Meaning
--internal off Only links with the page’s origin (scheme, host and port).

reap crawl URL

Crawl starting from URL.

Option Default Meaning
--css QUERY CSS selector applied to each page; one item per match.
--follow off Follow same-origin links found on each page.
--allowed [DOMAIN ...] no restriction Allowed domains.
--concurrency N 8 Worker threads.
--max-pages N 50 Page budget.
--max-depth N 3 Maximum link depth.
--robots off Respect robots.txt.
-o PATH, --out PATH standard output Write items to .jsonl, .csv or .db (SQLite), chosen by extension.

reap doctor [target]

Fetch a fingerprint echo service as target and compare the observed fingerprints with the values captured for it.

Argument or option Default Meaning
target --impersonate, else chrome Any of the 79 names. Aliases and legacy names are compared with the capture they render.
--url URL https://tls.peet.ws/api/all The fingerprint echo service.
--list off Print every name, one per line (aliases as name -> target), and exit.

Compared rows: ja4, ja4_r, ja3 (only for targets whose JA3 is stable), akamai (the Akamai HTTP/2 fingerprint) and headers (the request header order). A row that cannot be compared prints n/a and does not fail the check. peetprint is printed but not compared. The Command line guide shows the output.

reap serve

Run the private job API with Uvicorn. Needs the server extra (pip install "curl-reap[server]").

Option Default Meaning
--host HOST 127.0.0.1 Address to listen on.
--port PORT 8000 Port to listen on.

Environment: REAP_API_KEY (required), REAP_ALLOWED_ORIGINS (required, comma-separated scheme://host:port list) and REAP_REDIS_URL (default redis://127.0.0.1:6379/0).

reap worker

Run one job worker that claims jobs from Redis and executes them until stopped. No options. Needs the server extra and the same environment as serve, plus the optional REAP_PROXY_URL for the worker’s requests.

Exit codes

Code Meaning
0 get, meta, links: final status 200 to 399. crawl: the crawl finished. doctor: every comparable row matched, or --list.
1 get, meta, links: any other status. doctor: at least one row differs. Also any uncaught error, such as a connection failure or an unknown --impersonate name, which prints a traceback.
2 Invalid arguments (reported by argparse). doctor: an unknown target, or the echo service could not be reached. serve, worker: a missing variable, a missing server extra, or an unusable REAP_REDIS_URL.
130 Interrupted with Ctrl-C.