Skip to content
curl_reap

curl_reap / Guides

Command line

Use the reap command to fetch pages, extract with CSS or XPath, crawl to JSONL, CSV or SQLite, check fingerprints with reap doctor, and more.

On this page
  1. First principles
  2. Global options go first
  3. Fetch one page
  4. Metadata and links
  5. Crawl
  6. Check your fingerprint: reap doctor
  7. Job service: reap serve and reap worker
  8. Exit status

Installing curl_reap also installs a reap command. It covers the common jobs without writing a script: fetch a page, pull values out of it, list its links, crawl a site, check your fingerprint, and run the private job service.

First principles

  • Each subcommand is one library call. reap get is Session.get plus a printer, reap crawl is reap.run with a generated spider, and reap doctor is curl_reap.fingerprint.check. Anything the command line cannot express, the library can.
  • Options that build the session come before the subcommand. The profile, impersonation target and rotation decide how the request looks on the wire, so they belong to reap itself, not to one subcommand.
  • The exit status reports the outcome. Scripts can branch on it without parsing the output: success, a bad response or fingerprint, or a usage or configuration error.

Global options go first

--profile, --impersonate, --rotate and --http2 belong to reap itself, so they must come before the subcommand:

reap --profile firefox get https://example.com
reap --impersonate chrome124 get https://example.com

Written after the subcommand (reap get URL --profile firefox), they are rejected as unrecognized arguments.

  • --impersonate takes any name in reap.FINGERPRINTS (79 in 2.0.0) and overrides --profile. See Impersonation targets. An unknown name stops the command with an UnknownProfileError traceback.
  • --rotate rotates requests across the chrome, firefox and safari profiles. When it is set, --impersonate and --profile have no effect on the request.
  • HTTP/2 is negotiated by default in 2.0.0. --http2 is still accepted so old scripts keep working, but it no longer changes anything. There is no CLI option to turn HTTP/2 off.

Fetch one page

reap get https://example.com                     # readable markdown (the default)
reap get https://example.com --max-len 2000      # markdown, truncated
reap get https://example.com --html              # raw HTML
reap get https://example.com --css "h1::text"    # one line per match
reap get https://example.com --xpath "//a/@href"
reap get https://api.example.com/items --json    # pretty-printed JSON

Only one output mode is used. If several are given, the first of --css, --xpath, --json, --html wins, and markdown is the fallback. When a CSS or XPath match is an element rather than a string, its text is printed.

reap meta https://example.com                # {"meta": {...}, "jsonld": [...]} as JSON
reap links https://example.com               # one "url<TAB>text" line per link
reap links https://example.com --internal    # same-origin links only

meta prints the output of meta_tags() and jsonld() described in Structured extraction.

Crawl

reap crawl https://quotes.toscrape.com --css "span.text::text" --max-pages 20 -o out.jsonl
reap crawl https://example.com --follow --max-depth 2 --robots -o pages.csv
reap crawl https://example.com --follow --allowed example.com docs.example.com -o site.db
  • With --css (a Cascading Style Sheets selector), each page yields one item per match: {"url": ..., "value": ...}.
  • Without --css, each page yields {"url": ..., "title": ...} from the page’s <title>.
  • --follow queues every link on each page with the page’s origin (scheme, host and port).
  • -o picks the output format by extension: .jsonl (JsonLinesPipeline), .csv (CsvPipeline) or .db (SqlitePipeline). Any other extension writes no file.
  • Without -o, items are printed as JSON lines on standard output. With -o, a count is printed to standard error.

The crawl uses the global --profile and --rotate options. --impersonate applies to get, meta, links and doctor, not to crawl.

Check your fingerprint: reap doctor

reap doctor makes one request to a fingerprint echo service as a target, then compares what the service saw with the values captured from the real browser. It is new in 2.0.0.

reap doctor                         # the --impersonate target, else "chrome"
reap doctor firefox147
reap --impersonate safari doctor    # same as: reap doctor safari
reap doctor --list                  # all 79 names; aliases show their target
reap doctor chrome100 --url https://my-echo.example/api/all

The default service is https://tls.peet.ws/api/all. --url points at another service that returns the same JSON shape. The request is made with a session that impersonates the target, so you measure that target’s real traffic. The global --profile and --rotate options do not affect doctor.

Every name in reap.FINGERPRINTS works. An alias or a legacy name is compared with the capture it renders. When the name has no captured values of its own (four aliases and all 22 legacy names), the first line says so: target edge_99 (captured as edge99).

The output lists what the service observed, then one row per comparison. A real run against the default service, with the long ja3 and peetprint values shortened:

target      chrome
url         https://tls.peet.ws/api/all
http        h2
ja3         771,4865-4866-4867-49195-...
ja4         t13d1516h2_8daaf6152771_806a8c22fdea
peetprint   GREASE-772-771|2-1.1|GREASE-4588-29-23-24|...
akamai h2   1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p

  ja4        MATCH
  ja4_r      MATCH
  ja3        n/a  (not compared: this browser permutes its extensions per connection)
  akamai     MATCH
  headers    MATCH

[reap] chrome: every comparable fingerprint matches the capture
Row Compared with the capture
ja4 JA4
ja4_r the raw (unhashed) JA4
ja3 JA3, only for targets whose JA3 is stable; n/a for browsers that permute their extensions on each connection
akamai the Akamai HTTP/2 fingerprint
headers the order of the regular request headers, against the target’s set for the protocol in use; n/a when the service did not echo the header block

peetprint is printed but not compared. A row marked n/a does not fail the check. When a row differs, the command prints [reap] <target> differs from the capture (...) and the expected values of the differing rows to standard error.

Exit status:

  • 0: every comparable row matches, or --list.
  • 1: at least one row differs.
  • 2: the target is unknown (the message starts with [reap] unknown fingerprint target), or the service could not be reached ([reap] doctor could not reach).

The same check is available from Python:

from curl_reap.fingerprint import check

report = check("firefox147")
print(report["capture"], report["matches"])
# for example: firefox147 {"ja4": True, "ja4_r": True, "akamai": True, "ja3": True, "headers": True}

check(target="chrome", url="https://tls.peet.ws/api/all", session=None) creates and closes a session that impersonates target when you pass none. The report also holds the observed ja3, ja4, peetprint and akamai values, expected, headers, expected_headers and http_version. A None in matches means not comparable.

Job service: reap serve and reap worker

reap serve runs a private Hypertext Transfer Protocol (HTTP) application programming interface (API) that queues single-page extraction jobs in Redis, and reap worker runs a process that executes them. Both need the server extra (pip install "curl-reap[server]", which adds FastAPI, Uvicorn and redis-py) and read their settings from environment variables:

Variable Used by Meaning
REAP_API_KEY serve, worker Required. Clients send it in the X-Reap-Key header.
REAP_ALLOWED_ORIGINS serve, worker Required. Comma-separated scheme://host:port origins that jobs may fetch.
REAP_REDIS_URL serve, worker Redis URL. Default redis://127.0.0.1:6379/0. It is never printed.
REAP_PROXY_URL worker Optional proxy for the worker’s requests.
export REAP_API_KEY=change-me
export REAP_ALLOWED_ORIGINS=https://example.com:443
reap serve --host 127.0.0.1 --port 8000
reap worker

The API has three routes. Every route except /healthz requires the X-Reap-Key header.

  • GET /healthz: {"status": "ok"}, or 503 when the queue is unavailable.
  • POST /jobs with a JSON body {"url": ..., "css": {name: query}, "max_chars": 20000}: returns 202 with {"id": ..., "status": "queued"}. Only url is required. The URL’s origin must be in REAP_ALLOWED_ORIGINS. Bodies are limited to 64 KB.
  • GET /jobs/{id}: the job status, with result when it succeeded (url, status, fields, markdown, meta, jsonld) or error when it failed.

The worker makes requests without retries or proxy rotation, respects robots.txt, and spaces requests to the same origin (at least one second apart by default). serve and worker exit with 2 when a required variable is missing, the extra is not installed, or the Redis client cannot be created from REAP_REDIS_URL. Delivery guarantees, error reasons, deployment limits and a Docker image are covered in Private job service.

Exit status

  • get, meta and links exit with 0 when the final response status is 200 to 399, and 1 otherwise. Redirects are followed, so a 3xx is usually not the final status. For other statuses, get also prints [reap] HTTP <status> for <url> to standard error.
  • A network or Transport Layer Security (TLS) error is not caught: Python prints a traceback and the exit status is 1.
  • crawl exits with 0 when the crawl finishes.
  • doctor exits with 0, 1 or 2 as described above. serve and worker exit with 2 on a configuration error.
  • Invalid arguments exit with 2. Ctrl-C exits with 130.

Every flag is listed in the CLI reference.