Skip to content
curl_reap

curl_reap / Get started

Quickstart

Fetch a page with a browser profile, select data with CSS, extract structured data, and run a small concurrent crawl with curl_reap.

On this page
  1. First principles
  2. Fetch and parse
  3. Extract structured data
  4. Use a session
  5. Crawl a site
  6. From the shell
  7. Check what went on the wire
  8. Examples in the repository

This page assumes curl_reap is installed. See Installation.

First principles

Three objects carry almost everything:

  • A Response is both the result and the document. It holds the status, headers and body, and it parses the body on first use, so you fetch and select on the same object.
  • A Session is where state lives: pooled connections, cookies and defaults. reap.get uses a shared one; create your own as soon as you make more than a few requests.
  • A spider yields two kinds of things: items, which are your data, and requests, which are more pages to fetch. The engine schedules the requests and passes the items on.

Fetch and parse

import curl_reap as reap

page = reap.get("https://quotes.toscrape.com", profile="chrome")
print(page.status, page.ok, page.url)
print(page.css("span.text::text").getall())
print(page.css_first("small.author::text"))

reap.get sends the request through a shared keep-alive session. profile="chrome" selects the Chrome Transport Layer Security (TLS) and header profile, which is also the default. You can pass a versioned target instead:

page = reap.get("https://quotes.toscrape.com", impersonate="chrome124")

In a Cascading Style Sheets (CSS) query, ::text returns text content and ::attr(name) returns an attribute value. Without a pseudo element, css() returns element selectors you can query further:

for quote in page.css("div.quote"):
    text = quote.css_first("span.text::text")
    tags = quote.css("a.tag::text").getall()
    print(text, tags)

Extract structured data

The common answers are one call away, no selectors needed:

page = reap.get("https://example.com/product/42")
page.jsonld()                    # every JSON-LD block as a dict
page.meta_tags()                 # title, description, og:*, twitter:*, canonical
page.links(internal_only=True)   # [{"url": ..., "text": ...}] on the same origin
page.images()                    # [{"url": ..., "alt": ...}], handles data-src
page.tables()                    # every <table> as a list of rows
page.markdown()                  # readable page text as markdown

See Structured extraction.

Use a session

A Session keeps cookies and pooled connections and lets you set defaults once:

from curl_reap import Session

with Session(profile="safari", headers={"Accept-Language": "en"}, retries=3) as s:
    r = s.get("https://postman-echo.com/headers")
    print(r.status, r.json())

See Sessions, retries, proxies and cookies.

Crawl a site

A spider yields items (dicts) and further requests. reap.run runs it with a pool of worker threads and returns the items:

import curl_reap as reap


class Quotes(reap.Spider):
    start_urls = ["https://quotes.toscrape.com"]
    allowed_domains = ["quotes.toscrape.com"]

    def parse(self, page):
        for q in page.css("div.quote"):
            yield {"text": q.css_first("span.text::text"),
                   "author": q.css_first("small.author::text")}
        nxt = page.css_first("li.next a::attr(href)")
        if nxt:
            yield page.follow(nxt)   # resolves the relative URL


items = reap.run(Quotes, concurrency=8, throttle=True)
print(len(items), "items reaped")

To stream items to a file as they arrive, add a pipeline:

from curl_reap import JsonLinesPipeline

reap.run(Quotes, pipelines=[JsonLinesPipeline("quotes.jsonl")])

See Crawling and Pipelines.

From the shell

reap get https://example.com
reap get https://example.com --css "h1::text"
reap crawl https://quotes.toscrape.com --css "span.text::text" --max-pages 20 -o out.jsonl

See Command line.

Check what went on the wire

Each response says which protocol and TLS backend carried it, and whether the body looks like a block:

r = reap.get("https://example.com")
print(r.http_version, r.infos["tls_backend"], r.block_reason)
# for example: HTTP/2 native None

To compare your machine’s fingerprint with the captured browser, run reap doctor:

reap doctor chrome

See How it works for what these values mean, and Troubleshooting and limits when they are not what you expect.

Examples in the repository

The examples/ folder has two runnable scripts:

  • examples/quotes.py crawls quotes.toscrape.com with a Reaper, AutoThrottle and a JSON Lines pipeline, then prints the crawl stats.
  • examples/booking.py reads Booking.com’s public apartment landing pages for a few cities and geo-tags each property with the Geocoder, keeping building and district level hits. Its docstring notes that Booking’s search results page returns 202 with no data and that the example leaves it alone.