curl_reap / Get started
Quickstart
Fetch a page with a browser profile, select data with CSS, extract structured data, and run a small concurrent crawl with curl_reap.
On this page
This page assumes curl_reap is installed. See Installation.
First principles
Three objects carry almost everything:
- A
Responseis both the result and the document. It holds the status, headers and body, and it parses the body on first use, so you fetch and select on the same object. - A
Sessionis where state lives: pooled connections, cookies and defaults.reap.getuses a shared one; create your own as soon as you make more than a few requests. - A spider yields two kinds of things: items, which are your data, and requests, which are more pages to fetch. The engine schedules the requests and passes the items on.
Fetch and parse
import curl_reap as reap
page = reap.get("https://quotes.toscrape.com", profile="chrome")
print(page.status, page.ok, page.url)
print(page.css("span.text::text").getall())
print(page.css_first("small.author::text"))
reap.get sends the request through a shared keep-alive session. profile="chrome" selects the Chrome Transport Layer Security (TLS) and header profile, which is also the default. You can pass a versioned target instead:
page = reap.get("https://quotes.toscrape.com", impersonate="chrome124")
In a Cascading Style Sheets (CSS) query, ::text returns text content and ::attr(name) returns an attribute value. Without a pseudo element, css() returns element selectors you can query further:
for quote in page.css("div.quote"):
text = quote.css_first("span.text::text")
tags = quote.css("a.tag::text").getall()
print(text, tags)
Extract structured data
The common answers are one call away, no selectors needed:
page = reap.get("https://example.com/product/42")
page.jsonld() # every JSON-LD block as a dict
page.meta_tags() # title, description, og:*, twitter:*, canonical
page.links(internal_only=True) # [{"url": ..., "text": ...}] on the same origin
page.images() # [{"url": ..., "alt": ...}], handles data-src
page.tables() # every <table> as a list of rows
page.markdown() # readable page text as markdown
Use a session
A Session keeps cookies and pooled connections and lets you set defaults once:
from curl_reap import Session
with Session(profile="safari", headers={"Accept-Language": "en"}, retries=3) as s:
r = s.get("https://postman-echo.com/headers")
print(r.status, r.json())
See Sessions, retries, proxies and cookies.
Crawl a site
A spider yields items (dicts) and further requests. reap.run runs it with a pool of worker threads and returns the items:
import curl_reap as reap
class Quotes(reap.Spider):
start_urls = ["https://quotes.toscrape.com"]
allowed_domains = ["quotes.toscrape.com"]
def parse(self, page):
for q in page.css("div.quote"):
yield {"text": q.css_first("span.text::text"),
"author": q.css_first("small.author::text")}
nxt = page.css_first("li.next a::attr(href)")
if nxt:
yield page.follow(nxt) # resolves the relative URL
items = reap.run(Quotes, concurrency=8, throttle=True)
print(len(items), "items reaped")
To stream items to a file as they arrive, add a pipeline:
from curl_reap import JsonLinesPipeline
reap.run(Quotes, pipelines=[JsonLinesPipeline("quotes.jsonl")])
From the shell
reap get https://example.com
reap get https://example.com --css "h1::text"
reap crawl https://quotes.toscrape.com --css "span.text::text" --max-pages 20 -o out.jsonl
See Command line.
Check what went on the wire
Each response says which protocol and TLS backend carried it, and whether the body looks like a block:
r = reap.get("https://example.com")
print(r.http_version, r.infos["tls_backend"], r.block_reason)
# for example: HTTP/2 native None
To compare your machine’s fingerprint with the captured browser, run reap doctor:
reap doctor chrome
See How it works for what these values mean, and Troubleshooting and limits when they are not what you expect.
Examples in the repository
The examples/ folder has two runnable scripts:
examples/quotes.pycrawls quotes.toscrape.com with aReaper, AutoThrottle and a JSON Lines pipeline, then prints the crawl stats.examples/booking.pyreads Booking.com’s public apartment landing pages for a few cities and geo-tags each property with the Geocoder, keeping building and district level hits. Its docstring notes that Booking’s search results page returns 202 with no data and that the example leaves it alone.