Skip to content
curl_reap

curl_reap / Guides

Crawling

Write spiders, run them with reap.run or Reaper, limit depth and domains, respect robots.txt, crawl from sitemaps and tune AutoThrottle.

On this page
  1. First principles
  2. Spiders
  3. Requests
  4. Running a crawl
  5. Stats
  6. Logging
  7. robots.txt
  8. Crawling from sitemaps
  9. AutoThrottle

A crawl turns one fetch into many: a spider says where to start and how to read each page, and the engine schedules the requests, enforces limits and passes items to pipelines.

First principles

A crawl is made of four parts, and each one needs a limit:

  • A queue of pending requests (the frontier). Every page can add dozens of links, so the queue can grow faster than it drains. max_pending caps it, and max_depth and allowed_domains keep it on topic. The queue lives in memory, so a crash loses pending work.
  • A set of seen requests. Without it, a link that appears on every page is fetched on every page. Duplicates are dropped by fingerprint before they are queued.
  • A budget. max_pages is reserved before each request goes out, so concurrent workers cannot overshoot it.
  • A politeness clock per host. A site is someone else’s server. AutoThrottle spaces request starts per domain, per_domain_concurrency caps requests in flight per origin, robots.txt can be honored, and a Retry-After on a 429 or 503 response pauses that domain.

Spiders

A spider says where to start and how to parse each page. parse yields items (any non-None value, usually dicts) and further Request objects:

import curl_reap as reap


class Quotes(reap.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]
    allowed_domains = ["quotes.toscrape.com"]
    max_depth = 10

    def parse(self, page):
        for q in page.css("div.quote"):
            yield {"text": q.css_first("span.text::text"),
                   "author": q.css_first("small.author::text"),
                   "tags": q.css("a.tag::text").getall()}
        nxt = page.css_first("li.next a::attr(href)")
        if nxt:
            yield page.follow(nxt)


items = reap.run(Quotes, concurrency=8)

Spider class attributes:

  • name = "reap": a label.
  • start_urls = []: seed URLs, each parsed by parse.
  • allowed_domains = None: when set, requests to other hosts are dropped. Subdomains of an allowed domain are included.
  • max_depth = None: stop following links deeper than this. run(max_depth=...) overrides it.
  • custom_headers = None: headers added to every request of the crawl.

Override start(self) to yield your own seed Request objects, for example with custom callbacks, headers or meta.

Requests

Request(url, callback=None, method="GET", meta=None, priority=0, errback=None, dont_filter=False, **kw):

  • callback: the function that parses the response. Defaults to the spider’s parse.
  • meta: a dict carried onto Response.meta. The engine stores the link depth in meta["depth"].
  • priority: higher runs sooner. Equal priorities run in the order they were queued.
  • errback: called when the request does not produce a usable page. It receives the exception when the fetch fails after retries or the callback raises, and the Response when the status is 400 or higher or the body looks like a challenge page.
  • dont_filter=True: skip duplicate filtering for this request.
  • Other keywords (headers, data, json, params, proxy, profile, timeout, and so on) are passed to the session’s request. See Sessions, retries, proxies and cookies.

page.follow(target, callback=None, **kw) builds a Request for a URL relative to the page. target can be a string or a Selector, in which case its href (or its text) is used.

Duplicate requests are dropped by fingerprint: the method, the URL without its fragment, and a hash of data, json or params when present.

Responses with status 400 or higher, and responses whose body looks like a bot-wall challenge (Response.looks_like_challenge), never reach the callback. They go to the errback with the Response, and count in stats["errors"] and stats["blocked"]. Your parse only sees pages that loaded.

Running a crawl

reap.run(spider, **options) accepts a spider class or instance, runs it to completion and returns the list of items. reap.Reaper(spider_instance, **options) takes the same options and gives you the engine object, with .run(), .stats and .items:

reaper = reap.Reaper(Quotes(), concurrency=12, throttle=True,
                     on_item=lambda it: print("got", it["author"]))
items = reaper.run()
print(reaper.stats)

Options and defaults:

  • concurrency=8: worker threads.
  • retries=2: retries per request (see the retry policy in the sessions guide).
  • throttle=True: per-domain AutoThrottle, described below.
  • delay=0.0: minimum per-request delay per domain, used by AutoThrottle.
  • profile="chrome", rotate=None, proxy=None, cache=None: passed to the Session the engine creates.
  • session=None: use your own Session instead. retries, profile, rotate, proxy, cache and the spider’s custom_headers are then ignored.
  • pipelines=None: item pipelines. See Pipelines.
  • dedup=True: add a DedupPipeline() at the front unless one is already in pipelines.
  • on_item=None: a callable run for every item that passes the pipelines.
  • max_pages=None: a hard cap on pages. Each worker reserves one unit of the budget before it touches the network, so the crawl never fetches more than max_pages pages, whatever the concurrency. Failed fetches and URLs blocked by robots.txt use the budget too; the session’s retries of one page and the robots.txt fetches themselves do not. max_pages=0 fetches nothing.
  • max_depth=None: overrides the spider’s max_depth.
  • respect_robots=False: check robots.txt before each fetch.
  • collect_items=True: keep every item in Reaper.items and the returned list. False streams items through pipelines and on_item without storing them, for crawls too large to hold in memory. stats["items"] still counts them.
  • per_domain_concurrency=None: the most requests in flight to one origin (scheme, host and port) at a time. Defaults to concurrency.
  • max_pending=None: a bound on requests queued but not yet processed. When the queue is full, newly found requests are dropped and counted in stats["queue_dropped"]. A dropped request is not marked as seen, so a later link to it can still be queued.

Stats

Reaper.stats holds:

  • attempts: requests admitted against the budget.
  • requests: responses received.
  • statuses: a dict of response counts by status code.
  • items: items that passed every pipeline.
  • dropped: items a pipeline dropped.
  • errors: requests that ended in the errback (failed fetches, callback errors and blocked responses).
  • blocked: responses with status 400 or higher, or a challenge-looking body.
  • robots_blocked, offsite_dropped and queue_dropped: requests skipped by robots.txt, by allowed_domains, and by a full queue.
  • elapsed: seconds, after a crawl that had at least one seed.

Logging

The engine logs to the curl_reap logger: INFO for robots blocks and the final stats, WARNING for errors, DEBUG for each response.

import logging
logging.basicConfig(level=logging.INFO)

robots.txt

With respect_robots=True, the engine fetches and caches /robots.txt for each host and skips disallowed URLs, counting them in stats["robots_blocked"]. It checks rules for the * user agent. If robots.txt cannot be fetched or returns an error status, everything is allowed.

A Crawl-delay in robots.txt becomes the minimum delay for that host (the larger of it and delay=). Reaper.discovered_sitemaps lists Sitemap: URLs found in the robots.txt files of hosts crawled so far.

Crawling from sitemaps

SitemapSpider seeds itself from sitemaps instead of start_urls:

class Products(reap.SitemapSpider):
    sitemap_urls = ["https://shop.example.com"]   # /sitemap.xml is assumed
    url_pattern = r"/product/"

    def parse(self, page):
        yield {"url": page.url, "name": page.css_first("h1::text")}
  • A URL in sitemap_urls that does not end in .xml or .xml.gz gets /sitemap.xml appended.
  • Gzipped sitemaps are decompressed.
  • <loc> entries that end in .xml or .xml.gz are treated as nested sitemaps (sitemap index files) and followed.
  • Other <loc> URLs matching url_pattern (a regular expression searched in the URL; all URLs when None) are sent to parse.

AutoThrottle

With throttle=True, each domain gets an AutoThrottle that adapts a per-request delay to observed latency:

  • It keeps the last 20 response times for the domain. The target delay is the average latency divided by concurrency, but never less than the base delay (delay= or the robots Crawl-delay).
  • A 429 or 503 response doubles the delay immediately, treating a delay under 0.5 seconds as 0.5, up to 30 seconds.
  • After five healthy responses in a row, a delay above the target is halved, but not below the target.
  • A 429 or 503 response with Retry-After also starts a cooldown for the domain: no new request starts until it ends. The value is honored as sent (clamped to 24 hours), not capped at 30 seconds.

Start times are handed out as slots: each worker reserves the next free slot for the domain, one delay after the previous one, and sleeps until it arrives. With several workers on one domain, request starts are still spaced by the delay.

With throttle=False no delay is applied. That includes delay= and any robots.txt Crawl-delay; keep throttle=True when you need either. A Retry-After cooldown still applies.

reap.AutoThrottle(base_delay=0.0, target_concurrency=8, max_delay=30.0, enabled=True) is exported if you want to use the same logic in your own loop, with observe(latency, status=None) and wait().