Skip to content
curl_reap

curl_reap / Project

Responsible use and legal notes

What curl_reap does and does not do, your responsibilities when scraping, and the no-warranty terms, summarized from the project's LEGAL.md.

On this page
  1. What the library does
  2. What it does not do
  3. Your responsibilities
  4. No warranty, no liability
  5. Reporting misuse

This page summarizes the project’s LEGAL.md. It is not legal advice. Web scraping legality depends on your jurisdiction, what you scrape, and how.

What the library does

curl_reap is a Hypertext Transfer Protocol (HTTP) client and HyperText Markup Language (HTML) parsing library. It sends requests with browser-shaped Transport Layer Security (TLS) settings and headers, parses responses with Cascading Style Sheets (CSS) and XML Path Language (XPath) selectors that can re-locate elements after markup changes, and provides a concurrent crawl engine with throttling and retry logic.

Each of these is something a real browser or a well-behaved crawler already does. The library does not exploit a vulnerability, does not decrypt anything, and does not access systems you are not otherwise permitted to reach.

What it does not do

  • It does not solve CAPTCHA challenges (Completely Automated Public Turing tests to tell Computers and Humans Apart).
  • It does not bypass authentication, paywalls or rate-limit bans.
  • It is not designed or maintained to defeat dedicated anti-bot services such as Cloudflare, DataDome, PerimeterX or Akamai Bot Manager. If a site has deployed one of these and is actively blocking you, that block is the line the maintainers expect you to respect.

Your responsibilities

Before you scrape anything:

  1. Check robots.txt. It is not legally binding everywhere, but ignoring it is the first thing courts and opposing counsel point to. The crawl engine can enforce it with respect_robots=True (see Crawling).
  2. Read the site’s Terms of Service. Scraping public, unauthenticated pages sits on firmer legal ground (LEGAL.md cites hiQ Labs v. LinkedIn, 9th Cir.) than scraping behind a login or after you have been told to stop.
  3. Do not circumvent technical access controls. Looking like a normal browser at the TLS layer is different from actively defeating a bot-detection challenge a site deployed to block you. The first is generally low risk; the second raises exposure under the Computer Fraud and Abuse Act (CFAA) and the Digital Millennium Copyright Act (DMCA) in the United States, and analogous statutes elsewhere.
  4. Handle personal data lawfully. If what you scrape includes names, emails or other personal data, the General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA) or your local equivalent may apply to you as the data controller, however you collected it.
  5. Respect copyright. Scraping is not a license to republish. Extract facts and data; do not mirror copyrighted text or media wholesale.
  6. Throttle and identify yourself. Use reasonable concurrency and throttle=True, and consider a descriptive User-Agent or contact header so site operators can reach you. A custom Profile or a session header sets the User-Agent.

No warranty, no liability

curl_reap is provided under the MIT License, “as is”, without warranty of any kind. The maintainers are not responsible for how you use it, including any legal consequences of that use. See the LICENSE for the full text.

Reporting misuse

If you believe curl_reap is being used to violate a site’s terms, scrape data unlawfully, or evade an access control in a way that concerns you, open an issue at github.com/anishfyi/curl_reap/issues.