Skip to content
curl_reap

curl_reap / Guides

Structured extraction

Get JSON-LD, meta tags, links, images, tables, named fields and readable markdown from any page, with one curl_reap method call each.

On this page
  1. First principles
  2. jsonld()
  3. meta_tags()
  4. links()
  5. images()
  6. tables()
  7. extract()
  8. markdown()

The data most scrapes want is often already structured in the page. These methods work on any Response and on any Selector. They use absolute XML Path Language (XPath) queries (//a, //table), so they always read the whole document, even when called on a Selector for one element.

First principles

  • Sites publish structured data for machines already. Search engines read JavaScript Object Notation for Linked Data (JSON-LD), and link previews read Open Graph and Twitter meta tags. That data exists for other programs to read, so it does not depend on the page’s layout. Read it before you write a selector.
  • Some structure is in the markup itself. Links, images and tables have fixed HyperText Markup Language (HTML) elements, so one generic method can read them on any site.
  • What is left is prose. markdown() keeps the reading order and drops the layout, which suits search indexes and language models.
  • Named fields beat loose selectors. extract() turns a mapping of names to Cascading Style Sheets (CSS) queries into one dict, so a scrape’s output shape is written in one place.
import curl_reap as reap

page = reap.get("https://example.com/product/42")
page.jsonld()
page.meta_tags()
page.links(internal_only=True)
page.images()
page.tables()
page.markdown()

jsonld()

Returns every <script type="application/ld+json"> block as a list of dicts. A top-level JSON array is flattened into the list, and so is an object’s @graph array. Blocks that are not valid JSON are skipped.

for block in page.jsonld():
    if block.get("@type") == "Product":
        print(block.get("name"), block.get("offers"))

meta_tags()

Returns one dict of page metadata:

  • title: the <title> text.
  • One key per <meta> with a name, property or itemprop attribute and a content value, such as description, og:title or twitter:card. If a key appears twice, the first value wins.
  • canonical: the href of <link rel="canonical">, when present.
meta = page.meta_tags()
print(meta.get("title"), meta.get("og:image"), meta.get("canonical"))

links(absolute=True, unique=True, internal_only=False) returns [{"url": ..., "text": ...}] for every <a href>.

  • absolute=True resolves relative links against the page’s base URL: the first <base href> when the page has one, otherwise the page URL.
  • unique=True drops repeated URLs. The fragment (#...) is removed before comparing, and from the returned URL.
  • internal_only=True keeps links with the same origin as the page: the same scheme, host and port (default ports filled in). It compares with the real page URL, not with a <base href> that points elsewhere.
  • Only http and https links are returned. javascript:, mailto:, tel:, data: and other schemes are dropped in any letter case, and so are links that start with #.

images()

images(absolute=True) returns [{"url": ..., "alt": ...}] for every <img>. The URL comes from src, then data-src, then data-lazy-src, which covers common lazy-loading markup, and is resolved against the base URL like links. Only http and https images are returned, so inline data: images are skipped.

tables()

Returns every <table> as a list of rows, each row a list of cell strings from <th> and <td> cells. Header rows are returned like any other row, so the first row is the header when the table has one.

for table in page.tables():
    header, *rows = table
    print(header, len(rows))

extract()

extract(fields, *, many=False) takes a mapping of names to CSS queries and returns a dict with the same names:

page = reap.get("https://quotes.toscrape.com")
page.extract({"first_quote": "span.text::text", "next": "li.next a::attr(href)"})
# {"first_quote": "...", "next": "/page/2/"}

page.extract({"quotes": "span.text::text", "authors": "small.author::text"}, many=True)
# {"quotes": [...], "authors": [...]}
  • With many=False (the default), each name gets the first match, or None when nothing matches.
  • With many=True, each name gets the list of all matches.
  • A ::text or ::attr() query gives that string. A query that matches elements gives each element’s text.
  • fields must be a mapping of string names to string queries; anything else raises TypeError.

markdown()

markdown(max_len=None, main_only=False) returns the page’s readable content as markdown-style text. It collects h1 to h4 (as # headings), paragraphs, list items (as - ), blockquotes (as > ), pre blocks (as fenced code) and figure captions, in document order, separated by blank lines. With max_len, the output is cut at that many characters and ends with [truncated].

main_only=True reads only the first <main> or <article> element, after removing its nav, footer, aside, script and style elements from a copy (the page itself is not changed). Nested elements are emitted once, at their outermost level. If the page has no <main> or <article>, the result is an empty string, so fall back to the full page:

text = page.markdown(main_only=True, max_len=20000) or page.markdown(max_len=20000)

This is also what reap get URL prints by default. It is a convenient input for search indexes and language models.