Skip to content
curl_reap

curl_reap / Guides

Selectors and self-healing

Select with CSS, XPath and regex, find elements by text or similarity, and save signatures so selectors re-find elements after markup changes.

On this page
  1. First principles
  2. Selectors
  3. CSS
  4. XPath and regex
  5. Values
  6. SelectorList helpers
  7. Finders
  8. find_by_text
  9. find_similar
  10. Self-healing selectors
  11. Save a signature
  12. Relocate with auto_match
  13. How the score works
  14. Low-level functions

This page covers selecting data from HTML: Cascading Style Sheets (CSS) selectors, XML Path Language (XPath) and regular expressions, finders that need no selector, and self-healing selectors that survive markup changes.

First principles

  • A HyperText Markup Language (HTML) page is a tree of elements. A selector describes a set of nodes in that tree: CSS by tag, class, id and position, XPath by any path, including text and attribute nodes.
  • A selector is a bet that the markup stays the same. When a site renames a class, the query returns nothing, and it does so without an error. Check for empty results where it matters.
  • Self-healing spreads the bet. A saved signature records several independent signals about one element: tag, classes, id, attribute names, text and tag path. Losing one signal lowers the score; losing most of them means the element really changed.

Selectors

Every Response parses its body with lxml on first use and passes selector calls to a Selector. You can also build a Selector from HTML you already have:

from curl_reap import Selector

sel = Selector(content="<ul><li><a href='/a'>A</a></li></ul>", url="https://example.com/")

CSS

css(query, auto_match=False, identifier=None, storage=None) returns a SelectorList. Two pseudo elements are supported at the end of a query:

  • ::text returns the element’s text content, stripped, as strings.
  • ::attr(name) returns the attribute value as strings (or None where the attribute is missing).

Without a pseudo element, the list holds child Selector objects you can query again.

page.css("h1::text").get()                      # first match, or None
page.css("a.item::attr(href)").getall()         # every href as a list
page.css_first("span.price::text", default="n/a")

for card in page.css("div.card"):
    title = card.css_first(".title::text")
    link = card.css_first("a::attr(href)")

css_first(query, default=None, **kw) returns the first result or default. An invalid CSS query returns an empty list rather than raising, so check your selectors if a query unexpectedly finds nothing.

XPath and regex

page.xpath("//div[@class='quote']/span/text()").getall()
page.re(r"price:\s*([\d.]+)")                   # re.findall over the element's HTML
page.re_first(r"SKU-(\d+)", default=None)

xpath(query) returns strings for text and attribute results and Selector objects for elements. re(pattern, flags=0) and re_first(pattern, default=None, flags=0) run on the outer HTML of the element.

Values

el = page.css_first("a.author")
el.text          # text content, stripped
el.attr("href")  # one attribute, or a default: el.attr("href", "")
el.attrib        # dict of all attributes
el.html          # outer HTML

SelectorList helpers

A SelectorList is a Python list with extra methods:

  • get(default=None): the first item, or default.
  • getall(): all items as a plain list.
  • text(): text of each element (strings pass through).
  • attr(name, default=None): the attribute from each element.
  • css(query, **kw): runs the query on every element and concatenates the results.

Finders

find_by_text

find_by_text(text, partial=True, first=False, deepest=True) finds elements whose full text content contains text (or equals it, with partial=False). With deepest=True, ancestors of other matches are dropped, so you get the <a>, not every wrapper up to <html>. first=True stops after one result.

signin = page.find_by_text("Sign in", first=True).get()

find_similar

find_similar(sample, threshold=0.6, limit=None) returns elements structurally similar to a sample, best match first. It is useful for repeated cards or rows without writing a class selector:

first_card = page.css_first("div.card")
all_cards = page.find_similar(first_card)   # other cards like it
print(len(all_cards))

The sample itself is not included in the result.

Self-healing selectors

Selectors break when a site renames a class or reshuffles its DOM. curl_reap can save a structural signature of an element once and later relocate it by scoring every element against that signature.

Save a signature

btn = page.css_first("a.buy-btn")
btn.save("buy_button")                         # writes .reap_selectors.json
btn.save("buy_button", storage="selectors.json")

save(identifier, storage=None) stores the element’s tag, classes, id, attribute names, the first 48 characters of its direct text, its depth, its sibling index and its tag path. The default store is .reap_selectors.json in the current directory.

Relocate with auto_match

later = reap.get("https://shop.example.com/item/99")
btn = later.css_first("a.buy-btn", auto_match=True, identifier="buy_button")
if btn is not None:
    print(btn.attr("href"))

When the normal query finds nothing and auto_match=True, curl_reap loads the signature saved under identifier (or under the query string itself, if no identifier is given) from storage, scores every element, and returns the best one whose score is above 0.6. If nothing scores above 0.6, the result is empty.

How the score works

similarity returns a value from 0 to 1, a weighted sum divided by the total weight of 11:

  • same tag: 2
  • class set overlap (Jaccard): 3
  • same non-empty id: 2
  • attribute name overlap (Jaccard): 1
  • same non-empty direct text: 1.5
  • same tag path: 1.5, or 0.75 when only the last three tags match

A renamed class alone costs up to 3 of the 11 points. Relocation works best when the element keeps other signals: an id, some of its classes, its attributes, its text or its position. For example, an <a id="cta" class="buy-btn primary" data-track="x">Buy now</a> renamed to class="purchase-button primary" still scores well above the threshold. An element whose only distinguishing feature was a single renamed class may fall below it.

Low-level functions

For custom workflows, the same machinery is exported:

  • reap.signature(element): the signature dict of an lxml element.
  • reap.similarity(sig_a, sig_b): the 0 to 1 score.
  • reap.save(identifier, element, storage=None): store a signature.
  • reap.relocate(identifier, tree, storage=None, threshold=0.6): the best matching lxml element in tree, or None.

These work on raw lxml elements. From a Selector, use its save() and css(auto_match=True) methods.