Skip to content
curl_reap

curl_reap / Guides

JavaScript rendering

Render JavaScript-built pages in a Playwright browser with reap.render, only when needed with render_if_empty, or reuse one with BrowserSession.

On this page
  1. First principles
  2. Install
  3. render()
  4. render_if_empty()
  5. BrowserSession
  6. Before you render

Some pages are empty over plain Hypertext Transfer Protocol (HTTP): a single-page app ships an empty shell and builds the page with JavaScript. For these, curl_reap can load the page in a real browser through Playwright and give you the Document Object Model (DOM) after JavaScript has run. Rendering is for ordinary JavaScript-heavy pages on sites you are allowed to fetch. It is not wired to solve or retry bot-wall challenges.

First principles

  • An HTTP client gets the bytes the server sends, nothing more. When the server sends a shell and a script, the content only exists after a JavaScript engine runs that script. A browser is the only reliable way to get it.
  • A browser is expensive. Each render starts a browser process, loads every subresource and runs scripts. Render only when the plain response is hollow, and keep one browser alive for many pages with BrowserSession.
  • A browser is a separate client. It has its own network stack, so its handshake and headers are the browser’s own, not a curl_reap profile, and cookies do not move between a Session and the browser.
  • A rendered page is a snapshot. The returned Response holds the DOM at the moment the call returned, serialized as HTML. Everything that works on a Response works on it.

Install

Rendering is an optional extra:

pip install "curl-reap[js]"
playwright install chromium

Playwright is imported only when a browser is first needed. Without it, the call raises ImportError with these install instructions.

render()

import curl_reap as reap

page = reap.render("https://example.com/app")
print(page.status, page.url)
print(page.css("h1::text").get())
print(page.meta["rendered"])    # True

reap.render(url, *, timeout=30, wait_until="networkidle", proxy=None, user_agent=None, browser_channel="chromium", headless=True, script=None):

  • timeout: navigation timeout in seconds.
  • wait_until: the Playwright load state to wait for, such as "networkidle" or "load".
  • proxy: a proxy server Uniform Resource Locator (URL) for the browser. Credentials in the URL are passed to Playwright as a separate username and password.
  • user_agent: override the browser’s User-Agent.
  • browser_channel: "chromium", "firefox" or "webkit" picks that Playwright engine. Any other value, such as "chrome" or "msedge", launches Chromium with that Playwright channel, which needs the branded browser installed.
  • headless: run without a visible window.
  • script: a JavaScript string evaluated after load, for example to scroll. Its result is ignored.

Each call launches a browser, loads one page and closes the browser. The return value is a normal Response built from the rendered HTML, so css, xpath, jsonld(), markdown() and the other methods work. Things to know about it:

  • status is the real status of the navigation, or None when Playwright reports no response. Check it as you would for any response.
  • url is the final URL after redirects and client-side navigation.
  • headers are the navigation response’s headers, without Content-Encoding, Content-Length and Content-Type, plus Content-Type: text/html; charset=utf-8 for the snapshot.
  • meta holds rendered (True), request_url (the URL you asked for) and nav_headers (the navigation headers as reported).
  • Cookies from the browser are not copied into any Session.

render_if_empty()

reap.render_if_empty(response, *, min_bytes=2000, **kw) renders response.url only when the plain HTTP response looks hollow: it is ok, it does not look like a challenge page, and its body is shorter than min_bytes. Other keywords go to render.

Error responses (ok is False) and challenge-looking responses are returned as they are, without rendering. Rendering them would not fix a block, so the function leaves them to you.

It returns a tuple (response, rendered), where rendered is None when rendering was not needed:

r = reap.get("https://example.com/app")
r, rendered = reap.render_if_empty(r)
page = rendered or r

BrowserSession

reap.BrowserSession keeps one browser, one context and one page alive across navigations, so cookies and storage carry from one goto() to the next and you pay the browser’s start-up cost once:

import curl_reap as reap

allowed = lambda u: u.startswith("https://quotes.toscrape.com")

with reap.BrowserSession(url_policy=allowed) as browser:
    first = browser.goto("https://quotes.toscrape.com/js/")
    second = browser.follow("/page/2")
    print(second.status, second.url, len(second.css("div.quote")))

BrowserSession(*, timeout=30, wait_until="domcontentloaded", proxy=None, user_agent=None, browser_channel="chromium", headless=True, cdp_url=None, url_policy=None) takes the same browser options as render (with wait_until defaulting to "domcontentloaded", and script passed to goto() instead), plus:

  • cdp_url: attach to a remote browser you are authorized to use, through the Chrome DevTools Protocol (CDP), with an http://host:port or ws:// endpoint. The session creates and owns a fresh context and page there. Closing it disconnects and leaves the remote browser’s other contexts alone.
  • url_policy: a callable checked against every URL. goto() checks the target before the browser even launches and raises URLPolicyError when the policy rejects it. Inside the browser, every document and subresource request is routed through the policy, and a rejected request is aborted. In this mode service workers are blocked and the browser cannot follow redirects itself: any 3xx response is aborted. Use a Session to walk an allowed redirect chain and give the browser the final URL.

Methods and properties:

  • goto(url, *, wait_for=None, script=None, timeout=None, wait_until=None): navigate and return a Response snapshot, as render does. wait_for waits for a Cascading Style Sheets (CSS) selector, bounded by the navigation timeout. timeout and wait_until override the session defaults for this call.
  • follow(target, **kw): resolve target against the current page (its <base href> when it has one) and goto it.
  • page: the underlying Playwright page, opened on first access.
  • url: the current page URL, or None before the first navigation.
  • close(): close the context, the browser and the Playwright driver. Calling it twice is safe. The with block calls it for you.

The browser starts lazily, on the first goto(), on page, or when the with block is entered.

Before you render

Rendering a page a site serves to browsers is ordinary. Working through a bot-detection challenge that a site deployed to block you is a different matter. Read Responsible use and legal notes.