curl_reap / Guides
JavaScript rendering
Render JavaScript-built pages in a Playwright browser with reap.render, only when needed with render_if_empty, or reuse one with BrowserSession.
Some pages are empty over plain Hypertext Transfer Protocol (HTTP): a single-page app ships an empty shell and builds the page with JavaScript. For these, curl_reap can load the page in a real browser through Playwright and give you the Document Object Model (DOM) after JavaScript has run. Rendering is for ordinary JavaScript-heavy pages on sites you are allowed to fetch. It is not wired to solve or retry bot-wall challenges.
First principles
- An HTTP client gets the bytes the server sends, nothing more. When the server sends a shell and a script, the content only exists after a JavaScript engine runs that script. A browser is the only reliable way to get it.
- A browser is expensive. Each render starts a browser process, loads every subresource and runs scripts. Render only when the plain response is hollow, and keep one browser alive for many pages with
BrowserSession. - A browser is a separate client. It has its own network stack, so its handshake and headers are the browser’s own, not a curl_reap profile, and cookies do not move between a
Sessionand the browser. - A rendered page is a snapshot. The returned
Responseholds the DOM at the moment the call returned, serialized as HTML. Everything that works on aResponseworks on it.
Install
Rendering is an optional extra:
pip install "curl-reap[js]"
playwright install chromium
Playwright is imported only when a browser is first needed. Without it, the call raises ImportError with these install instructions.
render()
import curl_reap as reap
page = reap.render("https://example.com/app")
print(page.status, page.url)
print(page.css("h1::text").get())
print(page.meta["rendered"]) # True
reap.render(url, *, timeout=30, wait_until="networkidle", proxy=None, user_agent=None, browser_channel="chromium", headless=True, script=None):
timeout: navigation timeout in seconds.wait_until: the Playwright load state to wait for, such as"networkidle"or"load".proxy: a proxy server Uniform Resource Locator (URL) for the browser. Credentials in the URL are passed to Playwright as a separate username and password.user_agent: override the browser’s User-Agent.browser_channel:"chromium","firefox"or"webkit"picks that Playwright engine. Any other value, such as"chrome"or"msedge", launches Chromium with that Playwright channel, which needs the branded browser installed.headless: run without a visible window.script: a JavaScript string evaluated after load, for example to scroll. Its result is ignored.
Each call launches a browser, loads one page and closes the browser. The return value is a normal Response built from the rendered HTML, so css, xpath, jsonld(), markdown() and the other methods work. Things to know about it:
statusis the real status of the navigation, orNonewhen Playwright reports no response. Check it as you would for any response.urlis the final URL after redirects and client-side navigation.headersare the navigation response’s headers, withoutContent-Encoding,Content-LengthandContent-Type, plusContent-Type: text/html; charset=utf-8for the snapshot.metaholdsrendered(True),request_url(the URL you asked for) andnav_headers(the navigation headers as reported).- Cookies from the browser are not copied into any
Session.
render_if_empty()
reap.render_if_empty(response, *, min_bytes=2000, **kw) renders response.url only when the plain HTTP response looks hollow: it is ok, it does not look like a challenge page, and its body is shorter than min_bytes. Other keywords go to render.
Error responses (ok is False) and challenge-looking responses are returned as they are, without rendering. Rendering them would not fix a block, so the function leaves them to you.
It returns a tuple (response, rendered), where rendered is None when rendering was not needed:
r = reap.get("https://example.com/app")
r, rendered = reap.render_if_empty(r)
page = rendered or r
BrowserSession
reap.BrowserSession keeps one browser, one context and one page alive across navigations, so cookies and storage carry from one goto() to the next and you pay the browser’s start-up cost once:
import curl_reap as reap
allowed = lambda u: u.startswith("https://quotes.toscrape.com")
with reap.BrowserSession(url_policy=allowed) as browser:
first = browser.goto("https://quotes.toscrape.com/js/")
second = browser.follow("/page/2")
print(second.status, second.url, len(second.css("div.quote")))
BrowserSession(*, timeout=30, wait_until="domcontentloaded", proxy=None, user_agent=None, browser_channel="chromium", headless=True, cdp_url=None, url_policy=None) takes the same browser options as render (with wait_until defaulting to "domcontentloaded", and script passed to goto() instead), plus:
cdp_url: attach to a remote browser you are authorized to use, through the Chrome DevTools Protocol (CDP), with anhttp://host:portorws://endpoint. The session creates and owns a fresh context and page there. Closing it disconnects and leaves the remote browser’s other contexts alone.url_policy: a callable checked against every URL.goto()checks the target before the browser even launches and raisesURLPolicyErrorwhen the policy rejects it. Inside the browser, every document and subresource request is routed through the policy, and a rejected request is aborted. In this mode service workers are blocked and the browser cannot follow redirects itself: any 3xx response is aborted. Use aSessionto walk an allowed redirect chain and give the browser the final URL.
Methods and properties:
goto(url, *, wait_for=None, script=None, timeout=None, wait_until=None): navigate and return aResponsesnapshot, asrenderdoes.wait_forwaits for a Cascading Style Sheets (CSS) selector, bounded by the navigation timeout.timeoutandwait_untiloverride the session defaults for this call.follow(target, **kw): resolvetargetagainst the current page (its<base href>when it has one) andgotoit.page: the underlying Playwright page, opened on first access.url: the current page URL, orNonebefore the first navigation.close(): close the context, the browser and the Playwright driver. Calling it twice is safe. Thewithblock calls it for you.
The browser starts lazily, on the first goto(), on page, or when the with block is entered.
Before you render
Rendering a page a site serves to browsers is ordinary. Working through a bot-detection challenge that a site deployed to block you is a different matter. Read Responsible use and legal notes.