curl_reap / Guides
Pipelines
Validate, transform, deduplicate and export crawl items with curl_reap pipelines: JSON Lines, CSV, SQLite with upsert, and custom pipelines.
On this page
Every item a spider yields flows through a chain of pipelines, in order. Each pipeline can change the item or drop it by returning None. Items that pass every pipeline are added to the result list and sent to on_item.
import curl_reap as reap
from curl_reap import DedupPipeline, JsonLinesPipeline, Pipeline
class PriceToFloat(Pipeline):
def process(self, item):
item["price"] = float(item["price"].strip("$"))
return item # or return None to drop it
reap.run(MySpider, pipelines=[
DedupPipeline(key="url"),
PriceToFloat(),
JsonLinesPipeline("out.jsonl"),
])
First principles
- A pipeline is a function from item to item, or to nothing. Each step gets the output of the step before it and returns the item, changed or not, or
Noneto drop it. Dropping is explicit and counted. - Order matters. Deduplicate before you export, and convert a field before a step that validates it. The chain runs in the order of the list.
- Each exporter decides when data reaches disk. JSON Lines writes and flushes every item, so items written before the process stops are already in the file. Comma-separated values (CSV) needs every key for its header row, so it writes once at the end. SQLite commits every row.
Built-in pipelines
DedupPipeline(key=None)
Drops items already seen. With key, items are compared by that field; without it, by the whole item (serialized as sorted JSON). The engine adds a DedupPipeline() at the front of the chain automatically, unless you pass dedup=False or your list already contains a DedupPipeline.
JsonLinesPipeline(path)
Writes each item as one JSON line as soon as it arrives, and flushes after each line. The file is opened in write mode when the crawl starts, so an existing file is replaced. Safe to use with many worker threads.
CsvPipeline(path)
Collects items in memory and writes a CSV when the crawl finishes. The header is the union of keys across all dict items, in first-seen order. Non-dict items are passed through but not written. Nothing is written if there were no items.
SqlitePipeline(path, table=“items”, key=None)
Inserts each dict item into a SQLite table, adding columns as new keys appear. Dict and list values are stored as JSON text.
- Without
key, the table has an autoincrement_rowidand every item is a new row. - With
key, that column is the primary key and items are written withINSERT OR REPLACE, so a repeated key updates the row.
from curl_reap import SqlitePipeline
SqlitePipeline("out.db") # append rows
SqlitePipeline("out.db", table="products", key="url") # upsert on url
Writing a pipeline
Subclass reap.Pipeline and override any of three methods:
open(): called once before the crawl starts.process(item): return the item (changed or not) to keep it, orNoneto drop it.close(): called once after the crawl ends.
The engine runs the whole chain under one lock, so within a crawl, process calls from different worker threads do not overlap. A pipeline object shared by crawls running at the same time still needs its own lock, as the built-in JSON Lines and SQLite pipelines have.
Dropped items are counted in Reaper.stats["dropped"]. The pipeline classes are also available as the reap.pipelines module.