Skip to content
curl_reap

curl_reap / Guides

Geocoding

Turn scraped names and areas into coordinates with curl_reap's cascading, cached and rate-limited Geocoder over OpenStreetMap Nominatim.

On this page
  1. Usage
  2. The cascade
  3. Options
  4. Be a good citizen

Scraped listings often carry a name and an area but no coordinates. A single geocoding query fails often: the full marketing name (“Chelsea Cloisters Serviced Apartments”) is not in the gazetteer, and a bare district query puts every listing on the same centroid. Geocoder tries a cascade of queries and tells you how precise the hit is.

Usage

from curl_reap import Geocoder

geo = Geocoder()
hit = geo.geocode(name="Chelsea Cloisters Serviced Apartments",
                  area="Kensington", city="London", country="United Kingdom")
# {"lat": 51.49..., "lon": -0.17..., "precision": "name", ...}

if hit and hit["precision"] in ("name", "district"):
    listing["lat"], listing["lon"] = hit["lat"], hit["lon"]

reap.geocode(name=None, area=None, city=None, country=None) does the same with a shared default Geocoder.

The cascade

geocode(name=None, area=None, city=None, country=None) builds candidate queries and stops at the first one that returns a result:

  1. The name with area, city and country, then the name with city and country. Only when a name and a city or area are given.
  2. The same two queries with the name cleaned: trailing generic words such as “Serviced Apartments”, “Aparthotel”, “Hotel”, “Suites” or by <brand> are stripped. Only when cleaning changes the name.
  3. The area with city and country, labeled district.
  4. The city with country, labeled city.

The result is a dict with lat and lon (floats), precision ("name", "district" or "city"), importance and display_name from Nominatim, and the query that matched. It is None when nothing matched. Keep the name and district hits and drop or flag the coarse city ones.

clean_name(name) is available on its own if you want the cleaned name.

Options

Geocoder(endpoint="https://nominatim.openstreetmap.org/search", user_agent=..., cache=".reap_geocode.json", min_interval=1.1, profile="chrome", min_importance=0.0):

  • endpoint: a Nominatim-compatible search Uniform Resource Locator (URL).
  • user_agent: sent on every request. The default is curl_reap/2.0 (+https://github.com/anishfyi/curl_reap).
  • cache: a JavaScript Object Notation (JSON) file of past lookups, keyed by the four inputs. Misses are cached too, so repeat runs make no requests.
  • min_interval: minimum seconds between requests, enforced across threads.
  • profile: the transport profile for the underlying Session.
  • min_importance: ignore hits whose Nominatim importance is lower than this and try the next candidate.

Be a good citizen

Nominatim’s public service asks for an identifying User-Agent and at most one request per second. The defaults set both. Set your own user_agent with contact details for larger jobs, or point endpoint at your own Nominatim instance.