# Data This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. Output tiles are written to `www/static/tiles/`. We don't really need suuuuuuper up-to-date maps for this, since we trim a lot of data; the tiles are mainly used as a way to orient the user as they create their own map. All tile generation is done via the CLI: ```sh deno task data [options] deno task data --help ``` ## Dependencies The following tools must be installed and on `PATH`: - [`tilemaker`](https://github.com/systemed/tilemaker) — PBF → PMTiles conversion (`build`, `build:world`) - [`osmium`](https://osmcode.org/osmium-tool/) — PBF extraction (`extract` command) The following resources are necessary to generate pmtiles: - [Download a planet `.osm.pbf`](https://wiki.openstreetmap.org/wiki/Planet.osm) for the world-level pmtiles (put it in `./osm/planet-latest.osm.pbf`). For me, using a torrentfile has been the most stable way. - Download regional .osm.pbf files for regions if you don't want to self-extract (can download with `download:osm`) - Coastlines WGS84 projection from https://osmdata.openstreetmap.de/data/coastlines.html (put it in `./cli/shared/tilemaker/coastline`) (todo for me: create a `download:coastlines` cli cmd) ## CLI Commands | Command | Description | | ------------------------------------------ | ------------------------------------------------------------------- | | `list [--search ]` | Browse available region slugs | | `download:osm [--force]` | Download `.osm.pbf` from Geofabrik | | `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s) | | `extract --from ` | Carve a sub-region from a larger PBF using osmium | | `build ` | Convert `.osm.pbf` → `.pmtiles` using tilemaker | | `trim:world` | Strip planet PBF to only data needed for world tiles (saves memory) | | `build:world` | Build world-scale basemap tiles | | `clean --region \| --all` | Remove intermediate files | ## World Tiles The world basemap is built from OSM data using a tilemaker pipeline, sourced from a full planet PBF. Place `planet-latest.osm.pbf` in `data/osm/` and run: ```sh deno task data trim:world # recommended first step — see below deno task data build:world [--maxzoom <5|7|9>] # default maxzoom: 7 ``` Output is written to `www/static/tiles/world_z.pmtiles`. The config and Lua script used are separate from the regional ones: - `data/cli/shared/tilemaker/config.world.json` — layer definitions for the world overview - `data/cli/shared/tilemaker/process.world.lua` — feature processing logic Note: this takes a super long time to run. ### Trimming the planet file (recommended) The full planet PBF (~75 GB) contains a huge amount of data that is irrelevant at world zoom levels — buildings, addresses, shop/amenity POIs, minor roads, etc. Running `trim:world` uses `osmium tags-filter` to produce a much smaller `planet-trimmed.osm.pbf` containing only the tags that `process.world.lua` actually reads: - Place nodes (countries, states, cities, towns, villages) - Natural peaks and volcanoes - Administrative boundaries - Major roads (motorway → secondary) and ferry routes - Main-line railways - Rivers, waterways, water polygons - Landcover and landuse polygons - Parks and nature reserves ```sh deno task data trim:world # creates data/osm/planet-trimmed.osm.pbf deno task data trim:world --overwrite # re-run and replace an existing trimmed file ``` When `planet-trimmed.osm.pbf` is present, `build:world` will automatically use it instead of the full planet file. This significantly reduces tilemaker's peak memory usage and overall build time. ## Regional Tiles Regional tiles follow a three-stage pipeline: ``` Geofabrik (download) → osmium (extract; optional) → tilemaker (build) → .pmtiles ``` ### 1. Download Regional `.osm.pbf` data and `.poly` boundary files are sourced from [Geofabrik](https://download.geofabrik.de/). Region slugs match Geofabrik's hierarchy (e.g. `asia/japan/kansai`, `europe/monaco`). `.poly` files for all 499 Geofabrik regions are already committed to `data/poly/` and don't need to be refreshed under normal circumstances. OSM `.pbf` files are gitignored due to size and must be downloaded as needed. ```sh deno task data download:osm europe/monaco deno task data download:poly europe/monaco # already present; only needed to refresh ``` ### 2. Extract (optional) If you already have a large regional PBF, you can carve out a sub-region using `osmium` rather than downloading from Geofabrik again: ```sh deno task data extract asia/taiwan --from asia ``` This requires both the source PBF and the target `.poly` file to be present. ### 3. Build Convert `.osm.pbf` to `.pmtiles` using `tilemaker`: ```sh deno task data build europe/monaco ``` Output is written to `www/static/tiles/.pmtiles`. The tilemaker config and Lua processing script are: - `data/cli/shared/tilemaker/config.json` — layer definitions (OpenMapTiles schema, zoom 0–14) - `data/cli/shared/tilemaker/process.lua` — feature processing logic ## Common Workflows **Small country — direct download:** ```sh deno task data download:osm europe/monaco deno task data build europe/monaco ``` **Country carved from a larger extract:** ```sh deno task data download:osm asia # large, but reusable across multiple countries deno task data extract asia/taiwan --from asia deno task data build asia/taiwan ``` ## Directory Structure ``` data/ osm/ Downloaded/extracted .osm.pbf files (gitignored) poly/ Boundary .poly files for all 499 Geofabrik regions (committed) cli/ CLI source code ``` ## External Resources - **[Geofabrik](https://download.geofabrik.de)** — regional OSM extracts; provides both `.osm.pbf` and `.poly` files - **[OpenStreetMap](https://planet.openstreetmap.org)** — planet PBF source for world basemap tiles