Offline-capable geomap, meant for storing location bookmarks
1# Data
2
3This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. Output tiles are written to `www/static/tiles/`. We don't really need suuuuuuper up-to-date maps for this, since we trim a lot of data; the tiles are mainly used as a way to orient the user as they create their own map.
4
5All tile generation is done via the CLI:
6
7```sh
8deno task data <command> [options]
9deno task data --help
10```
11
12## Dependencies
13
14The following tools must be installed and on `PATH`:
15
16- [`tilemaker`](https://github.com/systemed/tilemaker) — PBF → PMTiles conversion (`build`, `build:world`)
17- [`osmium`](https://osmcode.org/osmium-tool/) — PBF extraction (`extract` command)
18
19The following resources are necessary to generate pmtiles:
20
21- [Download a planet `.osm.pbf`](https://wiki.openstreetmap.org/wiki/Planet.osm) for the world-level pmtiles (put it in `./osm/planet-latest.osm.pbf`). For me, using a torrentfile has been the most stable way.
22- Download regional .osm.pbf files for regions if you don't want to self-extract (can download with `download:osm`)
23- Coastlines WGS84 projection from https://osmdata.openstreetmap.de/data/coastlines.html (put it in `./cli/shared/tilemaker/coastline`) (todo for me: create a `download:coastlines` cli cmd)
24
25## CLI Commands
26
27| Command | Description |
28| ------------------------------------------ | ------------------------------------------------------------------- |
29| `list [--search <term>]` | Browse available region slugs |
30| `download:osm <region> [--force]` | Download `.osm.pbf` from Geofabrik |
31| `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s) |
32| `extract <region> --from <source>` | Carve a sub-region from a larger PBF using osmium |
33| `build <region>` | Convert `.osm.pbf` → `.pmtiles` using tilemaker |
34| `trim:world` | Strip planet PBF to only data needed for world tiles (saves memory) |
35| `build:world` | Build world-scale basemap tiles |
36| `clean --region <slug> \| --all` | Remove intermediate files |
37
38## World Tiles
39
40The world basemap is built from OSM data using a tilemaker pipeline, sourced from a full planet PBF. Place `planet-latest.osm.pbf` in `data/osm/` and run:
41
42```sh
43deno task data trim:world # recommended first step — see below
44deno task data build:world [--maxzoom <5|7|9>] # default maxzoom: 7
45```
46
47Output is written to `www/static/tiles/world_z<maxzoom>.pmtiles`. The config and Lua script used are separate from the regional ones:
48
49- `data/cli/shared/tilemaker/config.world.json` — layer definitions for the world overview
50- `data/cli/shared/tilemaker/process.world.lua` — feature processing logic
51
52Note: this takes a super long time to run.
53
54### Trimming the planet file (recommended)
55
56The full planet PBF (~75 GB) contains a huge amount of data that is irrelevant at world zoom levels — buildings, addresses, shop/amenity POIs, minor roads, etc. Running `trim:world` uses `osmium tags-filter` to produce a much smaller `planet-trimmed.osm.pbf` containing only the tags that `process.world.lua` actually reads:
57
58- Place nodes (countries, states, cities, towns, villages)
59- Natural peaks and volcanoes
60- Administrative boundaries
61- Major roads (motorway → secondary) and ferry routes
62- Main-line railways
63- Rivers, waterways, water polygons
64- Landcover and landuse polygons
65- Parks and nature reserves
66
67```sh
68deno task data trim:world # creates data/osm/planet-trimmed.osm.pbf
69deno task data trim:world --overwrite # re-run and replace an existing trimmed file
70```
71
72When `planet-trimmed.osm.pbf` is present, `build:world` will automatically use it instead of the full planet file. This significantly reduces tilemaker's peak memory usage and overall build time.
73
74## Regional Tiles
75
76Regional tiles follow a three-stage pipeline:
77
78```
79Geofabrik (download) → osmium (extract; optional) → tilemaker (build) → .pmtiles
80```
81
82### 1. Download
83
84Regional `.osm.pbf` data and `.poly` boundary files are sourced from [Geofabrik](https://download.geofabrik.de/). Region slugs match Geofabrik's hierarchy (e.g. `asia/japan/kansai`, `europe/monaco`).
85
86`.poly` files for all 499 Geofabrik regions are already committed to `data/poly/` and don't need to be refreshed under normal circumstances. OSM `.pbf` files are gitignored due to size and must be downloaded as needed.
87
88```sh
89deno task data download:osm europe/monaco
90deno task data download:poly europe/monaco # already present; only needed to refresh
91```
92
93### 2. Extract (optional)
94
95If you already have a large regional PBF, you can carve out a sub-region using `osmium` rather than downloading from Geofabrik again:
96
97```sh
98deno task data extract asia/taiwan --from asia
99```
100
101This requires both the source PBF and the target `.poly` file to be present.
102
103### 3. Build
104
105Convert `.osm.pbf` to `.pmtiles` using `tilemaker`:
106
107```sh
108deno task data build europe/monaco
109```
110
111Output is written to `www/static/tiles/<region-name>.pmtiles`. The tilemaker config and Lua processing script are:
112
113- `data/cli/shared/tilemaker/config.json` — layer definitions (OpenMapTiles schema, zoom 0–14)
114- `data/cli/shared/tilemaker/process.lua` — feature processing logic
115
116## Common Workflows
117
118**Small country — direct download:**
119
120```sh
121deno task data download:osm europe/monaco
122deno task data build europe/monaco
123```
124
125**Country carved from a larger extract:**
126
127```sh
128deno task data download:osm asia # large, but reusable across multiple countries
129deno task data extract asia/taiwan --from asia
130deno task data build asia/taiwan
131```
132
133## Directory Structure
134
135```
136data/
137 osm/ Downloaded/extracted .osm.pbf files (gitignored)
138 poly/ Boundary .poly files for all 499 Geofabrik regions (committed)
139 cli/ CLI source code
140```
141
142## External Resources
143
144- **[Geofabrik](https://download.geofabrik.de)** — regional OSM extracts; provides both `.osm.pbf` and `.poly` files
145- **[OpenStreetMap](https://planet.openstreetmap.org)** — planet PBF source for world basemap tiles