···11# Data
2233-This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. Output tiles are written to `www/static/tiles/`.
33+This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. Output tiles are written to `www/static/tiles/`. We don't really need suuuuuuper up-to-date maps for this, since we trim a lot of data; the tiles are mainly used as a way to orient the user as they create their own map.
4455All tile generation is done via the CLI:
66···1616- [`tilemaker`](https://github.com/systemed/tilemaker) — PBF → PMTiles conversion (`build`, `build:world`)
1717- [`osmium`](https://osmcode.org/osmium-tool/) — PBF extraction (`extract` command)
18181919+The following resources are necessary to generate pmtiles:
2020+- [Download a planet `.osm.pbf`](https://wiki.openstreetmap.org/wiki/Planet.osm) for the world-level pmtiles (put it in `./osm/planet-latest.osm.pbf`). For me, using a torrentfile has been the most stable way.
2121+- Download regional .osm.pbf files for regions if you don't want to self-extract (can download with `download:osm`)
2222+- Coastlines WGS84 projection from https://osmdata.openstreetmap.de/data/coastlines.html (put it in `./cli/shared/tilemaker/coastline`) (todo for me: create a `download:coastlines` cli cmd)
2323+1924## CLI Commands
20252121-| Command | Description |
2222-| ------------------------------------------ | ------------------------------------------------- |
2323-| `list [--search <term>]` | Browse available region slugs |
2424-| `download:osm <region> [--force]` | Download `.osm.pbf` from Geofabrik |
2525-| `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s) |
2626-| `extract <region> --from <source>` | Carve a sub-region from a larger PBF using osmium |
2727-| `build <region>` | Convert `.osm.pbf` → `.pmtiles` using tilemaker |
2828-| `build:world` | Build world-scale basemap tiles |
2929-| `clean --region <slug> \| --all` | Remove intermediate files |
2626+| Command | Description |
2727+| ------------------------------------------ | ------------------------------------------------------------------ |
2828+| `list [--search <term>]` | Browse available region slugs |
2929+| `download:osm <region> [--force]` | Download `.osm.pbf` from Geofabrik |
3030+| `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s) |
3131+| `extract <region> --from <source>` | Carve a sub-region from a larger PBF using osmium |
3232+| `build <region>` | Convert `.osm.pbf` → `.pmtiles` using tilemaker |
3333+| `trim:world` | Strip planet PBF to only data needed for world tiles (saves memory)|
3434+| `build:world` | Build world-scale basemap tiles |
3535+| `clean --region <slug> \| --all` | Remove intermediate files |
30363137## World Tiles
32383339The world basemap is built from OSM data using a tilemaker pipeline, sourced from a full planet PBF. Place `planet-latest.osm.pbf` in `data/osm/` and run:
34403541```sh
4242+deno task data trim:world # recommended first step — see below
3643deno task data build:world [--maxzoom <5|7|9>] # default maxzoom: 7
3744```
3845···4249- `data/cli/shared/tilemaker/process.world.lua` — feature processing logic
43504451Note: this takes a super long time to run.
5252+5353+### Trimming the planet file (recommended)
5454+5555+The full planet PBF (~75 GB) contains a huge amount of data that is irrelevant at world zoom levels — buildings, addresses, shop/amenity POIs, minor roads, etc. Running `trim:world` uses `osmium tags-filter` to produce a much smaller `planet-trimmed.osm.pbf` containing only the tags that `process.world.lua` actually reads:
5656+5757+- Place nodes (countries, states, cities, towns, villages)
5858+- Natural peaks and volcanoes
5959+- Administrative boundaries
6060+- Major roads (motorway → secondary) and ferry routes
6161+- Main-line railways
6262+- Rivers, waterways, water polygons
6363+- Landcover and landuse polygons
6464+- Parks and nature reserves
6565+6666+```sh
6767+deno task data trim:world # creates data/osm/planet-trimmed.osm.pbf
6868+deno task data trim:world --overwrite # re-run and replace an existing trimmed file
6969+```
7070+7171+When `planet-trimmed.osm.pbf` is present, `build:world` will automatically use it instead of the full planet file. This significantly reduces tilemaker's peak memory usage and overall build time.
45724673## Regional Tiles
4774
+20-3
data/cli/commands/build-world.ts
···1010import { run } from '../shared/run.ts'
11111212const PLANET_PBF = join(PBF_DIR, 'planet-latest.osm.pbf')
1313+const PLANET_TRIMMED_PBF = join(PBF_DIR, 'planet-trimmed.osm.pbf')
13141415export const buildWorldCmd = new Command()
1516 .name('build:world')
···2122 default: 7,
2223 })
2324 .action(async ({ maxzoom }) => {
2424- if (!(await Deno.stat(PLANET_PBF).catch(() => null))) {
2525- throw new Error(`Planet PBF not found: ${PLANET_PBF}`)
2525+ const hasTrimmed = !!(await Deno.stat(PLANET_TRIMMED_PBF).catch(() => null))
2626+ const hasPlanet = !!(await Deno.stat(PLANET_PBF).catch(() => null))
2727+2828+ if (!hasTrimmed && !hasPlanet) {
2929+ throw new Error(
3030+ `Planet PBF not found: ${PLANET_PBF}\n` +
3131+ `Run 'deno task data trim:world' first to create a trimmed copy, or place the full planet file at that path.`,
3232+ )
3333+ }
3434+3535+ const inputPbf = hasTrimmed ? PLANET_TRIMMED_PBF : PLANET_PBF
3636+ if (hasTrimmed) {
3737+ console.log(`Using trimmed planet file: ${PLANET_TRIMMED_PBF}`)
3838+ } else {
3939+ console.log(
4040+ `No trimmed file found — using full planet: ${PLANET_PBF}\n` +
4141+ `Tip: run 'deno task data trim:world' first to reduce memory usage.`,
4242+ )
2643 }
27442845 await ensureDir(join(TILES_OUT_DIR, 'world'))
···4663 try {
4764 await run('tilemaker', [
4865 '--input',
4949- PLANET_PBF,
6666+ inputPbf,
5067 '--output',
5168 outPath,
5269 '--config',
+196
data/cli/commands/trim-world.ts
···11+import { Command } from '@cliffy/command'
22+import { join } from '@std/path'
33+import { PBF_DIR } from '../shared/paths.ts'
44+import { run } from '../shared/run.ts'
55+66+const PLANET_PBF = join(PBF_DIR, 'planet-latest.osm.pbf')
77+const TRIMMED_PBF = join(PBF_DIR, 'planet-trimmed.osm.pbf')
88+99+/**
1010+ * osmium tags-filter expressions covering every tag read by process.world.lua.
1111+ *
1212+ * Format: <object-type>/<key>=<value> or <object-type>/<key>
1313+ * n = node, w = way, r = relation, a = area (way or relation)
1414+ *
1515+ * We intentionally omit buildings, addresses, shop/amenity POIs, minor roads,
1616+ * and anything else that only appears at z12+ in the regional tile pipeline.
1717+ */
1818+const FILTER_EXPRESSIONS = `
1919+# ── Nodes ──────────────────────────────────────────────────────────────────
2020+# place nodes (countries, states, cities, towns, villages)
2121+n/place=continent
2222+n/place=country
2323+n/place=state
2424+n/place=province
2525+n/place=city
2626+n/place=town
2727+n/place=village
2828+2929+# natural peaks & volcanoes
3030+n/natural=peak
3131+n/natural=volcano
3232+3333+# ── Ways ───────────────────────────────────────────────────────────────────
3434+# Administrative boundaries
3535+w/boundary=administrative
3636+w/boundary=national_park
3737+3838+# Major roads only (motorway, trunk, primary, secondary + their links)
3939+w/highway=motorway
4040+w/highway=motorway_link
4141+w/highway=trunk
4242+w/highway=trunk_link
4343+w/highway=primary
4444+w/highway=primary_link
4545+w/highway=secondary
4646+w/highway=secondary_link
4747+4848+# Ferry routes
4949+w/route=ferry
5050+5151+# Main-line railways
5252+w/railway=rail
5353+w/railway=narrow_gauge
5454+w/railway=preserved
5555+w/railway=funicular
5656+5757+# Rivers & named waterways
5858+w/waterway=river
5959+w/waterway=canal
6060+6161+# Water polygons
6262+w/natural=water
6363+w/water
6464+6565+# Landcover
6666+w/natural=wood
6767+w/natural=wetland
6868+w/natural=beach
6969+w/natural=sand
7070+w/natural=dune
7171+w/natural=glacier
7272+w/natural=ice_shelf
7373+w/natural=bare_rock
7474+w/natural=scree
7575+w/natural=fell
7676+w/natural=grassland
7777+w/natural=grass
7878+w/natural=heath
7979+w/natural=meadow
8080+w/natural=scrub
8181+w/natural=shrubbery
8282+w/natural=tundra
8383+w/natural=bay
8484+8585+# Landuse (landcover-mapped keys)
8686+w/landuse=forest
8787+w/landuse=farmland
8888+w/landuse=farm
8989+w/landuse=orchard
9090+w/landuse=vineyard
9191+w/landuse=allotments
9292+w/landuse=village_green
9393+w/landuse=recreation_ground
9494+9595+# Landuse (landuse layer keys)
9696+w/landuse=school
9797+w/landuse=university
9898+w/landuse=hospital
9999+w/landuse=railway
100100+w/landuse=cemetery
101101+w/landuse=military
102102+w/landuse=residential
103103+w/landuse=commercial
104104+w/landuse=industrial
105105+w/landuse=retail
106106+w/landuse=stadium
107107+w/landuse=pitch
108108+109109+# Leisure (landcover + park)
110110+w/leisure=park
111111+w/leisure=garden
112112+w/leisure=nature_reserve
113113+w/leisure=stadium
114114+w/leisure=pitch
115115+116116+# Parks / nature reserves via boundary
117117+w/boundary=national_park
118118+w/leisure=nature_reserve
119119+120120+# Place islands (rendered as centroids)
121121+w/place=island
122122+123123+# ── Relations ──────────────────────────────────────────────────────────────
124124+# Administrative boundaries (required for boundary layer)
125125+r/type=boundary
126126+r/boundary=administrative
127127+r/boundary=national_park
128128+r/leisure=nature_reserve
129129+`.trim()
130130+131131+export const trimWorldCmd = new Command()
132132+ .name('trim:world')
133133+ .description(
134134+ 'Trim planet-latest.osm.pbf to only the data needed for world basemap tiles ' +
135135+ '(removes buildings, POIs, minor roads, addresses, etc.). ' +
136136+ `Output: ${TRIMMED_PBF}`,
137137+ )
138138+ .option('--overwrite', 'Overwrite existing trimmed PBF if it exists.', {
139139+ default: false,
140140+ })
141141+ .action(async ({ overwrite }) => {
142142+ // Check source file exists
143143+ if (!(await Deno.stat(PLANET_PBF).catch(() => null))) {
144144+ throw new Error(
145145+ `Planet PBF not found: ${PLANET_PBF}\n` +
146146+ `Download it from https://planet.openstreetmap.org and place it at that path.`,
147147+ )
148148+ }
149149+150150+ // Check output doesn't already exist unless --overwrite
151151+ if (!overwrite && (await Deno.stat(TRIMMED_PBF).catch(() => null))) {
152152+ console.log(`Trimmed PBF already exists: ${TRIMMED_PBF}`)
153153+ console.log(`Use --overwrite to replace it.`)
154154+ Deno.exit(0)
155155+ }
156156+157157+ // Write expressions to a temp file so we don't hit shell argument limits
158158+ // and so the list stays readable.
159159+ const tmpExpressions = join(PBF_DIR, '.trim-world-expressions.tmp')
160160+ await Deno.writeTextFile(tmpExpressions, FILTER_EXPRESSIONS)
161161+162162+ const { size: inputSize } = await Deno.stat(PLANET_PBF)
163163+ const inputSizeStr = (inputSize / 1024 ** 3).toFixed(1)
164164+ console.log(`Input: ${PLANET_PBF} (${inputSizeStr} GB)`)
165165+ console.log(`Output: ${TRIMMED_PBF}`)
166166+ console.log(`Running osmium tags-filter — this will take a while...`)
167167+168168+ try {
169169+ await run('osmium', [
170170+ 'tags-filter',
171171+ '--progress',
172172+ '--expressions',
173173+ tmpExpressions,
174174+ '--output',
175175+ TRIMMED_PBF,
176176+ ...(overwrite ? ['--overwrite'] : []),
177177+ PLANET_PBF,
178178+ ])
179179+ } finally {
180180+ await Deno.remove(tmpExpressions).catch(() => {})
181181+ }
182182+183183+ const { size: outputSize } = await Deno.stat(TRIMMED_PBF)
184184+ const outputSizeStr = outputSize < 1024 ** 3
185185+ ? `${Math.round(outputSize / 1024 ** 2)} MB`
186186+ : `${(outputSize / 1024 ** 3).toFixed(1)} GB`
187187+ const reduction = (((inputSize - outputSize) / inputSize) * 100).toFixed(0)
188188+189189+ console.log(`\nDone!`)
190190+ console.log(`Output: ${TRIMMED_PBF} (${outputSizeStr})`)
191191+ console.log(`Reduced planet by ~${reduction}%.`)
192192+ console.log(
193193+ `\nNext step: deno task data build:world\n` +
194194+ `(build:world will automatically use the trimmed file when present)`,
195195+ )
196196+ })
+2
data/cli/main.ts
···2121import { extractCmd } from './commands/extract.ts'
2222import { buildCmd } from './commands/build.ts'
2323import { buildWorldCmd } from './commands/build-world.ts'
2424+import { trimWorldCmd } from './commands/trim-world.ts'
2425import { cleanCmd } from './commands/clean.ts'
25262627await new Command()
···3637 .command('extract', extractCmd)
3738 .command('build', buildCmd)
3839 .command('build:world', buildWorldCmd)
4040+ .command('trim:world', trimWorldCmd)
3941 .command('clean', cleanCmd)
4042 .parse(Deno.args)