Offline-capable geomap, meant for storing location bookmarks
chore: update READMEs
+88
-135
+31
-54
README.md
+31
-54
README.md
···
1
1
# maps.bpev.me
2
2
3
-
This is a proof-of-concept for building offline maps as a PWA, using maplibregl and protomaps.
3
+
MapsApp is an offline-capable geomap, meant for storing location bookmarks.
4
4
5
-
This is meant to prove the dataflow for building something like Organic Maps:
5
+
It is a PWA built on [MapLibre GL](https://maplibre.org) and [PMTiles](https://protomaps.github.io), and serves as a proof-of-concept for the data flow behind something like [Organic Maps](https://organicmaps.app). A world basemap is fetched on first load, users can download regional tiles on demand with additional detail, and everything renders offline after that. Bookmarks, collections, and search history are stored locally.
6
6
7
-
- A world basemap is downloaded on app load/install
8
-
- User can download regional maps via the app
9
-
- Downloaded maps are displayed offline, and overlay the world basemap
7
+
## Technologies
10
8
11
-
## Technologies for App
9
+
**App**
12
10
13
-
- **[civility](https://jsr.io/@civility)**: PWA build system and service worker framework used to bundle the app and manage offline caching
14
-
- **[pmtiles](https://protomaps.github.io)**: I settled on pmtiles because the ability to run maplibregl directly from static files simplifies the process on the client
15
-
- **[maplibregl](https://maplibre.org)**: a powerful solution that has a defined happy path with pmtiles
16
-
- **[deno](https://deno.com)**: I'm just using this for my build system for my other projects, so used it here for my convenience. Could probably make it work in vanilla or a node build system without too much effort.
11
+
- **[maplibre-gl](https://maplibre.org)** — vector tile rendering engine
12
+
- **[pmtiles](https://protomaps.github.io)** — single-file tile archive format; enables serving maps from static files with no tile server
13
+
- **[pmtiles-offline](https://github.com/jtbaker/pmtiles-offline)** — utility for serving pmtiles from local data
14
+
- **[nominatim](https://nominatim.org)** - api for geocoding
15
+
- **[civility](https://jsr.io/@civility)** — PWA build system, service worker framework, router, and state management
16
+
- **[lit](https://lit.dev)** — reactive web components for route views
17
+
- **[deno](https://deno.com)** — runtime and toolchain
17
18
18
-
### Technologies for Tile Creation
19
+
**Tile creation** — see [data/README.md](data/README.md) for the full pipeline.
19
20
20
-
- **[geofabrik](https://download.geofabrik.de)**: I don't really want to deal with defining regions, so geofabrik seems like a natural choice; I can use their `oms.pbf` and `.poly` files to help define my download regions in the future
21
-
- [**NaturalEarthData**](https://www.naturalearthdata.com): Very fast way to create planet-scale basemaps. I think for greater control, however, I might prefer to build directly from planet.osm for a full product.
22
-
- [**ogr2ogr**](https://gdal.org/en/stable/programs/ogr2ogr.html): Basically used just as a way to move from Natural Earth Data to data consumable by tippecanoe
23
-
- [**tippecanoe**](https://github.com/felt/tippecanoe): Their docs are [basically the process I used](https://github.com/felt/tippecanoe?tab=readme-ov-file#show-countries-at-low-zoom-levels-but-states-at-higher-zoom-levels) for the basemap tile creation, and I wish I found it way earlier, because it was a pain in the ass without it.
24
-
- [**tilemaker**](http://tilemaker.org): Fills similar space as tippecanoe. Seems like it might be easier to deal with filters using it. But using either one of this or tippecanoe would probably be good enough.
21
+
## Development
25
22
26
-
## Usage
27
-
28
-
The only prerequisite is to have [deno](https://deno.com) installed.
23
+
Running the app requires [Deno](https://deno.com) and [Civility](https://jsr.io/@civility/cli):
29
24
30
25
```sh
31
-
# install deno
32
26
curl -fsSL https://deno.land/install.sh | sh
33
-
34
-
# Run App
35
-
deno task dev
27
+
deno install -gA jsr:@civility/cli --name civ
36
28
```
37
29
38
-
### Things you'd probably want to integrate if taking it further:
30
+
Before running the app, you need tiles, built via the [CLI](data/README.md):
39
31
40
-
- More of an automated way of creating region tiles
41
-
- Scrolling on the map on some zoom levels will expose download links for viewable regions
42
-
- Reading user position could also be used to dynamically load maps a user has already downloaded
43
-
- Report download progress
44
-
- Make sure that region tiles overlay basemap in such a way that the janky guesstimations aren't seen by the user
45
-
- The regions on geofabrik are quite heavy; might want to either trim the data, or make even more local maps available.
32
+
```sh
33
+
deno task data list
34
+
# First, download or extract osm files to /data/osm
35
+
deno task data download:osm europe/spain
36
+
deno task data extract europe/spain --from planet
46
37
47
-
## Building the Regional Tiles
48
-
49
-
I generated the local regional tiles via:
50
-
51
-
- download `.osm.pbf` from geofabrik
52
-
- `tilemaker thailand-latest.osm.pbf thailand.pmtiles`, using `config-openmaptiles.json` and `process-openmaptiles.lua` (I cloned these into scripts/tilemaker)
53
-
54
-
## Building the Basemap Tiles
55
-
56
-
Download data the following data from https://www.naturalearthdata.com
38
+
# Then build pmtiles
39
+
deno task data build europe/spain
40
+
```
57
41
58
-
- ne_10m_admin_0_boundary_lines_land
59
-
- ne_10m_admin_0_countries
42
+
The you can then serve or build the app with Civility CLI:
60
43
61
44
```sh
62
-
# Convert the shp files into geojson
63
-
# note: you do need the other files to exist alongside the shp files
64
-
ogr2ogr -f GeoJSON boundaries.geojson ne_10m_admin_0_boundary_lines_land.shp
65
-
ogr2ogr -f GeoJSON countries.geojson ne_10m_admin_0_countries.shp
45
+
civ start # Start a server with hot reload
46
+
civ build # Builds static files to serve from /www
47
+
```
66
48
67
-
# process the geojson into mbtiles via tippecanoe
68
-
tippecanoe -Z4 -zg -o boundaries.mbtiles --coalesce-densest-as-needed --extend-zooms-if-still-dropping boundaries.geojson
69
-
tippecanoe -z3 -zg -o countries.mbtiles --coalesce-densest-as-needed countries.geojson
49
+
**Lint and type-check:**
70
50
71
-
# Join together the tiles into world tiles
72
-
tile-join -o world.mbtiles countries.mbtiles boundaries.mbtiles
73
-
74
-
#Convert the world file into pmtiles
75
-
pmtiles convert world.mbtiles world.pmtiles
51
+
```sh
52
+
deno task test
76
53
```
+57
-81
data/README.md
+57
-81
data/README.md
···
1
1
# Data
2
2
3
-
This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. The output tiles live in `www/static/tiles/`.
3
+
This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. Output tiles are written to `www/static/tiles/`.
4
4
5
5
All tile generation is done via the CLI:
6
6
···
9
9
deno task data --help
10
10
```
11
11
12
-
## Tilemaking Pipeline
12
+
## Dependencies
13
+
14
+
The following tools must be installed and on `PATH`:
15
+
16
+
- [`tilemaker`](https://github.com/systemed/tilemaker) — PBF → PMTiles conversion (`build`, `build:world`)
17
+
- [`osmium`](https://osmcode.org/osmium-tool/) — PBF extraction (`extract` command)
18
+
19
+
## CLI Commands
20
+
21
+
| Command | Description |
22
+
| ------------------------------------------ | ------------------------------------------------- |
23
+
| `list [--search <term>]` | Browse available region slugs |
24
+
| `download:osm <region> [--force]` | Download `.osm.pbf` from Geofabrik |
25
+
| `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s) |
26
+
| `extract <region> --from <source>` | Carve a sub-region from a larger PBF using osmium |
27
+
| `build <region>` | Convert `.osm.pbf` → `.pmtiles` using tilemaker |
28
+
| `build:world` | Build world-scale basemap tiles |
29
+
| `clean --region <slug> \| --all` | Remove intermediate files |
30
+
31
+
## World Tiles
32
+
33
+
The world basemap is built from OSM data using a tilemaker pipeline, sourced from a full planet PBF. Place `planet-latest.osm.pbf` in `data/osm/` and run:
34
+
35
+
```sh
36
+
deno task data build:world [--maxzoom <5|7|9>] # default maxzoom: 7
37
+
```
38
+
39
+
Output is written to `www/static/tiles/world_z<maxzoom>.pmtiles`. The config and Lua script used are separate from the regional ones:
40
+
41
+
- `data/cli/shared/tilemaker/config.world.json` — layer definitions for the world overview
42
+
- `data/cli/shared/tilemaker/process.world.lua` — feature processing logic
43
+
44
+
Note: this takes a super long time to run.
45
+
46
+
## Regional Tiles
13
47
14
-
Tiles are generated from [OpenStreetMap](https://www.openstreetmap.org/) data. The pipeline has three stages:
48
+
Regional tiles follow a three-stage pipeline:
15
49
16
50
```
17
-
Geofabrik (download) → osmium (extract) → tilemaker (build) → .pmtiles
51
+
Geofabrik (download) → osmium (extract; optional) → tilemaker (build) → .pmtiles
18
52
```
19
53
20
54
### 1. Download
21
55
22
-
Regional `.osm.pbf` data files are sourced from [Geofabrik](https://download.geofabrik.de/). All available regions are listed in [`../data/geofabrik/regions.txt`](../data/geofabrik/regions.txt).
56
+
Regional `.osm.pbf` data and `.poly` boundary files are sourced from [Geofabrik](https://download.geofabrik.de/). Region slugs match Geofabrik's hierarchy (e.g. `asia/japan/kansai`, `europe/monaco`).
23
57
24
-
`.poly` boundary files for all regions are already committed to `data/poly/` and do not need to be downloaded. If you need to refresh them, use:
58
+
`.poly` files for all 499 Geofabrik regions are already committed to `data/poly/` and don't need to be refreshed under normal circumstances. OSM `.pbf` files are gitignored due to size and must be downloaded as needed.
25
59
26
60
```sh
27
-
deno task data download:osm europe/monaco # OSM map data
28
-
deno task data download:poly europe/monaco # boundary polygon (already present)
61
+
deno task data download:osm europe/monaco
62
+
deno task data download:poly europe/monaco # already present; only needed to refresh
29
63
```
30
-
31
-
Downloaded OSM files are saved to `data/osm/<region>-latest.osm.pbf` (gitignored due to size).
32
64
33
65
### 2. Extract (optional)
34
66
35
-
If you already have a large regional PBF (e.g. an entire continent), you can carve out a smaller region using `osmium` rather than downloading from Geofabrik again:
67
+
If you already have a large regional PBF, you can carve out a sub-region using `osmium` rather than downloading from Geofabrik again:
36
68
37
69
```sh
38
70
deno task data extract asia/taiwan --from asia
39
71
```
40
72
41
-
This requires both the source PBF and the target `.poly` file to already be present.
73
+
This requires both the source PBF and the target `.poly` file to be present.
42
74
43
75
### 3. Build
44
76
45
-
Convert the `.osm.pbf` to a `.pmtiles` file using `tilemaker`:
77
+
Convert `.osm.pbf` to `.pmtiles` using `tilemaker`:
46
78
47
79
```sh
48
80
deno task data build europe/monaco
49
81
```
50
82
51
-
Output is written to `www/static/tiles/<region-name>.pmtiles`. The tilemaker config and processing script used are:
83
+
Output is written to `www/static/tiles/<region-name>.pmtiles`. The tilemaker config and Lua processing script are:
52
84
53
-
- [`../data/tilemaker/config.json`](../data/tilemaker/config.json) — layer definitions (OpenMapTiles schema, zoom 0–14)
54
-
- [`../data/tilemaker/process.lua`](../data/tilemaker/process.lua) — feature processing logic
85
+
- `data/cli/shared/tilemaker/config.json` — layer definitions (OpenMapTiles schema, zoom 0–14)
86
+
- `data/cli/shared/tilemaker/process.lua` — feature processing logic
55
87
56
88
## Common Workflows
57
89
58
-
### Small country (direct download)
90
+
**Small country — direct download:**
59
91
60
92
```sh
61
93
deno task data download:osm europe/monaco
62
94
deno task data build europe/monaco
63
95
```
64
96
65
-
### Country carved from a larger extract
97
+
**Country carved from a larger extract:**
66
98
67
99
```sh
68
-
deno task data download:osm asia # large, but reusable for multiple countries
100
+
deno task data download:osm asia # large, but reusable across multiple countries
69
101
deno task data extract asia/taiwan --from asia
70
102
deno task data build asia/taiwan
71
103
```
72
104
73
-
## CLI Commands
74
-
75
-
| Command | Description |
76
-
| ------------------------------------------ | ------------------------------------ |
77
-
| `list [--search <term>]` | Browse available region slugs |
78
-
| `download:osm <region> [--force]` | Download `.osm.pbf` from Geofabrik |
79
-
| `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s) |
80
-
| `extract <region> --from <source>` | Carve a sub-region from a larger PBF |
81
-
| `build <region>` | Run tilemaker → `.pmtiles` |
82
-
| `clean --region <slug> \| --all` | Remove intermediate files |
83
-
84
-
## Dependencies
85
-
86
-
The following tools must be installed and available on `PATH`:
87
-
88
-
- [`osmium`](https://osmcode.org/osmium-tool/) — OSM data processing (`extract` command)
89
-
- [`tilemaker`](https://github.com/systemed/tilemaker) — PBF to vector tile conversion (`build` command)
90
-
91
105
## Directory Structure
92
106
93
107
```
94
108
data/
95
-
osm/ Downloaded/extracted .osm.pbf files (gitignored — large)
96
-
asia/
97
-
taiwan-latest.osm.pbf
98
-
europe/
99
-
monaco-latest.osm.pbf
100
-
poly/ Boundary polygon files (committed)
101
-
asia/
102
-
taiwan.poly
103
-
europe/
104
-
monaco.poly
105
-
cli/ The CLI source code
109
+
osm/ Downloaded/extracted .osm.pbf files (gitignored)
110
+
poly/ Boundary .poly files for all 499 Geofabrik regions (committed)
111
+
cli/ CLI source code
106
112
```
107
113
108
-
`data/osm/` is gitignored due to file size. Use `deno task data download:osm` to fetch OSM data as needed. `data/poly/` is committed — all 499 Geofabrik regions are already present.
114
+
## External Resources
109
115
110
-
## Resources
111
-
112
-
### Shapefiles
113
-
114
-
- https://osmdata.openstreetmap.de/data/water-polygons.html
115
-
116
-
### geofabrik
117
-
118
-
Basically, to fully realize a project like this, we will need scripts to organize the different downloadable regions, and also to find the urls for us to download. This doesn't include region naming/localization, so maybe need to figure that out later.
119
-
120
-
The files we need are:
121
-
122
-
1. `[insert-region]-latest.osm.pbf`: map data
123
-
2. `[insert-region].poly`: the borders of each region
124
-
125
-
### tilemaker
126
-
127
-
We want tilemaker to create tiles that are compatible with our protomaps styles. Basically, you just need these files in the directory when you need to process your pbf:
128
-
129
-
```
130
-
tilemaker monaco-latest.osm.pbf monaco.pmtiles
131
-
```
132
-
133
-
# World Tiles
134
-
135
-
Three output tiers, all generated from the same filtered planet PBF:
136
-
137
-
| File | Zoom | Approx size | What's in it |
138
-
| ---------------- | ----- | ----------- | --------------------------------------------- |
139
-
| world_z5.pmtiles | z0–z5 | ~5–10 MB | Ocean, country fills, country names, capitals |
140
-
| world_z7.pmtiles | z0–z7 | ~25–50 MB | + major cities, state boundaries, main rivers |
141
-
| world_z9.pmtiles | z0–z9 | ~100–180 MB | + towns, all highways, detailed water |
116
+
- **[Geofabrik](https://download.geofabrik.de)** — regional OSM extracts; provides both `.osm.pbf` and `.poly` files
117
+
- **[OpenStreetMap](https://planet.openstreetmap.org)** — planet PBF source for world basemap tiles