wisp.place-monorepo
Monorepo for wisp.place. A static site hosting service built on top of the AT Protocol. wisp.place
Star 104
Fork 7
Watch

perf(tiered-storage): promote streamed reads into warm and hot #87

Merged nekomimi.pet
targeting main from perf/stream-promotion
Merged opened by and assigned to nekomimi.pet
targeting main from perf/stream-promotion

Problem #

Hosting serves almost every asset through TieredStorage.getStream (apps/hosting-service/src/lib/file-serving.ts:526). On main, getStream never promotes a warm or cold hit. Its doc comment says so (packages/@wispplace/tiered-storage/src/TieredStorage.ts:462-464), and the loop at :493-536 returns the tier's stream as is. A cold asset is therefore fetched from S3 on every request until a buffered get() of the same key happens to promote it. Every cold read also fires updateAccessStats(key, 'cold') (:383 buffered, :511 streamed; body :1466-1478). That is getMetadata, an S3 HeadObject, followed by setMetadata. In hosting that call is a no-op, because ReadOnlyS3Tier.setMetadata is one (apps/hosting-service/src/lib/storage.ts:573). Everywhere else it is a CopyObject with MetadataDirective: REPLACE (S3StorageTier.ts:436-447). It has no fence, so it can write the metadata it just read onto an object that a concurrent write replaced in the meantime.

Scope: in hosting, every public-site read that misses hot and warm. That is every non-index asset, because only promotionStrategy: 'eager' buffered reads ever filled warm. Background: rust-rewrite plan Phase 0 #5 and hosting-contract §5.4/§11.

Solution #

  • Streamed promotion (TieredStorage.getStream → new promoteStream), only with promotionStrategy: 'eager', as buffered promotion already works. Hosting's private storage is lazy and cold-only, so nothing changes there.
    • A cold hit goes through a Transform tee. Each chunk is written to a staged write in the warm tier before it is passed on, so memory stays bounded by the normal stream buffers and nothing is accumulated. When the source ends, the staged file is renamed into place.
    • Hot gets a copy only if the key's placement rules include hot and the object is ≤ streamHotPromotionMaxBytes (new config, default 256 KiB). The tee drops its hot buffer as soon as the object exceeds that, and MemoryStorageTier still enforces its own byte cap. A streamed warm hit promotes to hot under the same rule.
    • Commit goes through the existing promoteIfCurrent. That means the key mutation queue, the generation fence and the upper-tier epoch, all checked while the read's fence is still held, before the caller's stream ends. A write or invalidation that started after the read was opened therefore aborts the staged write and drops the hot copy.
    • Nothing is promoted, and the staged file is deleted, when the stream is destroyed (client abort, HEAD/304/range disposal), when the source errors, or when an uncompressed source ends with fewer bytes than metadata.size.
    • Concurrent streamed reads of one key stage one copy, not N.
  • StorageTier.stageWrite? is new and optional, implemented by DiskStorageTier. It writes a 0600 file under <cache>/.staging/, and commit() publishes it inside withMutation through the same eviction/index path as set(). That path is the extracted storeUnlocked, which setUnlocked now uses too; behaviour is the same, with the write step switched between writeFile and rename. .staging is skipped by the index rebuild, listKeys and getStats the same way .invalidated already was, and it is emptied on startup, so a crash leaves nothing behind.
  • Cold access stats removed from both read paths. The cold tier is the source of truth, and the update cost a HEAD on every cold read (plus a full-object copy outside hosting) for numbers nothing in the apps reads. Hot and warm stats are unchanged; warm LRU still uses them.

Alternatives I rejected:

  • warm.setStream(key, tee): it holds the disk tier's mutation lock for as long as the client takes to read, and it writes metadata before the data, so a partial file is visible.
  • Buffering the whole body and then calling set(): unbounded memory for large media.
  • Draining a disposed stream to finish the promotion: it changes hosting's HEAD/304/range behaviour and costs bandwidth. Left as a possible follow-up.

Before / After #

Method: packages/@wispplace/tiered-storage/scripts/bench-stream-promotion.ts (committed). It builds hosting's layout: MemoryStorageTier hot (100 MiB / 500 items), DiskStorageTier warm in a temp dir, and a counting cold tier that stands in for S3. The cold tier adds 25 ms per call and generates its bytes on demand, so the source never holds the object. Placement is the same shape as hosting's: **/*.{css,js} → hot+warm+cold, everything else → warm+cold. Eager promotion, no compression. The script makes 20 sequential getStream reads of a 40 KiB .css and of a 2 MiB .png, each fully drained. It counts cold GET (getStream/get/getWithMetadata), HEAD (getMetadata) and metadata writes, and records the serving tier and the latency to drain each read. It also does one 256 MiB .mp4 cold read with RSS sampled every 2 ms (after Bun.gc(true)), and reports which tier serves the next read.

Rerun:

# after (this branch)
cd packages/@wispplace/tiered-storage && bun scripts/bench-stream-promotion.ts            # --large-mib 1024 for the 1 GiB case
# before: same script against main's src
git worktree add --detach /tmp/base origin/main && mkdir -p /tmp/base/packages/@wispplace/tiered-storage/scripts \
  && cp scripts/bench-stream-promotion.ts /tmp/base/packages/@wispplace/tiered-storage/scripts/ \
  && (cd /tmp/base && bun install) && cd /tmp/base/packages/@wispplace/tiered-storage && bun scripts/bench-stream-promotion.ts

These are 3 runs each (ranges where they varied), Bun 1.3.14, macOS arm64, local disk.

Before (main 8324c85) #

object, 20 reads cold GET cold HEAD cold metadata writes served from 1st read median later read total
app.css 40 KiB 20 20 20 cold ×20 25.4-27.3 ms 26.5-27.2 ms 532 ms
photo.png 2 MiB 20 20 20 cold ×20 26.2-27.3 ms 27.3 ms 537-540 ms

Large object: a 256 MiB cold stream peaks at +33.5-33.7 MiB RSS, and the next read is cold again. At 1 GiB: +48.7 MiB, next read cold.

After (this branch, 402c023) #

object, 20 reads cold GET cold HEAD cold metadata writes served from 1st read median later read total
app.css 40 KiB 1 0 0 cold ×1, hot ×19 27.3-28.4 ms 0.12-0.25 ms 29.5-32 ms
photo.png 2 MiB 1 0 0 cold ×1, warm ×19 27.4-29.4 ms 1.1-1.2 ms 48.5-53.9 ms

Large object: a 256 MiB cold stream peaks at +44-45.8 MiB RSS and is then served from warm. At 1 GiB: +51.9 MiB (before: +48.7), then warm. Growth stays flat as the object goes from 256 MiB to 1 GiB, so it is not buffered. The ~10 MiB over main at 256 MiB is file-write buffers and Bun heap noise, not the object.

The first, promoting read costs about +1-2 ms: opening the staged file, plus the rename and metadata write before the stream ends.

Tests #

  • New test/TieredStorage.stream-promotion.test.ts, 15 tests. On main, 9 fail (promotion, no cold HEAD/copy, hot cap/placement, warm→hot, single staging, DiskStorageTier staging). The other 6 are safety guards that pass on main only because it never promotes: abort, source error, short source, invalidation mid-stream, write mid-stream, lazy. I checked that they guard the new code by disabling the generation check in promoteIfCurrent: the invalidation and write tests then fail.
  • bun test --isolate: tiered-storage 189 pass / 0 fail; hosting-service 345 pass / 3 skip / 0 fail; firehose-service 404 pass / 83 skip / 0 fail; main-app 303 pass / 1 skip / 0 fail.
  • tsc --noEmit in the package is clean. The root bun run check reports only blog/src/pages/rss.xml.ts: Cannot find module 'astro', which main reports too (blog deps aren't installed in a worktree). biome check on the changed files is clean.

Not measured yet #

  • Production effect (S3 GET/HEAD rate, X-Cache-Tier mix, p95 latency, warm disk growth) needs staging and caddy/S3 metrics; staging is down until valefar's pool is back. The bench uses a synthetic 25 ms cold tier, not R2/S3.
  • Requests that dispose the stream without reading it to the end (HEAD, 304, ranges that stop early) still open a cold stream and don't promote. They are served from warm once any full GET of the key has promoted it.
  • Warm now grows with every object streamed in full, up to WARM_CACHE_SIZE and the existing eviction. Before, only buffered reads (mostly HTML) filled it. Disk usage on the edges should be watched after deploy.
  • Cold lastAccessed/accessCount stop changing on reads. Only bootstrapWarm({ sinceDate }) and bootstrapHot read them, and nothing in the apps calls either.
Commits 0