ATProto Sync Implementation Progress#

Completed Work ✅#

1. Unified Media Module (media-atproto.mjs)#

Status: ✅ Complete and tested

Features:

Single source of truth for all ATProto media operations
Support for 5 media types: paintings, moods, pieces, kidlisp, tapes
Proper lexicon implementation with full schemas
Blob generation for paintings (thumbnails via /api/pixel)
Recording detection for paintings (checks ZIP existence)
Guest/anonymous user support

Lexicons Implemented:

✅ Paintings: computer.aesthetic.painting
- Fields: slug, code, imageUrl, acUrl, when, ref
- Blob: 512x512 thumbnail (via /api/pixel :contain mode)
- Optional: recordingUrl (if ZIP exists)
✅ Moods: computer.aesthetic.mood
- Fields: text, when, ref
- No blob, no anonymous support
✅ Pieces: computer.aesthetic.piece
- Fields: slug, when, ref
- No blob, supports anonymous
✅ Kidlisp: computer.aesthetic.kidlisp
- Fields: code, source, acUrl, when, ref
- No blob, supports anonymous
✅ Tapes: computer.aesthetic.tape
- Fields: slug, code, zipUrl, acUrl, when, ref
- Blob: MP4 video (TODO: conversion not implemented yet)
- Supports anonymous

URL Formats:

Paintings: https://aesthetic.computer/#${code} (hash symbol)
Kidlisp: https://aesthetic.computer/$${code} (dollar symbol)
Tapes: https://aesthetic.computer/!${code} (exclamation symbol)

Media URLs:

Paintings image: https://aesthetic.computer/media/paintings/${code}
Paintings recording: https://aesthetic.computer/media/paintings/${code}.zip
Tapes ZIP: Direct to DigitalOcean Spaces (no short code routing yet)

2. Sync Script Optimizations (sync-atproto.mjs)#

Status: ✅ Complete

Performance:

Parallel user processing: 50 users at a time
Batch creates: 10 concurrent (reduced from 50 to avoid rate limiting)
Batch MongoDB updates: bulkWrite with 100 ops per batch

Admin Tools:

--wipe=handle: Delete all ATProto records + clear MongoDB rkeys
--restore=handle: Sync MongoDB → ATProto per user
Content type filtering: --paintings-only, --moods-only, etc.
Dry run support: Test without making changes

Anonymous/Guest Support:

✅ Wipe supports art.at.aesthetic.computer
✅ Restore supports art.at.aesthetic.computer
✅ Anonymous content query: { user: { $exists: false } }
✅ User content query: { user: userId }

3. Real-Time Sync Integration#

Status: ✅ Complete

Netlify Functions Updated:

✅ track-media.mjs: Auto-syncs paintings/pieces after MongoDB insert
✅ store-kidlisp.mjs: Auto-syncs kidlisp after MongoDB insert
✅ Background sync: Fire-and-forget, doesn't block response
✅ MongoDB updates: Stores rkey after successful ATProto creation

4. Testing Completed#

Status: ✅ Verified with jeffrey.at.aesthetic.computer

Test Results:

✅ Moods: 226/226 synced successfully
✅ Pieces: 26/26 synced successfully
✅ Kidlisp: 168/2220 synced (partial test, working)
✅ Paintings: 19/395 synced (partial test, working with blobs)
✅ Blob generation: Confirmed working (~9KB thumbnails)
✅ Lexicons: All fields present and correct
✅ Rate limiting: 10/batch stable, no errors

Current Sprint 🎯#

Anonymous Tape Testing#

Goal: Test MP4 blob workflow with anonymous tapes

Plan:

✅ Update wipe/restore to support art.at.aesthetic.computer
✅ Add anonymous content queries (no user field)
⏳ Wipe art.at.aesthetic.computer ATProto data
⏳ Restore ONLY tapes (32 anonymous tapes)
⏳ Test MP4 blob generation workflow
⏳ Verify tape records in ATProto

Anonymous Tape Data:

Total: 32 tapes
User: undefined (no user field)
Bucket: art-aesthetic-computer
Format: ZIP with frames + timing.json + metadata.json

Sample Tapes:

[
  {
    "code": "9Yo",
    "slug": "OJZDQoBh",
    "when": "2025-10-17T23:43:17.100Z",
    "bucket": "art-aesthetic-computer"
  },
  {
    "code": "Eue",
    "slug": "lYrrBWh5",
    "when": "2025-10-18T05:28:11.538Z",
    "bucket": "art-aesthetic-computer"
  }
]

Next Steps:

Run wipe: node sync-atproto.mjs live --wipe=art.at.aesthetic.computer --tapes-only
Run restore: node sync-atproto.mjs live --restore=art.at.aesthetic.computer --tapes-only
Observe MP4 blob workflow (will show TODO warning for now)
Document what's needed for MP4 conversion

Pending Work 📋#

1. Tape MP4 Conversion#

Status: ⏳ Documented, not implemented

Plan: See TAPE-MP4-CONVERSION-PLAN.md

Implementation Phases:

Phase 1: ✅ Basic sync without MP4 (current state)
Phase 2: ⏳ Local MP4 conversion using ffmpeg
Phase 3: ⏳ Production deployment

Technical Requirements:

Download ZIP from storage
Extract frames and timing.json
Convert to MP4 using ffmpeg
Upload MP4 blob to ATProto
Include video blob in tape record

2. Full User Sync#

Status: ⏳ Ready to run after tape testing

jeffrey.at.aesthetic.computer:

395 paintings (with blobs + recordings)
226 moods
26 pieces
2220 kidlisp
0 tapes (user tapes don't exist yet)

Estimated Time:

Paintings: ~40 minutes (blob generation + rate limiting)
Moods: ~2 minutes
Pieces: ~1 minute
Kidlisp: ~20 minutes
Total: ~1 hour for full sync

3. Production Rollout#

Status: ⏳ After testing complete

Tasks:

Commit all changes to main
Full sync for all 4,201 users
Monitor logs for errors
Verify data quality in ATProto
Deploy Netlify function updates

Files to Commit:

NEW: media-atproto.mjs (330+ lines)
NEW: ATPROTO-SYNC-CONSOLIDATION-PLAN.md
NEW: ATPROTO-SYNC-PROGRESS.md
NEW: TAPE-MP4-CONVERSION-PLAN.md
MODIFIED: sync-atproto.mjs (refactored to use media-atproto)
MODIFIED: track-media.mjs (real-time sync)
MODIFIED: store-kidlisp.mjs (real-time sync)
MODIFIED: tape-atproto.mjs (added acUrl)

4. Deprecation#

Status: ⏳ After commit

Files to Remove:

painting-atproto.mjs (logic moved to media-atproto.mjs)

Architecture Decisions 📐#

Why Unified Module?#

Before: Duplicate logic in painting-atproto.mjs, sync-atproto.mjs, Netlify functions
After: Single source of truth in media-atproto.mjs
Benefits:
- Consistent lexicons across all entry points
- Easier to maintain and extend
- Single place to update ATProto logic

Why Short Code URLs?#

Permanent: Codes don't change even if handles change
Shareable: Short, memorable codes (#abc, $def, !ghi)
Routable: Edge function handles redirects to storage
ATProto-friendly: Clean URLs for external apps

Why Separate Tape Module?#

Before: Considered adding tapes to unified module
After: Tapes keep separate tape-atproto.mjs due to async MP4 workflow
Rationale: MP4 conversion is async webhook-driven, different from other media

Why Local MP4 Conversion for Sync?#

Flexibility: Control quality and conversion parameters
Testing: Easier to iterate and debug locally
Deployment: Can be deployed incrementally
Async: Production can use webhook service if needed

Metrics 📊#

Performance#

Users processed: ~50/batch
Creates per batch: 10 concurrent
Throughput: ~100 records/minute
Rate limit avoidance: 10/batch stable

Data Quality#

Blob sizes: 7-12KB thumbnails
Lexicon compliance: 100%
Recording detection: Working (HEAD request to ZIP)
URL formats: All using short codes

Coverage#

Total users: 4,201
Users with ATProto accounts: TBD
Anonymous tapes: 32
Authenticated tapes: 3 (jeffrey has 0)

Known Issues 🐛#

1. MP4 Conversion Not Implemented#

Status: Expected, documented Workaround: Tapes sync without video blob for now Fix: Implement tape-to-mp4.mjs module (see TAPE-MP4-CONVERSION-PLAN.md)

2. No Short Code Routing for Tape ZIPs#

Status: Known limitation Impact: Tape zipUrl uses direct DigitalOcean URL Fix: Add /media/tapes/CODE routing to edge function

3. Rate Limiting at 50 Concurrent#

Status: Fixed Solution: Reduced to 10 concurrent creates Result: Stable, no more "Internal Server Error"

Next Actions 🎬#

Immediate (Today):

✅ Document progress
⏳ Wipe art.at.aesthetic.computer
⏳ Restore tapes only
⏳ Test and document MP4 workflow needs

Short Term (This Week):

Implement tape MP4 conversion
Full sync for jeffrey.at.aesthetic.computer
Verify all data in ATProto
Commit changes to main

Long Term (Next Week):

Full sync for all 4,201 users
Monitor production performance
Optimize if needed
Document lessons learned

Configure Feed