feat: add automatic duplicate prevention and update tooling · karitham.dev/atproto-lastfm-importer@0091441

+115 -38

README.md

··· 2 2 3 3 Import your Last.fm and Spotify listening history to the AT Protocol network using the `fm.teal.alpha.feed.play` lexicon. 4 4 5 + **Repository:** [atproto-lastfm-importer](https://github.com/ewanc26/atproto-lastfm-importer) 5 6 [Also available on Tangled](https://tangled.org/@did:plc:ofrbh253gwicbkc5nktqepol/atproto-lastfm-importer) 6 7 7 8 ## ⚠️ Important: Rate Limits ··· 18 19 19 20 ## Quick Start 20 21 22 + **Note:** You must build the project first, then run with arguments. 23 + 21 24 ```bash 22 - # Install dependencies 23 - npm install 25 + # Install dependencies and build 26 + pnpm install 27 + pnpm build 24 28 25 - # Build the project 26 - npm run build 29 + # Show help 30 + pnpm start -- --help 27 31 28 - # Run with interactive prompts 29 - npm start 32 + # Run with command line arguments 33 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 30 34 31 - # Or run with command line arguments 32 - npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 35 + # Alternative: run directly with node (no -- needed) 36 + node dist/index.js -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 33 37 ``` 34 38 35 39 ## Features ··· 42 46 - ✅ **Duplicate Removal**: Clean up accidentally imported duplicate records 43 47 44 48 ### Performance & Safety 49 + - ✅ **Automatic Duplicate Prevention**: Automatically checks Teal and skips records that already exist (no duplicates!) 50 + - ✅ **Input Deduplication**: Removes duplicate entries within the source file before submission 45 51 - ✅ **Batch Operations**: Uses `com.atproto.repo.applyWrites` for efficient batch publishing (up to 200 records per call) 46 52 - ✅ **Rate Limiting**: Automatic daily limits prevent PDS rate limiting 47 53 - ✅ **Multi-Day Imports**: Large imports automatically span multiple days with 24-hour pauses ··· 71 77 72 78 ```bash 73 79 # Preview the merged import 74 - npm start -- -i lastfm.csv --spotify-input spotify-export/ -m combined --dry-run 80 + pnpm start -- -i lastfm.csv --spotify-input spotify-export/ -m combined --dry-run 75 81 76 82 # Perform the combined import 77 - npm start -- -i lastfm.csv --spotify-input spotify-export/ -m combined -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 83 + pnpm start -- -i lastfm.csv --spotify-input spotify-export/ -m combined -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 78 84 ``` 79 85 80 86 **What combined mode does:** ··· 111 117 112 118 ```bash 113 119 # Preview what will be synced 114 - npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync --dry-run 120 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync --dry-run 115 121 116 122 # Perform the sync 117 - npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync -y 123 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync -y 118 124 ``` 119 125 120 126 **Perfect for:** ··· 130 136 131 137 ```bash 132 138 # Preview duplicates (dry run) 133 - npm start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx --dry-run 139 + pnpm start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx --dry-run 134 140 135 141 # Remove duplicates (keeps first occurrence) 136 - npm start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx 142 + pnpm start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx 137 143 ``` 138 144 139 145 ### Import from Spotify 140 146 141 147 ```bash 142 148 # Import single Spotify JSON file 143 - npm start -- -i Streaming_History_Audio_2021-2023_0.json -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 149 + pnpm start -- -i Streaming_History_Audio_2021-2023_0.json -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 144 150 145 151 # Import directory with multiple Spotify files (recommended) 146 - npm start -- -i '/path/to/Spotify Extended Streaming History' -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 152 + pnpm start -- -i '/path/to/Spotify Extended Streaming History' -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 147 153 ``` 148 154 149 155 ### Import from Last.fm 150 156 151 157 ```bash 152 158 # Standard Last.fm import 153 - npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 159 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 154 160 155 161 # Preview without publishing 156 - npm start -- -i lastfm.csv --dry-run 162 + pnpm start -- -i lastfm.csv --dry-run 157 163 158 164 # Process newest tracks first 159 - npm start -- -i lastfm.csv -h alice.bsky.social -r -y 165 + pnpm start -- -i lastfm.csv -h alice.bsky.social -r -y 160 166 161 167 # Verbose debug output 162 - npm start -- -i lastfm.csv --dry-run -v 168 + pnpm start -- -i lastfm.csv --dry-run -v 163 169 164 170 # Quiet mode (only warnings and errors) 165 - npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -q -y 171 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -q -y 166 172 ``` 167 173 168 174 ### Advanced Options 169 175 170 176 ```bash 171 177 # Custom batch settings (advanced users only) 172 - npm start -- -i lastfm.csv -h alice.bsky.social -b 20 -d 3000 178 + pnpm start -- -i lastfm.csv -h alice.bsky.social -b 20 -d 3000 173 179 174 180 # Full automation with all flags 175 - npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y -q 181 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y -q 176 182 ``` 177 183 178 184 ## Command Line Options 185 + 186 + **Note:** When importing data (not in deduplicate mode), you must provide `--input`, `--handle`, and `--password`. The `--yes` flag skips confirmation prompts for automation. 179 187 180 188 ### Required Options 181 189 ··· 256 264 - **trackName**: The name of the track 257 265 - **artists**: Array of artist objects (requires `artistName`, optional `artistMbId` for Last.fm) 258 266 - **playedTime**: ISO 8601 timestamp of when you listened 259 - - **submissionClientAgent**: Identifies this importer (`malachite/v0.6.1`) 267 + - **submissionClientAgent**: Identifies this importer (`malachite/v0.6.2`) 260 268 - **musicServiceBaseDomain**: Set to `last.fm` or `spotify.com` 261 269 262 270 ### Optional Fields ··· 283 291 "recordingMbId": "3a390ad3-fe56-45f2-a073-bebc45d6bde1", 284 292 "playedTime": "2025-11-13T23:49:36Z", 285 293 "originUrl": "https://www.last.fm/music/Cjbeards/_/Paint+My+Masterpiece", 286 - "submissionClientAgent": "malachite/v0.6.1", 294 + "submissionClientAgent": "malachite/v0.6.2", 287 295 "musicServiceBaseDomain": "last.fm" 288 296 } 289 297 ``` ··· 301 309 "releaseName": "Twenty", 302 310 "playedTime": "2021-09-09T10:34:08Z", 303 311 "originUrl": "https://open.spotify.com/track/3gZqDJkMZipOYCRjlHWgOV", 304 - "submissionClientAgent": "malachite/v0.6.1", 312 + "submissionClientAgent": "malachite/v0.6.2", 305 313 "musicServiceBaseDomain": "spotify.com" 306 314 } 307 315 ``` ··· 315 323 2. **Filters data**: 316 324 - Spotify: Automatically removes podcasts, audiobooks, and non-music content 317 325 3. **Converts to schema**: Maps to `fm.teal.alpha.feed.play` format 318 - 4. **Sorts records**: Chronologically (oldest first) or reverse with `-r` flag 319 - 5. **Generates TID-based keys**: From `playedTime` for chronological ordering 320 - 6. **Validates fields**: Ensures required fields are present 321 - 7. **Publishes in batches**: Uses `com.atproto.repo.applyWrites` (up to 200 records per call) 326 + 4. **Deduplicates input**: Removes duplicate entries from the source data (keeps first occurrence) 327 + 5. **Checks Teal**: Fetches existing records and skips any that are already imported (prevents duplicates) 328 + 6. **Sorts records**: Chronologically (oldest first) or reverse with `-r` flag 329 + 7. **Generates TID-based keys**: From `playedTime` for chronological ordering 330 + 8. **Validates fields**: Ensures required fields are present 331 + 9. **Publishes in batches**: Uses `com.atproto.repo.applyWrites` (up to 200 records per call) 332 + 333 + ### Automatic Duplicate Prevention 334 + 335 + The importer has **two layers of duplicate prevention** to ensure you never import the same record twice: 336 + 337 + #### Step 1: Input File Deduplication 338 + 339 + Removes duplicates within your source file(s): 340 + 341 + **How duplicates are identified:** 342 + - Same track name (case-insensitive) 343 + - Same artist name (case-insensitive) 344 + - Same timestamp (exact match) 345 + 346 + **What happens:** 347 + - First occurrence is kept 348 + - Subsequent duplicates are removed 349 + - Shows message: "No duplicates found in input data" or "Removed X duplicate(s)" 350 + 351 + #### Step 2: Teal Comparison (Automatic & Adaptive) 352 + 353 + **Automatically checks your existing Teal records** and skips any that are already imported: 354 + 355 + **What happens:** 356 + - Fetches all existing records from your Teal feed with **adaptive batch sizing** 357 + - Starts with small batches (25 records) and automatically adjusts based on network performance 358 + - Increases batch size (up to 100) when network is fast 359 + - Decreases batch size (down to 10) when network is slow 360 + - Shows real-time progress with fetch rate (records/second) and current batch size 361 + - Compares against your input file 362 + - Only imports records that don't already exist 363 + - Shows: "Found X record(s) already in Teal (skipping)" 364 + 365 + **Example output:** 366 + ``` 367 + ✓ Loaded 10,234 records 368 + ℹ No duplicates found in input data 369 + 370 + === Checking Existing Records === 371 + ℹ Fetching records from Teal to avoid duplicates... 372 + → Fetched 1,000 records (125 rec/s, batch: 37, 8.0s)... 373 + 📈 Network good: batch size 37 → 55 374 + → Fetched 2,000 records (140 rec/s, batch: 82, 14.3s)... 375 + 📈 Network good: batch size 82 → 100 376 + → Fetched 3,000 records (155 rec/s, batch: 100, 19.4s)... 377 + ... 378 + ✓ Found 9,500 existing records in 61.3s (avg 155 rec/s) 379 + 380 + === Identifying New Records === 381 + ℹ Total: 10,234 records 382 + ℹ Existing: 9,100 already in Teal 383 + ℹ New: 1,134 to import 384 + ``` 385 + 386 + **This means:** 387 + - ✅ Safe to re-run imports with updated exports 388 + - ✅ Won't create duplicates if you run the import twice 389 + - ✅ Only pays for API calls on new records 390 + - ✅ Works automatically - no special mode needed 391 + - ✅ Adapts to your network speed - faster on good connections, stable on slow ones 392 + - ✅ Batch size shown in debug mode (`-v`) for transparency 393 + 394 + **Note:** 395 + - This duplicate prevention happens automatically for all imports (default behavior) 396 + - **Credentials required**: Even `--dry-run` needs `--handle` and `--password` to check Teal 397 + - **Sync mode** (`-m sync`): Now primarily just shows detailed statistics about what's being synced 398 + - **Deduplicate mode** (`-m deduplicate`): Removes duplicates from already-imported Teal records (cleanup tool) 322 399 323 400 ### Rate Limiting Algorithm 324 401 1. Calculates safe daily limit (75% of 10K = 7,500 records/day by default) ··· 366 443 367 444 **Default Mode**: Standard operational messages 368 445 ```bash 369 - npm start -- -i lastfm.csv -h alice.bsky.social -p pass 446 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p pass 370 447 ``` 371 448 372 449 **Verbose Mode** (`-v`): Detailed debug information including batch timing and API calls 373 450 ```bash 374 - npm start -- -i lastfm.csv -h alice.bsky.social -p pass -v 451 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p pass -v 375 452 ``` 376 453 377 454 **Quiet Mode** (`-q`): Only warnings and errors 378 455 ```bash 379 - npm start -- -i lastfm.csv -h alice.bsky.social -p pass -q 456 + pnpm start -- -i lastfm.csv -h alice.bsky.social -p pass -q 380 457 ``` 381 458 382 459 ## Error Handling ··· 435 512 436 513 ```bash 437 514 # Type checking 438 - npm run type-check 515 + pnpm run type-check 439 516 440 517 # Build 441 - npm run build 518 + pnpm run build 442 519 443 520 # Development mode (rebuild + run) 444 - npm run dev 521 + pnpm run dev 445 522 446 523 # Run tests 447 - npm run test 524 + pnpm run test 448 525 449 526 # Clean build artifacts 450 - npm run clean 527 + pnpm run clean 451 528 ``` 452 529 453 530 ## Project Structure

-498

package-lock.json

··· 1 - { 2 - "name": "malachite", 3 - "version": "0.6.0", 4 - "lockfileVersion": 3, 5 - "requires": true, 6 - "packages": { 7 - "": { 8 - "name": "malachite", 9 - "version": "0.6.0", 10 - "license": "AGPL-3.0-only", 11 - "dependencies": { 12 - "@atproto/api": "^0.18.13", 13 - "chalk": "^5.6.2", 14 - "cli-progress": "^3.12.0", 15 - "csv-parse": "^6.1.0", 16 - "ora": "^9.0.0" 17 - }, 18 - "bin": { 19 - "lastfm-import": "dist/index.js" 20 - }, 21 - "devDependencies": { 22 - "@types/node": "^20.19.27", 23 - "typescript": "^5.9.3" 24 - } 25 - }, 26 - "node_modules/@atproto/api": { 27 - "version": "0.18.13", 28 - "resolved": "https://registry.npmjs.org/@atproto/api/-/api-0.18.13.tgz", 29 - "integrity": "sha512-CULZ01pSJDltLS/Gc9MMrhFzB6OM3ezyZw7KoeLT/sBfwgA1ddA4mWdTh7DIRosPRigXtA05bnoiCutZbQDo+Q==", 30 - "license": "MIT", 31 - "dependencies": { 32 - "@atproto/common-web": "^0.4.11", 33 - "@atproto/lexicon": "^0.6.0", 34 - "@atproto/syntax": "^0.4.2", 35 - "@atproto/xrpc": "^0.7.7", 36 - "await-lock": "^2.2.2", 37 - "multiformats": "^9.9.0", 38 - "tlds": "^1.234.0", 39 - "zod": "^3.23.8" 40 - } 41 - }, 42 - "node_modules/@atproto/common-web": { 43 - "version": "0.4.11", 44 - "resolved": "https://registry.npmjs.org/@atproto/common-web/-/common-web-0.4.11.tgz", 45 - "integrity": "sha512-VHejNmSABU8/03VrQ3e36AmT5U3UIeio+qSUqCrO1oNgrJcWfGy1rpj0FVtUugWF8Un29+yzkukzWGZfXL70rQ==", 46 - "license": "MIT", 47 - "dependencies": { 48 - "@atproto/lex-data": "0.0.7", 49 - "@atproto/lex-json": "0.0.7", 50 - "zod": "^3.23.8" 51 - } 52 - }, 53 - "node_modules/@atproto/lex-data": { 54 - "version": "0.0.7", 55 - "resolved": "https://registry.npmjs.org/@atproto/lex-data/-/lex-data-0.0.7.tgz", 56 - "integrity": "sha512-W/Q5o9o7n2Sv3UywckChu01X5lwQUtaiiOkGJLnRsdkQTyC6813nPgY+p2sG7NwwM+82lu+FUV9fE/Ul3VqaJw==", 57 - "license": "MIT", 58 - "dependencies": { 59 - "@atproto/syntax": "0.4.2", 60 - "multiformats": "^9.9.0", 61 - "tslib": "^2.8.1", 62 - "uint8arrays": "3.0.0", 63 - "unicode-segmenter": "^0.14.0" 64 - } 65 - }, 66 - "node_modules/@atproto/lex-json": { 67 - "version": "0.0.7", 68 - "resolved": "https://registry.npmjs.org/@atproto/lex-json/-/lex-json-0.0.7.tgz", 69 - "integrity": "sha512-bjNPD5M/MhLfjNM7tcxuls80UgXpHqxdOxDXEUouAtZQV/nIDhGjmNUvKxOmOgnDsiZRnT2g5y3onrnjH3a44g==", 70 - "license": "MIT", 71 - "dependencies": { 72 - "@atproto/lex-data": "0.0.7", 73 - "tslib": "^2.8.1" 74 - } 75 - }, 76 - "node_modules/@atproto/lexicon": { 77 - "version": "0.6.0", 78 - "resolved": "https://registry.npmjs.org/@atproto/lexicon/-/lexicon-0.6.0.tgz", 79 - "integrity": "sha512-5veb8aD+J5M0qszLJ+73KSFsFrJBgAY/nM1TSAJvGY7fNc9ZAT+PSUlmIyrdye9YznAZ07yktalls/TwNV7cHQ==", 80 - "license": "MIT", 81 - "dependencies": { 82 - "@atproto/common-web": "^0.4.7", 83 - "@atproto/syntax": "^0.4.2", 84 - "iso-datestring-validator": "^2.2.2", 85 - "multiformats": "^9.9.0", 86 - "zod": "^3.23.8" 87 - } 88 - }, 89 - "node_modules/@atproto/syntax": { 90 - "version": "0.4.2", 91 - "resolved": "https://registry.npmjs.org/@atproto/syntax/-/syntax-0.4.2.tgz", 92 - "integrity": "sha512-X9XSRPinBy/0VQ677j8VXlBsYSsUXaiqxWVpGGxJYsAhugdQRb0jqaVKJFtm6RskeNkV6y9xclSUi9UYG/COrA==", 93 - "license": "MIT" 94 - }, 95 - "node_modules/@atproto/xrpc": { 96 - "version": "0.7.7", 97 - "resolved": "https://registry.npmjs.org/@atproto/xrpc/-/xrpc-0.7.7.tgz", 98 - "integrity": "sha512-K1ZyO/BU8JNtXX5dmPp7b5UrkLMMqpsIa/Lrj5D3Su+j1Xwq1m6QJ2XJ1AgjEjkI1v4Muzm7klianLE6XGxtmA==", 99 - "license": "MIT", 100 - "dependencies": { 101 - "@atproto/lexicon": "^0.6.0", 102 - "zod": "^3.23.8" 103 - } 104 - }, 105 - "node_modules/@types/node": { 106 - "version": "20.19.27", 107 - "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.27.tgz", 108 - "integrity": "sha512-N2clP5pJhB2YnZJ3PIHFk5RkygRX5WO/5f0WC08tp0wd+sv0rsJk3MqWn3CbNmT2J505a5336jaQj4ph1AdMug==", 109 - "dev": true, 110 - "license": "MIT", 111 - "dependencies": { 112 - "undici-types": "~6.21.0" 113 - } 114 - }, 115 - "node_modules/ansi-regex": { 116 - "version": "6.2.2", 117 - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.2.2.tgz", 118 - "integrity": "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg==", 119 - "license": "MIT", 120 - "engines": { 121 - "node": ">=12" 122 - }, 123 - "funding": { 124 - "url": "https://github.com/chalk/ansi-regex?sponsor=1" 125 - } 126 - }, 127 - "node_modules/await-lock": { 128 - "version": "2.2.2", 129 - "resolved": "https://registry.npmjs.org/await-lock/-/await-lock-2.2.2.tgz", 130 - "integrity": "sha512-aDczADvlvTGajTDjcjpJMqRkOF6Qdz3YbPZm/PyW6tKPkx2hlYBzxMhEywM/tU72HrVZjgl5VCdRuMlA7pZ8Gw==", 131 - "license": "MIT" 132 - }, 133 - "node_modules/chalk": { 134 - "version": "5.6.2", 135 - "resolved": "https://registry.npmjs.org/chalk/-/chalk-5.6.2.tgz", 136 - "integrity": "sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==", 137 - "license": "MIT", 138 - "engines": { 139 - "node": "^12.17.0 || ^14.13 || >=16.0.0" 140 - }, 141 - "funding": { 142 - "url": "https://github.com/chalk/chalk?sponsor=1" 143 - } 144 - }, 145 - "node_modules/cli-cursor": { 146 - "version": "5.0.0", 147 - "resolved": "https://registry.npmjs.org/cli-cursor/-/cli-cursor-5.0.0.tgz", 148 - "integrity": "sha512-aCj4O5wKyszjMmDT4tZj93kxyydN/K5zPWSCe6/0AV/AA1pqe5ZBIw0a2ZfPQV7lL5/yb5HsUreJ6UFAF1tEQw==", 149 - "license": "MIT", 150 - "dependencies": { 151 - "restore-cursor": "^5.0.0" 152 - }, 153 - "engines": { 154 - "node": ">=18" 155 - }, 156 - "funding": { 157 - "url": "https://github.com/sponsors/sindresorhus" 158 - } 159 - }, 160 - "node_modules/cli-progress": { 161 - "version": "3.12.0", 162 - "resolved": "https://registry.npmjs.org/cli-progress/-/cli-progress-3.12.0.tgz", 163 - "integrity": "sha512-tRkV3HJ1ASwm19THiiLIXLO7Im7wlTuKnvkYaTkyoAPefqjNg7W7DHKUlGRxy9vxDvbyCYQkQozvptuMkGCg8A==", 164 - "license": "MIT", 165 - "dependencies": { 166 - "string-width": "^4.2.3" 167 - }, 168 - "engines": { 169 - "node": ">=4" 170 - } 171 - }, 172 - "node_modules/cli-spinners": { 173 - "version": "3.3.0", 174 - "resolved": "https://registry.npmjs.org/cli-spinners/-/cli-spinners-3.3.0.tgz", 175 - "integrity": "sha512-/+40ljC3ONVnYIttjMWrlL51nItDAbBrq2upN8BPyvGU/2n5Oxw3tbNwORCaNuNqLJnxGqOfjUuhsv7l5Q4IsQ==", 176 - "license": "MIT", 177 - "engines": { 178 - "node": ">=18.20" 179 - }, 180 - "funding": { 181 - "url": "https://github.com/sponsors/sindresorhus" 182 - } 183 - }, 184 - "node_modules/csv-parse": { 185 - "version": "6.1.0", 186 - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", 187 - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", 188 - "license": "MIT" 189 - }, 190 - "node_modules/emoji-regex": { 191 - "version": "8.0.0", 192 - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", 193 - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", 194 - "license": "MIT" 195 - }, 196 - "node_modules/get-east-asian-width": { 197 - "version": "1.4.0", 198 - "resolved": "https://registry.npmjs.org/get-east-asian-width/-/get-east-asian-width-1.4.0.tgz", 199 - "integrity": "sha512-QZjmEOC+IT1uk6Rx0sX22V6uHWVwbdbxf1faPqJ1QhLdGgsRGCZoyaQBm/piRdJy/D2um6hM1UP7ZEeQ4EkP+Q==", 200 - "license": "MIT", 201 - "engines": { 202 - "node": ">=18" 203 - }, 204 - "funding": { 205 - "url": "https://github.com/sponsors/sindresorhus" 206 - } 207 - }, 208 - "node_modules/is-fullwidth-code-point": { 209 - "version": "3.0.0", 210 - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", 211 - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", 212 - "license": "MIT", 213 - "engines": { 214 - "node": ">=8" 215 - } 216 - }, 217 - "node_modules/is-interactive": { 218 - "version": "2.0.0", 219 - "resolved": "https://registry.npmjs.org/is-interactive/-/is-interactive-2.0.0.tgz", 220 - "integrity": "sha512-qP1vozQRI+BMOPcjFzrjXuQvdak2pHNUMZoeG2eRbiSqyvbEf/wQtEOTOX1guk6E3t36RkaqiSt8A/6YElNxLQ==", 221 - "license": "MIT", 222 - "engines": { 223 - "node": ">=12" 224 - }, 225 - "funding": { 226 - "url": "https://github.com/sponsors/sindresorhus" 227 - } 228 - }, 229 - "node_modules/is-unicode-supported": { 230 - "version": "2.1.0", 231 - "resolved": "https://registry.npmjs.org/is-unicode-supported/-/is-unicode-supported-2.1.0.tgz", 232 - "integrity": "sha512-mE00Gnza5EEB3Ds0HfMyllZzbBrmLOX3vfWoj9A9PEnTfratQ/BcaJOuMhnkhjXvb2+FkY3VuHqtAGpTPmglFQ==", 233 - "license": "MIT", 234 - "engines": { 235 - "node": ">=18" 236 - }, 237 - "funding": { 238 - "url": "https://github.com/sponsors/sindresorhus" 239 - } 240 - }, 241 - "node_modules/iso-datestring-validator": { 242 - "version": "2.2.2", 243 - "resolved": "https://registry.npmjs.org/iso-datestring-validator/-/iso-datestring-validator-2.2.2.tgz", 244 - "integrity": "sha512-yLEMkBbLZTlVQqOnQ4FiMujR6T4DEcCb1xizmvXS+OxuhwcbtynoosRzdMA69zZCShCNAbi+gJ71FxZBBXx1SA==", 245 - "license": "MIT" 246 - }, 247 - "node_modules/log-symbols": { 248 - "version": "7.0.1", 249 - "resolved": "https://registry.npmjs.org/log-symbols/-/log-symbols-7.0.1.tgz", 250 - "integrity": "sha512-ja1E3yCr9i/0hmBVaM0bfwDjnGy8I/s6PP4DFp+yP+a+mrHO4Rm7DtmnqROTUkHIkqffC84YY7AeqX6oFk0WFg==", 251 - "license": "MIT", 252 - "dependencies": { 253 - "is-unicode-supported": "^2.0.0", 254 - "yoctocolors": "^2.1.1" 255 - }, 256 - "engines": { 257 - "node": ">=18" 258 - }, 259 - "funding": { 260 - "url": "https://github.com/sponsors/sindresorhus" 261 - } 262 - }, 263 - "node_modules/mimic-function": { 264 - "version": "5.0.1", 265 - "resolved": "https://registry.npmjs.org/mimic-function/-/mimic-function-5.0.1.tgz", 266 - "integrity": "sha512-VP79XUPxV2CigYP3jWwAUFSku2aKqBH7uTAapFWCBqutsbmDo96KY5o8uh6U+/YSIn5OxJnXp73beVkpqMIGhA==", 267 - "license": "MIT", 268 - "engines": { 269 - "node": ">=18" 270 - }, 271 - "funding": { 272 - "url": "https://github.com/sponsors/sindresorhus" 273 - } 274 - }, 275 - "node_modules/multiformats": { 276 - "version": "9.9.0", 277 - "resolved": "https://registry.npmjs.org/multiformats/-/multiformats-9.9.0.tgz", 278 - "integrity": "sha512-HoMUjhH9T8DDBNT+6xzkrd9ga/XiBI4xLr58LJACwK6G3HTOPeMz4nB4KJs33L2BelrIJa7P0VuNaVF3hMYfjg==", 279 - "license": "(Apache-2.0 AND MIT)" 280 - }, 281 - "node_modules/onetime": { 282 - "version": "7.0.0", 283 - "resolved": "https://registry.npmjs.org/onetime/-/onetime-7.0.0.tgz", 284 - "integrity": "sha512-VXJjc87FScF88uafS3JllDgvAm+c/Slfz06lorj2uAY34rlUu0Nt+v8wreiImcrgAjjIHp1rXpTDlLOGw29WwQ==", 285 - "license": "MIT", 286 - "dependencies": { 287 - "mimic-function": "^5.0.0" 288 - }, 289 - "engines": { 290 - "node": ">=18" 291 - }, 292 - "funding": { 293 - "url": "https://github.com/sponsors/sindresorhus" 294 - } 295 - }, 296 - "node_modules/ora": { 297 - "version": "9.0.0", 298 - "resolved": "https://registry.npmjs.org/ora/-/ora-9.0.0.tgz", 299 - "integrity": "sha512-m0pg2zscbYgWbqRR6ABga5c3sZdEon7bSgjnlXC64kxtxLOyjRcbbUkLj7HFyy/FTD+P2xdBWu8snGhYI0jc4A==", 300 - "license": "MIT", 301 - "dependencies": { 302 - "chalk": "^5.6.2", 303 - "cli-cursor": "^5.0.0", 304 - "cli-spinners": "^3.2.0", 305 - "is-interactive": "^2.0.0", 306 - "is-unicode-supported": "^2.1.0", 307 - "log-symbols": "^7.0.1", 308 - "stdin-discarder": "^0.2.2", 309 - "string-width": "^8.1.0", 310 - "strip-ansi": "^7.1.2" 311 - }, 312 - "engines": { 313 - "node": ">=20" 314 - }, 315 - "funding": { 316 - "url": "https://github.com/sponsors/sindresorhus" 317 - } 318 - }, 319 - "node_modules/ora/node_modules/string-width": { 320 - "version": "8.1.0", 321 - "resolved": "https://registry.npmjs.org/string-width/-/string-width-8.1.0.tgz", 322 - "integrity": "sha512-Kxl3KJGb/gxkaUMOjRsQ8IrXiGW75O4E3RPjFIINOVH8AMl2SQ/yWdTzWwF3FevIX9LcMAjJW+GRwAlAbTSXdg==", 323 - "license": "MIT", 324 - "dependencies": { 325 - "get-east-asian-width": "^1.3.0", 326 - "strip-ansi": "^7.1.0" 327 - }, 328 - "engines": { 329 - "node": ">=20" 330 - }, 331 - "funding": { 332 - "url": "https://github.com/sponsors/sindresorhus" 333 - } 334 - }, 335 - "node_modules/restore-cursor": { 336 - "version": "5.1.0", 337 - "resolved": "https://registry.npmjs.org/restore-cursor/-/restore-cursor-5.1.0.tgz", 338 - "integrity": "sha512-oMA2dcrw6u0YfxJQXm342bFKX/E4sG9rbTzO9ptUcR/e8A33cHuvStiYOwH7fszkZlZ1z/ta9AAoPk2F4qIOHA==", 339 - "license": "MIT", 340 - "dependencies": { 341 - "onetime": "^7.0.0", 342 - "signal-exit": "^4.1.0" 343 - }, 344 - "engines": { 345 - "node": ">=18" 346 - }, 347 - "funding": { 348 - "url": "https://github.com/sponsors/sindresorhus" 349 - } 350 - }, 351 - "node_modules/signal-exit": { 352 - "version": "4.1.0", 353 - "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz", 354 - "integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==", 355 - "license": "ISC", 356 - "engines": { 357 - "node": ">=14" 358 - }, 359 - "funding": { 360 - "url": "https://github.com/sponsors/isaacs" 361 - } 362 - }, 363 - "node_modules/stdin-discarder": { 364 - "version": "0.2.2", 365 - "resolved": "https://registry.npmjs.org/stdin-discarder/-/stdin-discarder-0.2.2.tgz", 366 - "integrity": "sha512-UhDfHmA92YAlNnCfhmq0VeNL5bDbiZGg7sZ2IvPsXubGkiNa9EC+tUTsjBRsYUAz87btI6/1wf4XoVvQ3uRnmQ==", 367 - "license": "MIT", 368 - "engines": { 369 - "node": ">=18" 370 - }, 371 - "funding": { 372 - "url": "https://github.com/sponsors/sindresorhus" 373 - } 374 - }, 375 - "node_modules/string-width": { 376 - "version": "4.2.3", 377 - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", 378 - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", 379 - "license": "MIT", 380 - "dependencies": { 381 - "emoji-regex": "^8.0.0", 382 - "is-fullwidth-code-point": "^3.0.0", 383 - "strip-ansi": "^6.0.1" 384 - }, 385 - "engines": { 386 - "node": ">=8" 387 - } 388 - }, 389 - "node_modules/string-width/node_modules/ansi-regex": { 390 - "version": "5.0.1", 391 - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", 392 - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", 393 - "license": "MIT", 394 - "engines": { 395 - "node": ">=8" 396 - } 397 - }, 398 - "node_modules/string-width/node_modules/strip-ansi": { 399 - "version": "6.0.1", 400 - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", 401 - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", 402 - "license": "MIT", 403 - "dependencies": { 404 - "ansi-regex": "^5.0.1" 405 - }, 406 - "engines": { 407 - "node": ">=8" 408 - } 409 - }, 410 - "node_modules/strip-ansi": { 411 - "version": "7.1.2", 412 - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.1.2.tgz", 413 - "integrity": "sha512-gmBGslpoQJtgnMAvOVqGZpEz9dyoKTCzy2nfz/n8aIFhN/jCE/rCmcxabB6jOOHV+0WNnylOxaxBQPSvcWklhA==", 414 - "license": "MIT", 415 - "dependencies": { 416 - "ansi-regex": "^6.0.1" 417 - }, 418 - "engines": { 419 - "node": ">=12" 420 - }, 421 - "funding": { 422 - "url": "https://github.com/chalk/strip-ansi?sponsor=1" 423 - } 424 - }, 425 - "node_modules/tlds": { 426 - "version": "1.261.0", 427 - "resolved": "https://registry.npmjs.org/tlds/-/tlds-1.261.0.tgz", 428 - "integrity": "sha512-QXqwfEl9ddlGBaRFXIvNKK6OhipSiLXuRuLJX5DErz0o0Q0rYxulWLdFryTkV5PkdZct5iMInwYEGe/eR++1AA==", 429 - "license": "MIT", 430 - "bin": { 431 - "tlds": "bin.js" 432 - } 433 - }, 434 - "node_modules/tslib": { 435 - "version": "2.8.1", 436 - "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", 437 - "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", 438 - "license": "0BSD" 439 - }, 440 - "node_modules/typescript": { 441 - "version": "5.9.3", 442 - "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", 443 - "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", 444 - "dev": true, 445 - "license": "Apache-2.0", 446 - "bin": { 447 - "tsc": "bin/tsc", 448 - "tsserver": "bin/tsserver" 449 - }, 450 - "engines": { 451 - "node": ">=14.17" 452 - } 453 - }, 454 - "node_modules/uint8arrays": { 455 - "version": "3.0.0", 456 - "resolved": "https://registry.npmjs.org/uint8arrays/-/uint8arrays-3.0.0.tgz", 457 - "integrity": "sha512-HRCx0q6O9Bfbp+HHSfQQKD7wU70+lydKVt4EghkdOvlK/NlrF90z+eXV34mUd48rNvVJXwkrMSPpCATkct8fJA==", 458 - "license": "MIT", 459 - "dependencies": { 460 - "multiformats": "^9.4.2" 461 - } 462 - }, 463 - "node_modules/undici-types": { 464 - "version": "6.21.0", 465 - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", 466 - "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", 467 - "dev": true, 468 - "license": "MIT" 469 - }, 470 - "node_modules/unicode-segmenter": { 471 - "version": "0.14.5", 472 - "resolved": "https://registry.npmjs.org/unicode-segmenter/-/unicode-segmenter-0.14.5.tgz", 473 - "integrity": "sha512-jHGmj2LUuqDcX3hqY12Ql+uhUTn8huuxNZGq7GvtF6bSybzH3aFgedYu/KTzQStEgt1Ra2F3HxadNXsNjb3m3g==", 474 - "license": "MIT" 475 - }, 476 - "node_modules/yoctocolors": { 477 - "version": "2.1.2", 478 - "resolved": "https://registry.npmjs.org/yoctocolors/-/yoctocolors-2.1.2.tgz", 479 - "integrity": "sha512-CzhO+pFNo8ajLM2d2IW/R93ipy99LWjtwblvC1RsoSUMZgyLbYFr221TnSNT7GjGdYui6P459mw9JH/g/zW2ug==", 480 - "license": "MIT", 481 - "engines": { 482 - "node": ">=18" 483 - }, 484 - "funding": { 485 - "url": "https://github.com/sponsors/sindresorhus" 486 - } 487 - }, 488 - "node_modules/zod": { 489 - "version": "3.25.76", 490 - "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", 491 - "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", 492 - "license": "MIT", 493 - "funding": { 494 - "url": "https://github.com/sponsors/colinhacks" 495 - } 496 - } 497 - } 498 - }

+2 -2

package.json

··· 1 1 { 2 2 "name": "malachite", 3 - "version": "0.6.1", 3 + "version": "0.6.2", 4 4 "description": "Import Last.fm scrobbles to ATProto with rate limiting", 5 5 "type": "module", 6 6 "main": "./dist/index.js", ··· 10 10 }, 11 11 "scripts": { 12 12 "build": "tsc", 13 - "start": "npm run build && node dist/index.js", 13 + "start": "node dist/index.js", 14 14 "dev": "tsc && node dist/index.js", 15 15 "dry-run": "npm run build && node dist/index.js --dry-run", 16 16 "clean": "rm -rf dist",

+213 -39

pnpm-lock.yaml

··· 9 9 .: 10 10 dependencies: 11 11 '@atproto/api': 12 - specifier: ^0.13.35 13 - version: 0.13.35 12 + specifier: ^0.18.13 13 + version: 0.18.15 14 + chalk: 15 + specifier: ^5.6.2 16 + version: 5.6.2 17 + cli-progress: 18 + specifier: ^3.12.0 19 + version: 3.12.0 14 20 csv-parse: 15 - specifier: ^5.6.0 16 - version: 5.6.0 21 + specifier: ^6.1.0 22 + version: 6.1.0 23 + ora: 24 + specifier: ^9.0.0 25 + version: 9.0.0 17 26 devDependencies: 18 27 '@types/node': 19 28 specifier: ^20.19.27 ··· 24 33 25 34 packages: 26 35 27 - '@atproto/api@0.13.35': 28 - resolution: {integrity: sha512-vsEfBj0C333TLjDppvTdTE0IdKlXuljKSveAeI4PPx/l6eUKNnDTsYxvILtXUVzwUlTDmSRqy5O4Ryh78n1b7g==} 36 + '@atproto/api@0.18.15': 37 + resolution: {integrity: sha512-GeaTP7HMRZa8jD6trMuTACa8t2jkFtRmcwWgrB0FT7l9jVCXrKpYupWeIeauEgWHNwWUUiaq3LmCox+HBy8ZMQ==} 29 38 30 - '@atproto/common-web@0.4.9': 31 - resolution: {integrity: sha512-RGt1rUjVC8FEUlF5JQyN3xYlqZJbFTN0XSBBxl+HozjZGhhVtAVFGa+F+TR6BCVs7q7TcitOv/y/YWz4jJWn9g==} 39 + '@atproto/common-web@0.4.12': 40 + resolution: {integrity: sha512-3aCJemqM/fkHQrVPbTCHCdiVstKFI+2LkFLvUhO6XZP0EqUZa/rg/CIZBKTFUWu9I5iYiaEiXL9VwcDRpEevSw==} 32 41 33 - '@atproto/lex-data@0.0.5': 34 - resolution: {integrity: sha512-nasD4eo2wKLyhHozC0vy7Jhp/fBwCKnYhQQogYtraUlT9il6lK1drhT8CNpWlglOhb0T73jLG5WpfNsPp6Pr/w==} 42 + '@atproto/lex-data@0.0.8': 43 + resolution: {integrity: sha512-1Y5tz7BkS7380QuLNXaE8GW8Xba+mRWugt8BKM4BUFYjjUZdmirU8lr72iM4XlEBrzRu8Cfvj+MbsbYaZv+IgA==} 35 44 36 - '@atproto/lex-json@0.0.5': 37 - resolution: {integrity: sha512-wgmET7fIWi77jxqHnrr0RvpAGhiFqIqjdO9Py3JK2whHMITyYgFRU0HfEtIeWSzx0Vb9z0S7F/fQW3P3gqb+yA==} 38 - 39 - '@atproto/lexicon@0.4.14': 40 - resolution: {integrity: sha512-jiKpmH1QER3Gvc7JVY5brwrfo+etFoe57tKPQX/SmPwjvUsFnJAow5xLIryuBaJgFAhnTZViXKs41t//pahGHQ==} 45 + '@atproto/lex-json@0.0.8': 46 + resolution: {integrity: sha512-w1Qmkae1QhmNz+i1Zm3xr3jp0UPPRENmdlpU0qIrdxWDo9W4Mzkeyc3eSoa+Zs+zN8xkRSQw7RLZte/B7Ipdwg==} 41 47 42 - '@atproto/syntax@0.3.4': 43 - resolution: {integrity: sha512-8CNmi5DipOLaVeSMPggMe7FCksVag0aO6XZy9WflbduTKM4dFZVCs4686UeMLfGRXX+X966XgwECHoLYrovMMg==} 48 + '@atproto/lexicon@0.6.0': 49 + resolution: {integrity: sha512-5veb8aD+J5M0qszLJ+73KSFsFrJBgAY/nM1TSAJvGY7fNc9ZAT+PSUlmIyrdye9YznAZ07yktalls/TwNV7cHQ==} 44 50 45 51 '@atproto/syntax@0.4.2': 46 52 resolution: {integrity: sha512-X9XSRPinBy/0VQ677j8VXlBsYSsUXaiqxWVpGGxJYsAhugdQRb0jqaVKJFtm6RskeNkV6y9xclSUi9UYG/COrA==} 47 53 48 - '@atproto/xrpc@0.6.12': 49 - resolution: {integrity: sha512-Ut3iISNLujlmY9Gu8sNU+SPDJDvqlVzWddU8qUr0Yae5oD4SguaUFjjhireMGhQ3M5E0KljQgDbTmnBo1kIZ3w==} 54 + '@atproto/xrpc@0.7.7': 55 + resolution: {integrity: sha512-K1ZyO/BU8JNtXX5dmPp7b5UrkLMMqpsIa/Lrj5D3Su+j1Xwq1m6QJ2XJ1AgjEjkI1v4Muzm7klianLE6XGxtmA==} 50 56 51 57 '@types/node@20.19.27': 52 58 resolution: {integrity: sha512-N2clP5pJhB2YnZJ3PIHFk5RkygRX5WO/5f0WC08tp0wd+sv0rsJk3MqWn3CbNmT2J505a5336jaQj4ph1AdMug==} 53 59 60 + ansi-regex@5.0.1: 61 + resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==} 62 + engines: {node: '>=8'} 63 + 64 + ansi-regex@6.2.2: 65 + resolution: {integrity: sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg==} 66 + engines: {node: '>=12'} 67 + 54 68 await-lock@2.2.2: 55 69 resolution: {integrity: sha512-aDczADvlvTGajTDjcjpJMqRkOF6Qdz3YbPZm/PyW6tKPkx2hlYBzxMhEywM/tU72HrVZjgl5VCdRuMlA7pZ8Gw==} 56 70 57 - csv-parse@5.6.0: 58 - resolution: {integrity: sha512-l3nz3euub2QMg5ouu5U09Ew9Wf6/wQ8I++ch1loQ0ljmzhmfZYrH9fflS22i/PQEvsPvxCwxgz5q7UB8K1JO4Q==} 71 + chalk@5.6.2: 72 + resolution: {integrity: sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==} 73 + engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} 74 + 75 + cli-cursor@5.0.0: 76 + resolution: {integrity: sha512-aCj4O5wKyszjMmDT4tZj93kxyydN/K5zPWSCe6/0AV/AA1pqe5ZBIw0a2ZfPQV7lL5/yb5HsUreJ6UFAF1tEQw==} 77 + engines: {node: '>=18'} 78 + 79 + cli-progress@3.12.0: 80 + resolution: {integrity: sha512-tRkV3HJ1ASwm19THiiLIXLO7Im7wlTuKnvkYaTkyoAPefqjNg7W7DHKUlGRxy9vxDvbyCYQkQozvptuMkGCg8A==} 81 + engines: {node: '>=4'} 82 + 83 + cli-spinners@3.4.0: 84 + resolution: {integrity: sha512-bXfOC4QcT1tKXGorxL3wbJm6XJPDqEnij2gQ2m7ESQuE+/z9YFIWnl/5RpTiKWbMq3EVKR4fRLJGn6DVfu0mpw==} 85 + engines: {node: '>=18.20'} 86 + 87 + csv-parse@6.1.0: 88 + resolution: {integrity: sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==} 89 + 90 + emoji-regex@8.0.0: 91 + resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} 92 + 93 + get-east-asian-width@1.4.0: 94 + resolution: {integrity: sha512-QZjmEOC+IT1uk6Rx0sX22V6uHWVwbdbxf1faPqJ1QhLdGgsRGCZoyaQBm/piRdJy/D2um6hM1UP7ZEeQ4EkP+Q==} 95 + engines: {node: '>=18'} 96 + 97 + is-fullwidth-code-point@3.0.0: 98 + resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==} 99 + engines: {node: '>=8'} 100 + 101 + is-interactive@2.0.0: 102 + resolution: {integrity: sha512-qP1vozQRI+BMOPcjFzrjXuQvdak2pHNUMZoeG2eRbiSqyvbEf/wQtEOTOX1guk6E3t36RkaqiSt8A/6YElNxLQ==} 103 + engines: {node: '>=12'} 104 + 105 + is-unicode-supported@2.1.0: 106 + resolution: {integrity: sha512-mE00Gnza5EEB3Ds0HfMyllZzbBrmLOX3vfWoj9A9PEnTfratQ/BcaJOuMhnkhjXvb2+FkY3VuHqtAGpTPmglFQ==} 107 + engines: {node: '>=18'} 59 108 60 109 iso-datestring-validator@2.2.2: 61 110 resolution: {integrity: sha512-yLEMkBbLZTlVQqOnQ4FiMujR6T4DEcCb1xizmvXS+OxuhwcbtynoosRzdMA69zZCShCNAbi+gJ71FxZBBXx1SA==} 62 111 112 + log-symbols@7.0.1: 113 + resolution: {integrity: sha512-ja1E3yCr9i/0hmBVaM0bfwDjnGy8I/s6PP4DFp+yP+a+mrHO4Rm7DtmnqROTUkHIkqffC84YY7AeqX6oFk0WFg==} 114 + engines: {node: '>=18'} 115 + 116 + mimic-function@5.0.1: 117 + resolution: {integrity: sha512-VP79XUPxV2CigYP3jWwAUFSku2aKqBH7uTAapFWCBqutsbmDo96KY5o8uh6U+/YSIn5OxJnXp73beVkpqMIGhA==} 118 + engines: {node: '>=18'} 119 + 63 120 multiformats@9.9.0: 64 121 resolution: {integrity: sha512-HoMUjhH9T8DDBNT+6xzkrd9ga/XiBI4xLr58LJACwK6G3HTOPeMz4nB4KJs33L2BelrIJa7P0VuNaVF3hMYfjg==} 65 122 123 + onetime@7.0.0: 124 + resolution: {integrity: sha512-VXJjc87FScF88uafS3JllDgvAm+c/Slfz06lorj2uAY34rlUu0Nt+v8wreiImcrgAjjIHp1rXpTDlLOGw29WwQ==} 125 + engines: {node: '>=18'} 126 + 127 + ora@9.0.0: 128 + resolution: {integrity: sha512-m0pg2zscbYgWbqRR6ABga5c3sZdEon7bSgjnlXC64kxtxLOyjRcbbUkLj7HFyy/FTD+P2xdBWu8snGhYI0jc4A==} 129 + engines: {node: '>=20'} 130 + 131 + restore-cursor@5.1.0: 132 + resolution: {integrity: sha512-oMA2dcrw6u0YfxJQXm342bFKX/E4sG9rbTzO9ptUcR/e8A33cHuvStiYOwH7fszkZlZ1z/ta9AAoPk2F4qIOHA==} 133 + engines: {node: '>=18'} 134 + 135 + signal-exit@4.1.0: 136 + resolution: {integrity: sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==} 137 + engines: {node: '>=14'} 138 + 139 + stdin-discarder@0.2.2: 140 + resolution: {integrity: sha512-UhDfHmA92YAlNnCfhmq0VeNL5bDbiZGg7sZ2IvPsXubGkiNa9EC+tUTsjBRsYUAz87btI6/1wf4XoVvQ3uRnmQ==} 141 + engines: {node: '>=18'} 142 + 143 + string-width@4.2.3: 144 + resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==} 145 + engines: {node: '>=8'} 146 + 147 + string-width@8.1.0: 148 + resolution: {integrity: sha512-Kxl3KJGb/gxkaUMOjRsQ8IrXiGW75O4E3RPjFIINOVH8AMl2SQ/yWdTzWwF3FevIX9LcMAjJW+GRwAlAbTSXdg==} 149 + engines: {node: '>=20'} 150 + 151 + strip-ansi@6.0.1: 152 + resolution: {integrity: sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==} 153 + engines: {node: '>=8'} 154 + 155 + strip-ansi@7.1.2: 156 + resolution: {integrity: sha512-gmBGslpoQJtgnMAvOVqGZpEz9dyoKTCzy2nfz/n8aIFhN/jCE/rCmcxabB6jOOHV+0WNnylOxaxBQPSvcWklhA==} 157 + engines: {node: '>=12'} 158 + 66 159 tlds@1.261.0: 67 160 resolution: {integrity: sha512-QXqwfEl9ddlGBaRFXIvNKK6OhipSiLXuRuLJX5DErz0o0Q0rYxulWLdFryTkV5PkdZct5iMInwYEGe/eR++1AA==} 68 161 hasBin: true ··· 84 177 unicode-segmenter@0.14.5: 85 178 resolution: {integrity: sha512-jHGmj2LUuqDcX3hqY12Ql+uhUTn8huuxNZGq7GvtF6bSybzH3aFgedYu/KTzQStEgt1Ra2F3HxadNXsNjb3m3g==} 86 179 180 + yoctocolors@2.1.2: 181 + resolution: {integrity: sha512-CzhO+pFNo8ajLM2d2IW/R93ipy99LWjtwblvC1RsoSUMZgyLbYFr221TnSNT7GjGdYui6P459mw9JH/g/zW2ug==} 182 + engines: {node: '>=18'} 183 + 87 184 zod@3.25.76: 88 185 resolution: {integrity: sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==} 89 186 90 187 snapshots: 91 188 92 - '@atproto/api@0.13.35': 189 + '@atproto/api@0.18.15': 93 190 dependencies: 94 - '@atproto/common-web': 0.4.9 95 - '@atproto/lexicon': 0.4.14 96 - '@atproto/syntax': 0.3.4 97 - '@atproto/xrpc': 0.6.12 191 + '@atproto/common-web': 0.4.12 192 + '@atproto/lexicon': 0.6.0 193 + '@atproto/syntax': 0.4.2 194 + '@atproto/xrpc': 0.7.7 98 195 await-lock: 2.2.2 99 196 multiformats: 9.9.0 100 197 tlds: 1.261.0 101 198 zod: 3.25.76 102 199 103 - '@atproto/common-web@0.4.9': 200 + '@atproto/common-web@0.4.12': 104 201 dependencies: 105 - '@atproto/lex-data': 0.0.5 106 - '@atproto/lex-json': 0.0.5 202 + '@atproto/lex-data': 0.0.8 203 + '@atproto/lex-json': 0.0.8 107 204 zod: 3.25.76 108 205 109 - '@atproto/lex-data@0.0.5': 206 + '@atproto/lex-data@0.0.8': 110 207 dependencies: 111 208 '@atproto/syntax': 0.4.2 112 209 multiformats: 9.9.0 ··· 114 211 uint8arrays: 3.0.0 115 212 unicode-segmenter: 0.14.5 116 213 117 - '@atproto/lex-json@0.0.5': 214 + '@atproto/lex-json@0.0.8': 118 215 dependencies: 119 - '@atproto/lex-data': 0.0.5 216 + '@atproto/lex-data': 0.0.8 120 217 tslib: 2.8.1 121 218 122 - '@atproto/lexicon@0.4.14': 219 + '@atproto/lexicon@0.6.0': 123 220 dependencies: 124 - '@atproto/common-web': 0.4.9 221 + '@atproto/common-web': 0.4.12 125 222 '@atproto/syntax': 0.4.2 126 223 iso-datestring-validator: 2.2.2 127 224 multiformats: 9.9.0 128 225 zod: 3.25.76 129 226 130 - '@atproto/syntax@0.3.4': {} 131 - 132 227 '@atproto/syntax@0.4.2': {} 133 228 134 - '@atproto/xrpc@0.6.12': 229 + '@atproto/xrpc@0.7.7': 135 230 dependencies: 136 - '@atproto/lexicon': 0.4.14 231 + '@atproto/lexicon': 0.6.0 137 232 zod: 3.25.76 138 233 139 234 '@types/node@20.19.27': 140 235 dependencies: 141 236 undici-types: 6.21.0 142 237 238 + ansi-regex@5.0.1: {} 239 + 240 + ansi-regex@6.2.2: {} 241 + 143 242 await-lock@2.2.2: {} 144 243 145 - csv-parse@5.6.0: {} 244 + chalk@5.6.2: {} 245 + 246 + cli-cursor@5.0.0: 247 + dependencies: 248 + restore-cursor: 5.1.0 249 + 250 + cli-progress@3.12.0: 251 + dependencies: 252 + string-width: 4.2.3 253 + 254 + cli-spinners@3.4.0: {} 255 + 256 + csv-parse@6.1.0: {} 257 + 258 + emoji-regex@8.0.0: {} 259 + 260 + get-east-asian-width@1.4.0: {} 261 + 262 + is-fullwidth-code-point@3.0.0: {} 263 + 264 + is-interactive@2.0.0: {} 265 + 266 + is-unicode-supported@2.1.0: {} 146 267 147 268 iso-datestring-validator@2.2.2: {} 148 269 270 + log-symbols@7.0.1: 271 + dependencies: 272 + is-unicode-supported: 2.1.0 273 + yoctocolors: 2.1.2 274 + 275 + mimic-function@5.0.1: {} 276 + 149 277 multiformats@9.9.0: {} 150 278 279 + onetime@7.0.0: 280 + dependencies: 281 + mimic-function: 5.0.1 282 + 283 + ora@9.0.0: 284 + dependencies: 285 + chalk: 5.6.2 286 + cli-cursor: 5.0.0 287 + cli-spinners: 3.4.0 288 + is-interactive: 2.0.0 289 + is-unicode-supported: 2.1.0 290 + log-symbols: 7.0.1 291 + stdin-discarder: 0.2.2 292 + string-width: 8.1.0 293 + strip-ansi: 7.1.2 294 + 295 + restore-cursor@5.1.0: 296 + dependencies: 297 + onetime: 7.0.0 298 + signal-exit: 4.1.0 299 + 300 + signal-exit@4.1.0: {} 301 + 302 + stdin-discarder@0.2.2: {} 303 + 304 + string-width@4.2.3: 305 + dependencies: 306 + emoji-regex: 8.0.0 307 + is-fullwidth-code-point: 3.0.0 308 + strip-ansi: 6.0.1 309 + 310 + string-width@8.1.0: 311 + dependencies: 312 + get-east-asian-width: 1.4.0 313 + strip-ansi: 7.1.2 314 + 315 + strip-ansi@6.0.1: 316 + dependencies: 317 + ansi-regex: 5.0.1 318 + 319 + strip-ansi@7.1.2: 320 + dependencies: 321 + ansi-regex: 6.2.2 322 + 151 323 tlds@1.261.0: {} 152 324 153 325 tslib@2.8.1: {} ··· 161 333 undici-types@6.21.0: {} 162 334 163 335 unicode-segmenter@0.14.5: {} 336 + 337 + yoctocolors@2.1.2: {} 164 338 165 339 zod@3.25.76: {}

+5 -16

src/config.ts

··· 20 20 export const RECORD_TYPE = 'fm.teal.alpha.feed.play'; 21 21 22 22 // Build client agent string 23 - export function buildClientAgent(debug = false) { 24 - if (!debug) { 25 - return 'malachite/v0.6.1'; 26 - } 27 - 28 - const PLATFORM_LABELS: Record<string, string> = { 29 - darwin: 'macOS', 30 - linux: 'Linux', 31 - win32: 'Windows', 32 - }; 33 - 34 - const platform = 35 - PLATFORM_LABELS[process.platform] ?? process.platform; 36 - 37 - return `malachite/v0.6.1 (${platform}; Node/${process.version})`; 23 + export function buildClientAgent(_debug = false) { 24 + // Always return just the version, regardless of debug mode 25 + // The debug parameter is kept for backwards compatibility but unused 26 + return 'malachite/v0.6.2'; 38 27 } 39 28 40 29 // Default batch configuration - conservative for PDS safety ··· 51 40 // Slingshot resolver URL 52 41 export const SLINGSHOT_RESOLVER = 'https://slingshot.microcosm.blue'; 53 42 54 - const config = { 43 + const config: Config = { 55 44 RECORD_TYPE, 56 45 MIN_RECORDS_FOR_SCALING: 20, 57 46 BASE_BATCH_SIZE: 200, // Match DEFAULT_BATCH_SIZE for consistency

+89 -87

src/lib/cli.ts

··· 1 1 #!/usr/bin/env node 2 - 3 2 import { parseArgs } from 'node:util'; 4 3 import { AtpAgent } from '@atproto/api'; 5 4 import type { PlayRecord, Config, CommandLineArgs, PublishResult } from '../types.js'; ··· 11 10 import { prompt } from '../utils/input.js'; 12 11 import config from '../config.js'; 13 12 import { calculateOptimalBatchSize } from '../utils/helpers.js'; 14 - import { fetchExistingRecords, filterNewRecords, displaySyncStats, removeDuplicates } from './sync.js'; 13 + import { fetchExistingRecords, filterNewRecords, displaySyncStats, removeDuplicates, deduplicateInputRecords } from './sync.js'; 15 14 import { Logger, LogLevel, setGlobalLogger, log } from '../utils/logger.js'; 15 + import { registerKillswitch } from '../utils/killswitch.js'; 16 + import { clearCache, clearAllCaches } from '../utils/teal-cache.js'; 16 17 import { 17 18 loadImportState, 18 19 createImportState, ··· 26 27 */ 27 28 export function showHelp(): void { 28 29 console.log(` 29 - ${'\x1b[1m'}Last.fm to ATProto Importer v0.6.1${'\x1b[0m'} 30 + ${'\x1b[1m'}Last.fm to ATProto Importer v0.6.2${'\x1b[0m'} 30 31 31 32 ${'\x1b[1m'}USAGE:${'\x1b[0m'} 32 33 npm start [options] ··· 57 58 -y, --yes Skip confirmation prompts 58 59 --dry-run Preview without importing 59 60 --aggressive Faster imports (8,500/day vs 7,500/day default) 60 - --fresh Start fresh (ignore previous import state) 61 + --fresh Start fresh (ignore cache & previous import state) 62 + --clear-cache Clear cached records for current user 63 + --clear-all-caches Clear all cached records 61 64 62 65 ${'\x1b[1m'}OUTPUT:${'\x1b[0m'} 63 66 -v, --verbose Enable verbose logging (debug level) ··· 65 68 --help Show this help message 66 69 67 70 ${'\x1b[1m'}EXAMPLES:${'\x1b[0m'} 68 - 69 71 ${'\x1b[2m'}# Import Last.fm export${'\x1b[0m'} 70 72 npm start -- -i lastfm-export.csv -h user.bsky.social -p app-password 71 73 ··· 84 86 ${'\x1b[2m'}# Remove duplicate records${'\x1b[0m'} 85 87 npm start -- -m deduplicate -h user.bsky.social -p app-password 86 88 89 + ${'\x1b[2m'}# Clear cache for current user${'\x1b[0m'} 90 + npm start -- --clear-cache -h user.bsky.social -p app-password 91 + 92 + ${'\x1b[2m'}# Clear all caches${'\x1b[0m'} 93 + npm start -- --clear-all-caches 94 + 87 95 ${'\x1b[1m'}NOTES:${'\x1b[0m'} 88 96 • Rate limits: Max 10,000 records/day to avoid PDS rate limiting 89 97 • Import will auto-pause between days for large datasets ··· 101 109 */ 102 110 export function parseCommandLineArgs(): CommandLineArgs { 103 111 const options = { 104 - // Help 105 112 help: { type: 'boolean', default: false }, 106 - 107 - // Authentication 108 113 handle: { type: 'string', short: 'h' }, 109 114 password: { type: 'string', short: 'p' }, 110 - 111 - // Input 112 115 input: { type: 'string', short: 'i' }, 113 116 'spotify-input': { type: 'string' }, 114 - 115 - // Mode 116 117 mode: { type: 'string', short: 'm' }, 117 - 118 - // Batch configuration 119 118 'batch-size': { type: 'string', short: 'b' }, 120 119 'batch-delay': { type: 'string', short: 'd' }, 121 - 122 - // Import options 123 120 reverse: { type: 'boolean', short: 'r', default: false }, 124 121 yes: { type: 'boolean', short: 'y', default: false }, 125 122 'dry-run': { type: 'boolean', default: false }, 126 123 aggressive: { type: 'boolean', default: false }, 127 124 fresh: { type: 'boolean', default: false }, 128 - 129 - // Output 125 + 'clear-cache': { type: 'boolean', default: false }, 126 + 'clear-all-caches': { type: 'boolean', default: false }, 130 127 verbose: { type: 'boolean', short: 'v', default: false }, 131 128 quiet: { type: 'boolean', short: 'q', default: false }, 132 - 133 - // Legacy flags for backwards compatibility (hidden from help) 134 - file: { type: 'string', short: 'f' }, // Maps to --input 135 - 'spotify-file': { type: 'string' }, // Maps to --spotify-input 136 - identifier: { type: 'string' }, // Maps to --handle 137 - 'reverse-chronological': { type: 'boolean' }, // Maps to --reverse 138 - sync: { type: 'boolean', short: 's' }, // Maps to --mode sync 139 - spotify: { type: 'boolean' }, // Maps to --mode spotify 140 - combined: { type: 'boolean' }, // Maps to --mode combined 141 - 'remove-duplicates': { type: 'boolean' }, // Maps to --mode deduplicate 129 + file: { type: 'string', short: 'f' }, 130 + 'spotify-file': { type: 'string' }, 131 + identifier: { type: 'string' }, 132 + 'reverse-chronological': { type: 'boolean' }, 133 + sync: { type: 'boolean', short: 's' }, 134 + spotify: { type: 'boolean' }, 135 + combined: { type: 'boolean' }, 136 + 'remove-duplicates': { type: 'boolean' }, 142 137 } as const; 143 138 144 139 try { 145 140 const { values } = parseArgs({ options, allowPositionals: false }); 146 - 147 - // Handle legacy flag mappings 148 141 const normalizedArgs: CommandLineArgs = { 149 142 help: values.help, 150 143 handle: values.handle || values.identifier, ··· 158 151 'dry-run': values['dry-run'], 159 152 aggressive: values.aggressive, 160 153 fresh: values.fresh, 154 + 'clear-cache': values['clear-cache'], 155 + 'clear-all-caches': values['clear-all-caches'], 161 156 verbose: values.verbose, 162 157 quiet: values.quiet, 163 158 }; 164 - 165 - // Determine mode from new --mode flag or legacy flags 159 + 166 160 if (values.mode) { 167 161 normalizedArgs.mode = values.mode; 168 162 } else if (values['remove-duplicates']) { ··· 174 168 } else if (values.spotify) { 175 169 normalizedArgs.mode = 'spotify'; 176 170 } else { 177 - normalizedArgs.mode = 'lastfm'; // default 171 + normalizedArgs.mode = 'lastfm'; 178 172 } 179 - 173 + 180 174 return normalizedArgs; 181 175 } catch (error) { 182 176 const err = error as Error; ··· 192 186 function validateMode(mode: string): 'lastfm' | 'spotify' | 'combined' | 'sync' | 'deduplicate' { 193 187 const validModes = ['lastfm', 'spotify', 'combined', 'sync', 'deduplicate']; 194 188 const normalized = mode.toLowerCase(); 195 - 196 189 if (!validModes.includes(normalized)) { 197 - throw new Error( 198 - `Invalid mode: ${mode}. Must be one of: ${validModes.join(', ')}` 199 - ); 190 + throw new Error(`Invalid mode: ${mode}. Must be one of: ${validModes.join(', ')}`); 200 191 } 201 - 202 192 return normalized as 'lastfm' | 'spotify' | 'combined' | 'sync' | 'deduplicate'; 203 193 } 204 194 ··· 207 197 */ 208 198 export async function runCLI(): Promise<void> { 209 199 try { 200 + registerKillswitch(); 210 201 const args = parseCommandLineArgs(); 211 202 const cfg = config as Config; 203 + let agent: AtpAgent | null = null; 212 204 213 - // Setup logging 214 205 const logger = new Logger( 215 206 args.quiet ? LogLevel.WARN : 216 207 args.verbose ? LogLevel.DEBUG : ··· 223 214 return; 224 215 } 225 216 226 - // Validate and normalize mode 217 + if (args['clear-all-caches']) { 218 + log.section('Clear All Caches'); 219 + clearAllCaches(); 220 + log.success('All caches cleared successfully'); 221 + return; 222 + } 223 + 224 + if (args['clear-cache']) { 225 + if (!args.handle || !args.password) { 226 + throw new Error('--clear-cache requires --handle and --password to identify the cache'); 227 + } 228 + log.section('Clear Cache'); 229 + log.info('Authenticating to identify cache...'); 230 + agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 231 + const did = agent.session?.did; 232 + if (!did) { 233 + throw new Error('Failed to get DID from session'); 234 + } 235 + clearCache(did); 236 + log.success(`Cache cleared for ${args.handle} (${did})`); 237 + return; 238 + } 239 + 227 240 const mode = validateMode(args.mode || 'lastfm'); 228 241 const dryRun = args['dry-run'] ?? false; 229 - let agent: AtpAgent | null = null; 230 242 231 243 log.debug(`Mode: ${mode}`); 232 244 log.debug(`Dry run: ${dryRun}`); 233 245 log.debug(`Log level: ${args.verbose ? 'DEBUG' : args.quiet ? 'WARN' : 'INFO'}`); 234 246 235 - // Validate mode-specific requirements 236 247 if (mode === 'combined') { 237 248 if (!args.input || !args['spotify-input']) { 238 249 throw new Error('Combined mode requires both --input (Last.fm) and --spotify-input (Spotify)'); ··· 241 252 throw new Error('Missing required argument: --input <path>'); 242 253 } 243 254 244 - // Deduplicate mode 245 255 if (mode === 'deduplicate') { 246 256 if (!args.handle || !args.password) { 247 257 throw new Error('Deduplicate mode requires --handle and --password'); 248 258 } 249 - 250 259 log.section('Remove Duplicate Records'); 251 260 agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 252 - 253 261 const result = await removeDuplicates(agent, cfg, true); 254 - 255 262 if (result.totalDuplicates === 0) { 256 263 return; 257 264 } 258 - 259 265 if (!dryRun && !args.yes) { 260 266 log.warn(`This will permanently delete ${result.totalDuplicates} duplicate records from Teal.`); 261 267 log.info('The first occurrence of each duplicate will be kept.'); ··· 265 271 log.info('Duplicate removal cancelled by user.'); 266 272 process.exit(0); 267 273 } 268 - 269 274 await removeDuplicates(agent, cfg, false); 270 275 log.success('Duplicate removal complete!'); 271 276 } else if (dryRun) { 272 277 log.info('DRY RUN: No records were actually removed.'); 273 278 log.info('Remove --dry-run flag to actually delete duplicates.'); 274 279 } 275 - 276 280 return; 277 281 } 278 282 279 - // Authentication (required for sync mode, even in dry-run) 280 - if (!dryRun || mode === 'sync') { 281 - if (!args.handle || !args.password) { 282 - throw new Error('Missing required arguments: --handle and --password'); 283 - } 284 - log.debug('Authenticating...'); 285 - agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 286 - log.debug('Authentication successful'); 283 + if (!args.handle || !args.password) { 284 + throw new Error('Missing required arguments: --handle and --password'); 287 285 } 286 + log.debug('Authenticating...'); 287 + agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 288 + log.debug('Authentication successful'); 288 289 289 - // Parse and prepare records 290 290 log.section('Loading Records'); 291 291 let records: PlayRecord[]; 292 292 let rawRecordCount: number; 293 - 294 293 const isDebug = args.verbose ?? false; 295 294 296 295 if (mode === 'combined') { ··· 311 310 312 311 log.success(`Loaded ${rawRecordCount.toLocaleString()} records`); 313 312 314 - // Sync mode: filter existing records 315 - if (mode === 'sync' && agent) { 316 - log.section('Sync Mode'); 317 - log.info('Checking for existing records...'); 313 + const dedupResult = deduplicateInputRecords(records); 314 + records = dedupResult.unique; 315 + if (dedupResult.duplicates > 0) { 316 + log.warn(`Removed ${dedupResult.duplicates.toLocaleString()} duplicate(s) from input data`); 317 + log.info(`Unique records: ${records.length.toLocaleString()}`); 318 + } else { 319 + log.info(`No duplicates found in input data`); 320 + } 321 + log.blank(); 322 + 323 + if (agent) { 318 324 const originalRecords = [...records]; 319 - const existingRecords = await fetchExistingRecords(agent, cfg); 325 + const existingRecords = await fetchExistingRecords(agent, cfg, args.fresh ?? false); 320 326 records = filterNewRecords(records, existingRecords); 321 - 322 327 if (records.length === 0) { 323 328 log.success('All records already exist in Teal. Nothing to import!'); 324 329 process.exit(0); 325 330 } 326 - 327 - displaySyncStats(originalRecords, existingRecords, records); 331 + if (mode === 'sync' || mode === 'combined') { 332 + displaySyncStats(originalRecords, existingRecords, records); 333 + } else { 334 + const skipped = originalRecords.length - records.length; 335 + if (skipped > 0) { 336 + log.info(`Found ${skipped.toLocaleString()} record(s) already in Teal (skipping)`); 337 + log.info(`New records to import: ${records.length.toLocaleString()}`); 338 + } else { 339 + log.info(`All ${records.length.toLocaleString()} records are new`); 340 + } 341 + log.blank(); 342 + } 328 343 } 329 344 330 345 const totalRecords = records.length; 331 346 332 - // Sort records (skip for combined mode as it already sorts) 333 347 if (mode !== 'combined') { 334 348 log.debug(`Sorting records (reverse: ${args.reverse})...`); 335 349 records = mode === 'spotify' ··· 337 351 : sortRecords(records, args.reverse ?? false); 338 352 } 339 353 340 - // Determine batch parameters 341 354 log.section('Batch Configuration'); 342 - 343 355 let batchDelay = cfg.DEFAULT_BATCH_DELAY; 344 356 if (args['batch-delay']) { 345 357 const delay = parseInt(args['batch-delay'], 10); ··· 366 378 367 379 log.info(`Batch delay: ${batchDelay}ms`); 368 380 369 - // Apply aggressive mode if enabled 370 381 const safetyMargin = args.aggressive ? cfg.AGGRESSIVE_SAFETY_MARGIN : cfg.SAFETY_MARGIN; 371 382 if (args.aggressive) { 372 383 log.warn('⚡ Aggressive mode enabled: Using 85% of daily limit (8,500 records/day)'); 373 384 } 374 385 375 - // Show rate limiting information 376 386 log.section('Import Configuration'); 377 387 log.info(`Total records: ${totalRecords.toLocaleString()}`); 378 388 log.info(`Batch size: ${batchSize} records`); 379 389 log.info(`Batch delay: ${batchDelay}ms`); 380 - 390 + 381 391 const recordsPerDay = cfg.RECORDS_PER_DAY_LIMIT * safetyMargin; 382 392 const estimatedDays = Math.ceil(totalRecords / recordsPerDay); 383 - 384 393 if (estimatedDays > 1) { 385 394 log.info(`Duration: ${estimatedDays} days (${recordsPerDay.toLocaleString()} records/day limit)`); 386 395 log.warn('Large import will span multiple days with automatic pauses'); 387 396 } 388 - 389 397 log.blank(); 390 398 391 - // Check for existing import state (resume functionality) 392 399 let importState: ImportState | null = null; 393 400 if (!dryRun && args.input) { 394 - // Clear state if --fresh flag is used 395 401 if (args.fresh) { 396 402 clearImportState(args.input, mode); 397 403 log.info('Starting fresh import (previous state cleared)'); 398 404 } else { 399 - // Try to load existing state 400 405 importState = loadImportState(args.input, mode); 401 - 402 406 if (importState && !importState.completed) { 403 407 displayResumeInfo(importState); 404 - 405 408 if (!args.yes) { 406 409 const answer = await prompt('Resume from previous import? (Y/n) '); 407 410 if (answer.toLowerCase() === 'n') { ··· 420 423 clearImportState(args.input, mode); 421 424 } 422 425 } 423 - 424 - // Create new state if not resuming 425 426 if (!importState) { 426 427 importState = createImportState(args.input, mode, totalRecords); 427 428 log.debug('Created new import state'); 428 429 } 429 430 } 430 431 431 - // Confirmation prompt 432 432 if (!dryRun && !args.yes) { 433 433 const modeLabel = mode === 'combined' ? 'merged' : mode === 'sync' ? 'new' : ''; 434 434 const skippedInfo = mode === 'sync' ? ` (${rawRecordCount - totalRecords} skipped)` : ''; ··· 441 441 log.blank(); 442 442 } 443 443 444 - // Publish records 445 444 log.section('Publishing Records'); 446 445 const result: PublishResult = await publishRecordsWithApplyWrites( 447 446 agent, ··· 454 453 importState 455 454 ); 456 455 457 - // Final output 458 456 log.blank(); 459 457 if (result.cancelled) { 460 458 log.warn(`Stopped: ${result.successCount.toLocaleString()} processed`); ··· 472 470 } 473 471 } 474 472 } 475 - 476 473 } catch (error) { 477 474 const err = error as Error; 475 + if (err.message === 'Operation cancelled by user') { 476 + log.blank(); 477 + log.warn('Operation cancelled by user'); 478 + process.exit(0); 479 + } 478 480 log.blank(); 479 481 log.fatal('A fatal error occurred:'); 480 482 log.error(err.message);

+189 -23

src/lib/sync.ts

··· 3 3 import { formatDate, formatDateRange } from '../utils/helpers.js'; 4 4 import * as ui from '../utils/ui.js'; 5 5 import { log } from '../utils/logger.js'; 6 + import { isImportCancelled } from '../utils/killswitch.js'; 7 + import { isCacheValid, loadCache, saveCache, getCacheInfo } from '../utils/teal-cache.js'; 6 8 7 9 interface ExistingRecord { 8 10 uri: string; ··· 21 23 */ 22 24 export async function fetchExistingRecords( 23 25 agent: AtpAgent, 24 - config: Config 26 + config: Config, 27 + forceRefresh: boolean = false 25 28 ): Promise<Map<string, ExistingRecord>> { 26 - log.section('Fetching Existing Teal Records'); 29 + log.section('Checking Existing Records'); 27 30 const { RECORD_TYPE } = config; 28 31 const did = agent.session?.did; 29 32 ··· 31 34 throw new Error('No authenticated session found'); 32 35 } 33 36 37 + // Check cache first (unless force refresh) 38 + if (!forceRefresh && isCacheValid(did)) { 39 + const cacheInfo = getCacheInfo(did); 40 + log.info(`📂 Loading from cache (${cacheInfo.age!.toFixed(1)}h old, ${cacheInfo.records!.toLocaleString()} records)...`); 41 + 42 + const cached = loadCache(did); 43 + if (cached) { 44 + // Convert cached records to the format we need 45 + const existingRecords = new Map<string, ExistingRecord>(); 46 + for (const [, record] of cached.entries()) { 47 + const playRecord = record.value as PlayRecord; 48 + const key = createRecordKey(playRecord); 49 + existingRecords.set(key, record as ExistingRecord); 50 + } 51 + 52 + log.success(`✓ Loaded ${existingRecords.size.toLocaleString()} records from cache`); 53 + log.blank(); 54 + return existingRecords; 55 + } 56 + } 57 + 58 + // Cache miss or force refresh - fetch from Teal 59 + if (forceRefresh) { 60 + log.info('🔄 Force refresh - fetching from Teal...'); 61 + } else { 62 + log.info('Fetching records from Teal to avoid duplicates...'); 63 + } 64 + 34 65 const existingRecords = new Map<string, ExistingRecord>(); 66 + const cacheMap = new Map<string, { uri: string; cid: string; value: any }>(); 35 67 let cursor: string | undefined = undefined; 36 68 let totalFetched = 0; 69 + const startTime = Date.now(); 70 + 71 + // Adaptive batch sizing 72 + let batchSize = 25; // Start conservative 73 + let consecutiveFastRequests = 0; 74 + let consecutiveSlowRequests = 0; 75 + const TARGET_LATENCY_MS = 2000; // Target 2s per request 76 + const MIN_BATCH_SIZE = 10; 77 + const MAX_BATCH_SIZE = 100; // AT Protocol maximum 78 + let requestCount = 0; 37 79 38 80 try { 39 - // Fetch records in batches using listRecords 81 + // Fetch records in batches using listRecords with adaptive sizing 40 82 do { 83 + // Check for cancellation 84 + if (isImportCancelled()) { 85 + log.warn('Fetch cancelled by user'); 86 + throw new Error('Operation cancelled by user'); 87 + } 88 + 89 + requestCount++; 90 + const requestStart = Date.now(); 91 + 92 + log.debug(`Request #${requestCount}: Fetching batch of ${batchSize}...`); 93 + 41 94 const response = await agent.com.atproto.repo.listRecords({ 42 95 repo: did, 43 96 collection: RECORD_TYPE, 44 - limit: 100, 97 + limit: batchSize, 45 98 cursor: cursor, 46 99 }); 47 100 48 - for (const record of response.data.records) { 101 + const requestLatency = Date.now() - requestStart; 102 + const records = response.data.records; 103 + 104 + log.debug(`Request #${requestCount}: Got ${records.length} records in ${requestLatency}ms`); 105 + 106 + // Batch process records for better performance 107 + for (const record of records) { 49 108 const playRecord = record.value as unknown as PlayRecord; 50 - // Create a unique key based on track, artist, and timestamp 51 109 const key = createRecordKey(playRecord); 52 - // Note: This will overwrite duplicates, but that's OK for sync mode 53 - // For duplicate detection, we'll need to fetch all records again 54 - existingRecords.set(key, { 110 + const existingRecord = { 55 111 uri: record.uri, 56 112 cid: record.cid, 57 113 value: playRecord, 58 - }); 114 + }; 115 + existingRecords.set(key, existingRecord); 116 + // Also store for cache (using URI as key for cache) 117 + cacheMap.set(record.uri, existingRecord); 59 118 } 60 119 61 - totalFetched += response.data.records.length; 120 + totalFetched += records.length; 62 121 cursor = response.data.cursor; 63 122 64 - // Show progress 65 - if (totalFetched % 500 === 0 && totalFetched > 0) { 66 - log.progress(`Fetched ${totalFetched.toLocaleString()} records...`); 123 + // Adaptive batch size adjustment based on latency 124 + if (requestLatency < TARGET_LATENCY_MS) { 125 + // Request was fast - try to increase batch size 126 + consecutiveFastRequests++; 127 + consecutiveSlowRequests = 0; 128 + 129 + if (consecutiveFastRequests >= 3 && batchSize < MAX_BATCH_SIZE) { 130 + const oldSize = batchSize; 131 + batchSize = Math.min(MAX_BATCH_SIZE, Math.floor(batchSize * 1.5)); 132 + if (oldSize !== batchSize) { 133 + log.info(`⚡ Network performing well - increased batch size: ${oldSize} → ${batchSize}`); 134 + } 135 + consecutiveFastRequests = 0; 136 + } 137 + } else { 138 + // Request was slow - decrease batch size 139 + consecutiveSlowRequests++; 140 + consecutiveFastRequests = 0; 141 + 142 + if (consecutiveSlowRequests >= 2 && batchSize > MIN_BATCH_SIZE) { 143 + const oldSize = batchSize; 144 + batchSize = Math.max(MIN_BATCH_SIZE, Math.floor(batchSize * 0.7)); 145 + log.info(`🐌 Network slow - decreased batch size: ${oldSize} → ${batchSize}`); 146 + consecutiveSlowRequests = 0; 147 + } 148 + } 149 + 150 + // Show progress every 250 records or every request if less than 1000 total 151 + const showProgress = totalFetched % 250 === 0 && totalFetched > 0; 152 + if (showProgress || totalFetched < 1000) { 153 + const elapsed = ((Date.now() - startTime) / 1000).toFixed(1); 154 + const rate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0); 155 + log.progress(`Fetched ${totalFetched.toLocaleString()} records (${rate} rec/s, batch: ${batchSize}, ${elapsed}s)...`); 67 156 } 68 157 } while (cursor); 69 158 70 - log.success(`Found ${existingRecords.size.toLocaleString()} existing records`); 159 + const elapsed = ((Date.now() - startTime) / 1000).toFixed(1); 160 + const avgRate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0); 161 + log.success(`Found ${existingRecords.size.toLocaleString()} existing records in ${elapsed}s (avg ${avgRate} rec/s)`); 162 + 163 + // Save to cache 164 + log.debug('Saving records to cache...'); 165 + saveCache(did, cacheMap); 166 + 71 167 log.blank(); 72 168 return existingRecords; 73 169 } catch (error) { ··· 96 192 const allRecords: ExistingRecord[] = []; 97 193 let cursor: string | undefined = undefined; 98 194 let totalFetched = 0; 195 + const startTime = Date.now(); 196 + 197 + // Adaptive batch sizing 198 + let batchSize = 25; // Start conservative 199 + let consecutiveFastRequests = 0; 200 + let consecutiveSlowRequests = 0; 201 + const TARGET_LATENCY_MS = 2000; // Target 2s per request 202 + const MIN_BATCH_SIZE = 10; 203 + const MAX_BATCH_SIZE = 100; // AT Protocol maximum 204 + let requestCount = 0; 99 205 100 206 try { 101 - // Fetch records in batches using listRecords 207 + // Fetch records in batches using listRecords with adaptive sizing 102 208 do { 209 + // Check for cancellation 210 + if (isImportCancelled()) { 211 + ui.failSpinner('Fetch cancelled by user'); 212 + throw new Error('Operation cancelled by user'); 213 + } 214 + 215 + requestCount++; 216 + const requestStart = Date.now(); 217 + 103 218 const response = await agent.com.atproto.repo.listRecords({ 104 219 repo: did, 105 220 collection: RECORD_TYPE, 106 - limit: 100, 221 + limit: batchSize, 107 222 cursor: cursor, 108 223 }); 109 224 110 - for (const record of response.data.records) { 225 + const requestLatency = Date.now() - requestStart; 226 + const records = response.data.records; 227 + for (const record of records) { 111 228 const playRecord = record.value as unknown as PlayRecord; 112 229 allRecords.push({ 113 230 uri: record.uri, ··· 116 233 }); 117 234 } 118 235 119 - totalFetched += response.data.records.length; 236 + totalFetched += records.length; 120 237 cursor = response.data.cursor; 121 238 122 - // Update spinner with progress 123 - if (totalFetched % 500 === 0 && totalFetched > 0) { 124 - ui.updateSpinner(`Fetching records... ${totalFetched.toLocaleString()} found`); 239 + // Adaptive batch size adjustment based on latency 240 + if (requestLatency < TARGET_LATENCY_MS) { 241 + // Request was fast - try to increase batch size 242 + consecutiveFastRequests++; 243 + consecutiveSlowRequests = 0; 244 + 245 + if (consecutiveFastRequests >= 3 && batchSize < MAX_BATCH_SIZE) { 246 + batchSize = Math.min(MAX_BATCH_SIZE, Math.floor(batchSize * 1.5)); 247 + consecutiveFastRequests = 0; 248 + } 249 + } else { 250 + // Request was slow - decrease batch size 251 + consecutiveSlowRequests++; 252 + consecutiveFastRequests = 0; 253 + 254 + if (consecutiveSlowRequests >= 2 && batchSize > MIN_BATCH_SIZE) { 255 + batchSize = Math.max(MIN_BATCH_SIZE, Math.floor(batchSize * 0.7)); 256 + consecutiveSlowRequests = 0; 257 + } 258 + } 259 + 260 + // Update spinner with progress every 250 records or every request if less than 1000 total 261 + const showProgress = totalFetched % 250 === 0 && totalFetched > 0; 262 + if (showProgress || totalFetched < 1000) { 263 + const elapsed = ((Date.now() - startTime) / 1000).toFixed(1); 264 + const rate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0); 265 + ui.updateSpinner(`Fetching records... ${totalFetched.toLocaleString()} found (${rate} rec/s, batch: ${batchSize}, ${elapsed}s)`); 125 266 } 126 267 } while (cursor); 127 268 128 - ui.succeedSpinner(`Found ${allRecords.length.toLocaleString()} total records`); 269 + const elapsed = ((Date.now() - startTime) / 1000).toFixed(1); 270 + const avgRate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0); 271 + ui.succeedSpinner(`Found ${allRecords.length.toLocaleString()} total records in ${elapsed}s (avg ${avgRate} rec/s)`); 129 272 return allRecords; 130 273 } catch (error) { 131 274 ui.failSpinner('Failed to fetch existing records'); ··· 147 290 const normalizedTrack = track.toLowerCase().trim(); 148 291 149 292 return `${normalizedArtist}|||${normalizedTrack}|||${timestamp}`; 293 + } 294 + 295 + /** 296 + * Deduplicate input records before submission 297 + * Keeps the first occurrence of each duplicate 298 + */ 299 + export function deduplicateInputRecords(records: PlayRecord[]): { unique: PlayRecord[]; duplicates: number } { 300 + const seen = new Map<string, PlayRecord>(); 301 + const duplicates: PlayRecord[] = []; 302 + 303 + for (const record of records) { 304 + const key = createRecordKey(record); 305 + if (!seen.has(key)) { 306 + seen.set(key, record); 307 + } else { 308 + duplicates.push(record); 309 + } 310 + } 311 + 312 + return { 313 + unique: Array.from(seen.values()), 314 + duplicates: duplicates.length 315 + }; 150 316 } 151 317 152 318 /**

+2

src/types.ts

··· 59 59 'dry-run'?: boolean; 60 60 aggressive?: boolean; 61 61 fresh?: boolean; 62 + 'clear-cache'?: boolean; 63 + 'clear-all-caches'?: boolean; 62 64 63 65 // Output 64 66 verbose?: boolean;

+24 -5

src/utils/killswitch.ts

··· 1 1 let cancelled = false; 2 + let sigintHandlerRegistered = false; 2 3 3 - // Flip the killswitch when the user hits CTRL-C 4 - process.on('SIGINT', () => { 5 - console.log('\nCaught CTRL-C — stopping import…'); 6 - cancelled = true; 7 - }); 4 + /** 5 + * Register the SIGINT handler (should be called early in the application) 6 + */ 7 + export function registerKillswitch(): void { 8 + if (sigintHandlerRegistered) { 9 + return; 10 + } 11 + 12 + // Flip the killswitch when the user hits CTRL-C 13 + process.on('SIGINT', () => { 14 + if (cancelled) { 15 + // If already cancelled and user presses Ctrl+C again, force exit 16 + console.log('\n\nForce quit detected. Exiting immediately...'); 17 + process.exit(1); 18 + } 19 + 20 + console.log('\n\n⚠️ Ctrl+C detected — stopping after current batch completes...'); 21 + console.log('Press Ctrl+C again to force quit immediately.\n'); 22 + cancelled = true; 23 + }); 24 + 25 + sigintHandlerRegistered = true; 26 + } 8 27 9 28 /** 10 29 * Manually cancel the import if needed.

+166

src/utils/teal-cache.ts

··· 1 + import { existsSync, readFileSync, writeFileSync, mkdirSync, readdirSync, unlinkSync } from 'node:fs'; 2 + import { join } from 'node:path'; 3 + import { homedir } from 'node:os'; 4 + 5 + /** 6 + * Cache configuration 7 + */ 8 + const CACHE_VERSION = 1; 9 + const CACHE_TTL_HOURS = 24; // Cache validity period 10 + const CACHE_DIR = join(homedir(), '.malachite', 'cache'); 11 + 12 + /** 13 + * Cache file structure 14 + */ 15 + interface CacheFile { 16 + version: number; 17 + did: string; 18 + timestamp: number; 19 + records: Array<[string, { uri: string; cid: string; value: any }]>; 20 + } 21 + 22 + /** 23 + * Ensure cache directory exists 24 + */ 25 + function ensureCacheDir(): void { 26 + if (!existsSync(CACHE_DIR)) { 27 + mkdirSync(CACHE_DIR, { recursive: true }); 28 + } 29 + } 30 + 31 + /** 32 + * Get cache file path for a DID 33 + */ 34 + function getCachePath(did: string): string { 35 + // Sanitize DID for use in filename 36 + const sanitized = did.replace(/[^a-zA-Z0-9.-]/g, '_'); 37 + return join(CACHE_DIR, `${sanitized}.json`); 38 + } 39 + 40 + /** 41 + * Check if cache exists and is valid for a given DID 42 + */ 43 + export function isCacheValid(did: string): boolean { 44 + const cachePath = getCachePath(did); 45 + 46 + if (!existsSync(cachePath)) { 47 + return false; 48 + } 49 + 50 + try { 51 + const data = readFileSync(cachePath, 'utf-8'); 52 + const cache: CacheFile = JSON.parse(data); 53 + 54 + // Check version 55 + if (cache.version !== CACHE_VERSION) { 56 + return false; 57 + } 58 + 59 + // Check DID match 60 + if (cache.did !== did) { 61 + return false; 62 + } 63 + 64 + // Check age 65 + const ageHours = (Date.now() - cache.timestamp) / (1000 * 60 * 60); 66 + if (ageHours > CACHE_TTL_HOURS) { 67 + return false; 68 + } 69 + 70 + return true; 71 + } catch { 72 + // Invalid cache file 73 + return false; 74 + } 75 + } 76 + 77 + /** 78 + * Load cached records for a given DID 79 + * Returns null if cache doesn't exist or is invalid 80 + */ 81 + export function loadCache(did: string): Map<string, { uri: string; cid: string; value: any }> | null { 82 + const cachePath = getCachePath(did); 83 + 84 + if (!isCacheValid(did)) { 85 + return null; 86 + } 87 + 88 + try { 89 + const data = readFileSync(cachePath, 'utf-8'); 90 + const cache: CacheFile = JSON.parse(data); 91 + 92 + // Convert array back to Map 93 + return new Map(cache.records); 94 + } catch { 95 + return null; 96 + } 97 + } 98 + 99 + /** 100 + * Save records to cache for a given DID 101 + */ 102 + export function saveCache(did: string, records: Map<string, { uri: string; cid: string; value: any }>): void { 103 + ensureCacheDir(); 104 + 105 + const cache: CacheFile = { 106 + version: CACHE_VERSION, 107 + did, 108 + timestamp: Date.now(), 109 + records: Array.from(records.entries()), 110 + }; 111 + 112 + const cachePath = getCachePath(did); 113 + writeFileSync(cachePath, JSON.stringify(cache), 'utf-8'); 114 + } 115 + 116 + /** 117 + * Get cache information (age and record count) 118 + */ 119 + export function getCacheInfo(did: string): { age?: number; records?: number } { 120 + const cachePath = getCachePath(did); 121 + 122 + if (!existsSync(cachePath)) { 123 + return {}; 124 + } 125 + 126 + try { 127 + const data = readFileSync(cachePath, 'utf-8'); 128 + const cache: CacheFile = JSON.parse(data); 129 + 130 + const ageHours = (Date.now() - cache.timestamp) / (1000 * 60 * 60); 131 + 132 + return { 133 + age: ageHours, 134 + records: cache.records.length, 135 + }; 136 + } catch { 137 + return {}; 138 + } 139 + } 140 + 141 + /** 142 + * Clear cache for a specific DID 143 + */ 144 + export function clearCache(did: string): void { 145 + const cachePath = getCachePath(did); 146 + 147 + if (existsSync(cachePath)) { 148 + unlinkSync(cachePath); 149 + } 150 + } 151 + 152 + /** 153 + * Clear all caches 154 + */ 155 + export function clearAllCaches(): void { 156 + if (!existsSync(CACHE_DIR)) { 157 + return; 158 + } 159 + 160 + const files = readdirSync(CACHE_DIR); 161 + for (const file of files) { 162 + if (file.endsWith('.json')) { 163 + unlinkSync(join(CACHE_DIR, file)); 164 + } 165 + } 166 + }

Configure Feed

Configure Feed