Import your Last.fm and Spotify listening history to the AT Protocol network using the fm.teal.alpha.feed.play lexicon.
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

feat: add Spotify import support and TID-based record keys

- Add support for Spotify Extended Streaming History JSON exports
- Accept single files or directories
- Filter out podcasts and non-music content
- Introduce --spotify CLI flag and update help text
- Generate TID-based rkeys from playedTime for chronological ordering
- Update importer metadata and client agent to v0.3.0
- Clarify MusicBrainz support as Last.fm-only
- Expand README with Spotify setup, examples, and record schema

+281 -36
+82 -20
README.md
··· 7 7 ## Features 8 8 9 9 - ✅ **Batch Operations**: Uses `com.atproto.repo.applyWrites` for efficient batch publishing (up to 200 records per call) 10 - 10 + - ✅ **Spotify Support**: Import from Spotify Extended Streaming History (JSON format) 11 + - ✅ **TID-based Record Keys**: Records use timestamp-based identifiers for chronological ordering 11 12 - ✅ **Re-Sync Mode**: Check existing Teal records and only import new scrobbles (no duplicates!) 12 13 - ✅ **Rate Limiting**: Automatically limits imports to 1K records per day to prevent rate limiting your entire PDS 13 14 - ✅ **Multi-Day Imports**: Large imports (>1K records) automatically span multiple days with 24-hour pauses ··· 19 20 - ✅ **Batch Processing**: Configurable batching with rate limit safety 20 21 - ✅ **Progress Tracking**: Real-time progress with time estimates 21 22 - ✅ **Error Handling**: Continues on errors with detailed reporting 22 - - ✅ **MusicBrainz Support**: Preserves MusicBrainz IDs when available 23 + - ✅ **MusicBrainz Support**: Preserves MusicBrainz IDs when available (Last.fm only) 23 24 - ✅ **Chronological Ordering**: Processes oldest first (or newest with `-r` flag) 24 25 25 26 ## Important: Rate Limits ··· 82 83 For automation or scripting, provide all parameters via flags: 83 84 84 85 ```bash 85 - # Full automation 86 + # Full automation (Last.fm) 86 87 npm start -- -f lastfm.csv -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 88 + 89 + # Import from Spotify (single file) 90 + npm start -- -f Streaming_History_Audio_2021-2023_0.json --spotify -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 91 + 92 + # Import from Spotify (directory with multiple files - recommended) 93 + npm start -- -f '/path/to/Spotify Extended Streaming History' --spotify -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 87 94 88 95 # Preview without publishing 89 96 npm start -- -f lastfm.csv --dry-run 90 97 98 + # Preview Spotify import 99 + npm start -- -f '/path/to/Spotify Extended Streaming History' --spotify --dry-run 100 + 91 101 # Custom batch settings (advanced users) 92 102 npm start -- -f lastfm.csv -i alice.bsky.social -b 20 -d 3000 93 103 ··· 100 110 | Option | Short | Description | Default | 101 111 |--------|-------|-------------|---------| 102 112 | `--help` | `-h` | Show help message | - | 103 - | `--file <path>` | `-f` | Path to Last.fm CSV export file | (prompted) | 113 + | `--file <path>` | `-f` | Path to Last.fm CSV or Spotify JSON file/directory | (prompted) | 114 + | `--spotify` | | Import from Spotify JSON export instead of Last.fm | false | 104 115 | `--identifier <id>` | `-i` | ATProto handle or DID | (prompted) | 105 116 | `--password <pass>` | `-p` | ATProto app password | (prompted) | 106 117 | `--batch-size <num>` | `-b` | Records per batch | Auto-calculated | ··· 124 135 125 136 ⚠️ Lower delays increase speed but risk hitting rate limits. The automatic calculation is recommended. 126 137 127 - ## Getting Your Last.fm Data 138 + ## Getting Your Data 139 + 140 + ### Last.fm Export 128 141 129 142 1. Go to <https://mainstream.ghan.nl/export.html> 130 143 2. Request your data export in CSV format 131 144 3. Download the CSV file when ready 132 145 4. Use the CSV file path with this script 133 146 147 + ### Spotify Export 148 + 149 + 1. Go to your [Spotify Privacy Settings](https://www.spotify.com/account/privacy/) 150 + 2. Scroll down to "Download your data" and request your data 151 + 3. Select "Extended streaming history" (this can take up to 30 days) 152 + 4. When ready, download and extract the ZIP file 153 + 5. Use either: 154 + - A single JSON file: `Streaming_History_Audio_2021-2023_0.json` 155 + - The entire extracted directory (recommended) 156 + 157 + **Note**: Spotify exports include multiple JSON files. The importer automatically: 158 + - Reads all `Streaming_History_Audio_*.json` files in a directory 159 + - Filters out podcasts, audiobooks, and other non-music content 160 + - Combines all music tracks into a single import 161 + 134 162 ## What Gets Imported 135 163 136 - Each Last.fm scrobble becomes an `fm.teal.alpha.feed.play` record with: 164 + Each scrobble (from Last.fm or Spotify) becomes an `fm.teal.alpha.feed.play` record with: 137 165 138 166 ### Required Fields 139 167 - **trackName**: The name of the track 140 - - **artists**: Array of artist objects (requires `artistName`, optional `artistMbId`) 168 + - **artists**: Array of artist objects (requires `artistName`, optional `artistMbId` for Last.fm) 141 169 - **playedTime**: ISO 8601 timestamp of when you listened 142 - - **submissionClientAgent**: Identifies this importer (`lastfm-importer/v0.2.0`) 143 - - **musicServiceBaseDomain**: Always set to `last.fm` 170 + - **submissionClientAgent**: Identifies this importer (`lastfm-importer/v0.3.0`) 171 + - **musicServiceBaseDomain**: Set to `last.fm` or `spotify.com` depending on source 144 172 145 173 ### Optional Fields (when available) 146 174 - **releaseName**: Album/release name 147 - - **releaseMbId**: MusicBrainz release ID 148 - - **recordingMbId**: MusicBrainz recording/track ID 149 - - **originUrl**: Link to the track on Last.fm 175 + - **releaseMbId**: MusicBrainz release ID (Last.fm only) 176 + - **recordingMbId**: MusicBrainz recording/track ID (Last.fm only) 177 + - **originUrl**: Link to the track on Last.fm or Spotify 150 178 151 - ### Example Record 179 + ### Example Records 152 180 181 + **Last.fm Record:** 153 182 ```json 154 183 { 155 184 "$type": "fm.teal.alpha.feed.play", ··· 165 194 "recordingMbId": "3a390ad3-fe56-45f2-a073-bebc45d6bde1", 166 195 "playedTime": "2025-11-13T23:49:36Z", 167 196 "originUrl": "https://www.last.fm/music/Cjbeards/_/Paint+My+Masterpiece", 168 - "submissionClientAgent": "lastfm-importer/v0.2.0", 197 + "submissionClientAgent": "lastfm-importer/v0.3.0", 169 198 "musicServiceBaseDomain": "last.fm" 199 + } 200 + ``` 201 + 202 + **Spotify Record:** 203 + ```json 204 + { 205 + "$type": "fm.teal.alpha.feed.play", 206 + "trackName": "Don't Give Up", 207 + "artists": [ 208 + { 209 + "artistName": "Chicane" 210 + } 211 + ], 212 + "releaseName": "Twenty", 213 + "playedTime": "2021-09-09T10:34:08Z", 214 + "originUrl": "https://open.spotify.com/track/3gZqDJkMZipOYCRjlHWgOV", 215 + "submissionClientAgent": "lastfm-importer/v0.3.0", 216 + "musicServiceBaseDomain": "spotify.com" 170 217 } 171 218 ``` 172 219 ··· 285 332 5. Shows clear schedule before starting 286 333 287 334 ### Record Processing 288 - 1. Parses CSV using `csv-parse` library 289 - 2. Sorts records chronologically (or reverse if `-r` flag) 290 - 3. Converts Last.fm format to `fm.teal.alpha.feed.play` schema 291 - 4. Validates required fields 292 - 5. Publishes in batches using `com.atproto.repo.applyWrites` (up to 200 records per call, the PDS maximum) 335 + 1. Parses input file(s): 336 + - **Last.fm**: CSV using `csv-parse` library 337 + - **Spotify**: JSON files (single or multiple in directory) 338 + 2. Filters data: 339 + - **Spotify**: Automatically removes podcasts, audiobooks, and non-music content 340 + 3. Converts to `fm.teal.alpha.feed.play` schema 341 + 4. Sorts records chronologically (or reverse if `-r` flag) 342 + 5. Generates TID-based record keys from `playedTime` for chronological ordering 343 + 6. Validates required fields 344 + 7. Publishes in batches using `com.atproto.repo.applyWrites` (up to 200 records per call, the PDS maximum) 293 345 294 346 **Note:** The batch publishing uses `applyWrites` instead of individual `createRecord` calls for dramatically improved performance (up to 20x faster). 295 347 296 348 ### Data Mapping 349 + 350 + **Last.fm:** 297 351 - **Track info**: Direct mapping from CSV columns 298 352 - **Timestamps**: Converts Unix timestamps to ISO 8601 299 353 - **MusicBrainz IDs**: Preserved when present in CSV 300 354 - **URLs**: Generated from artist/track names 301 355 - **Artists**: Wrapped in array format with optional MBID 356 + 357 + **Spotify:** 358 + - **Track info**: Extracted from JSON fields 359 + - **Timestamps**: Already in ISO 8601 format (`ts` field) 360 + - **URLs**: Generated from `spotify_track_uri` field 361 + - **Artists**: Extracted from `master_metadata_album_artist_name` 362 + - **Albums**: Extracted from `master_metadata_album_album_name` 363 + - **Filtering**: Non-music content automatically excluded 302 364 303 365 ## Lexicon Reference 304 366 ··· 357 419 358 420 --- 359 421 360 - **Note**: This tool is for personal use. Respect Last.fm's terms of service and rate limits when exporting your data. 422 + **Note**: This tool is for personal use. Respect the terms of service and rate limits when exporting your data.
+2 -2
package-lock.json
··· 1 1 { 2 2 "name": "lastfm-importer", 3 - "version": "0.2.0", 3 + "version": "0.3.0", 4 4 "lockfileVersion": 3, 5 5 "requires": true, 6 6 "packages": { 7 7 "": { 8 8 "name": "lastfm-importer", 9 - "version": "0.2.0", 9 + "version": "0.3.0", 10 10 "license": "AGPL-3.0-only", 11 11 "dependencies": { 12 12 "@atproto/api": "^0.13.35",
+1 -1
package.json
··· 1 1 { 2 2 "name": "lastfm-importer", 3 - "version": "0.2.0", 3 + "version": "0.3.0", 4 4 "description": "Import Last.fm scrobbles to ATProto with rate limiting", 5 5 "type": "module", 6 6 "main": "./dist/index.js",
+1 -1
src/config.ts
··· 17 17 export const RECORD_TYPE = 'fm.teal.alpha.feed.play'; 18 18 19 19 // Client agent 20 - export const CLIENT_AGENT = 'lastfm-importer/v0.2.0'; 20 + export const CLIENT_AGENT = 'lastfm-importer/v0.3.0'; 21 21 22 22 // Default batch configuration - aggressive defaults for maximum speed 23 23 // Will dynamically adjust based on success/failure
+27 -11
src/lib/cli.ts
··· 2 2 import { AtpAgent } from '@atproto/api'; // Use AtpAgent for consistency 3 3 import type { PlayRecord, Config, CommandLineArgs, PublishResult } from '../types.js'; 4 4 import { login } from './auth.js'; 5 - import { parseLastFmCsv, convertToPlayRecord, sortRecords } from '../lib/csv.js'; 5 + import { parseLastFmCsv, convertToPlayRecord, sortRecords } from '../lib/csv.js'; 6 + import { parseSpotifyJson, convertSpotifyToPlayRecord, sortSpotifyRecords } from '../lib/spotify.js'; 6 7 import { publishRecordsWithApplyWrites } from './publisher.js'; 7 8 import { prompt } from '../utils/input.js'; 8 9 import config from '../config.js'; ··· 14 15 */ 15 16 export function showHelp(): void { 16 17 console.log(` 17 - Last.fm to ATProto Importer v0.2.0 18 + Last.fm to ATProto Importer v0.3.0 18 19 19 20 Usage: npm start [options] 20 21 21 22 Options: 22 23 -h, --help Show this help message 23 - -f, --file <path> Path to Last.fm CSV export file 24 + -f, --file <path> Path to Last.fm CSV or Spotify JSON export file/directory 24 25 -i, --identifier <id> ATProto handle or DID 25 26 -p, --password <pass> ATProto app password 26 27 -b, --batch-size <num> Number of records per batch (auto-calculated if not set) ··· 29 30 -n, --dry-run Preview records without publishing 30 31 -r, --reverse-chronological Process newest first (default: oldest first) 31 32 -s, --sync Re-sync mode: check existing Teal records and only import new ones 33 + --spotify Import from Spotify JSON export instead of Last.fm CSV 32 34 --remove-duplicates Remove duplicate records from Teal (keeps first occurrence) 33 35 `); 34 36 } ··· 49 51 'dry-run': { type: 'boolean', short: 'n', default: false }, 50 52 'reverse-chronological': { type: 'boolean', short: 'r', default: false }, 51 53 sync: { type: 'boolean', short: 's', default: false }, 54 + spotify: { type: 'boolean', default: false }, 52 55 'remove-duplicates': { type: 'boolean', default: false }, 53 56 } as const; 54 57 ··· 131 134 } 132 135 133 136 // 2. Parse and Prepare Records 134 - // This function is assumed to read the file path in args.file 135 - const csvRecords = parseLastFmCsv(args.file); 137 + const useSpotify = args.spotify ?? false; 138 + let records: PlayRecord[]; 139 + let rawRecordCount: number; 136 140 137 - // This function maps the raw CSV records to the standardized PlayRecord structure 138 - let records: PlayRecord[] = csvRecords.map(record => convertToPlayRecord(record, cfg)); 141 + if (useSpotify) { 142 + console.log('📀 Importing from Spotify export...\n'); 143 + const spotifyRecords = parseSpotifyJson(args.file); 144 + rawRecordCount = spotifyRecords.length; 145 + records = spotifyRecords.map(record => convertSpotifyToPlayRecord(record, cfg)); 146 + } else { 147 + console.log('📀 Importing from Last.fm CSV export...\n'); 148 + const csvRecords = parseLastFmCsv(args.file); 149 + rawRecordCount = csvRecords.length; 150 + records = csvRecords.map(record => convertToPlayRecord(record, cfg)); 151 + } 139 152 140 153 // 2.5. Sync Mode: Fetch existing records and filter duplicates 141 154 if (syncMode && agent) { 155 + const originalRecords = [...records]; // Save before filtering 142 156 const existingRecords = await fetchExistingRecords(agent, cfg); 143 157 records = filterNewRecords(records, existingRecords); 144 158 ··· 147 161 process.exit(0); 148 162 } 149 163 150 - displaySyncStats(csvRecords.map(r => convertToPlayRecord(r, cfg)), existingRecords, records); 164 + displaySyncStats(originalRecords, existingRecords, records); 151 165 } 152 166 153 167 const totalRecords = records.length; 154 168 155 169 const reverseChronological = args['reverse-chronological'] ?? false; 156 - const sortedRecords = sortRecords(records, reverseChronological); 170 + const sortedRecords = useSpotify 171 + ? sortSpotifyRecords(records, reverseChronological) 172 + : sortRecords(records, reverseChronological); 157 173 158 174 // 3. Determine Batching parameters 159 175 let batchDelay = cfg.DEFAULT_BATCH_DELAY; ··· 193 209 // 5. Confirmation Prompt 194 210 if (!dryRun && !(args.yes ?? false)) { 195 211 if (syncMode) { 196 - console.log(`\nReady to publish ${totalRecords.toLocaleString()} NEW records (${csvRecords.length - totalRecords} duplicates skipped).`); 212 + console.log(`\nReady to publish ${totalRecords.toLocaleString()} NEW records (${rawRecordCount - totalRecords} duplicates skipped).`); 197 213 } else { 198 214 console.log(`\nReady to publish ${totalRecords.toLocaleString()} records.`); 199 215 } ··· 224 240 console.log(`\n🎉 ${syncMode ? 'Sync' : 'Import'} Complete!`); 225 241 console.log(`Total records processed: ${result.successCount.toLocaleString()} (${result.errorCount.toLocaleString()} failed)`); 226 242 if (syncMode) { 227 - console.log(`Duplicates skipped: ${csvRecords.length - totalRecords}`); 243 + console.log(`Duplicates skipped: ${rawRecordCount - totalRecords}`); 228 244 } 229 245 } 230 246
+3 -1
src/lib/publisher.ts
··· 7 7 displayRateLimitInfo, 8 8 calculateRateLimitedBatches, 9 9 } from '../utils/rate-limiter.js'; 10 + import { generateTIDFromISO } from '../utils/tid.js'; 10 11 import type { PlayRecord, Config, PublishResult } from '../types.js'; 11 12 12 13 /** ··· 79 80 80 81 const batchStartTime = Date.now(); 81 82 82 - // Build writes array for applyWrites 83 + // Build writes array for applyWrites with TID-based rkeys 83 84 const writes = batch.map((record) => ({ 84 85 $type: 'com.atproto.repo.applyWrites#create', 85 86 collection: RECORD_TYPE, 87 + rkey: generateTIDFromISO(record.playedTime), 86 88 value: record, 87 89 })); 88 90
+131
src/lib/spotify.ts
··· 1 + import * as fs from 'fs'; 2 + import * as path from 'path'; 3 + import type { PlayRecord, Config } from '../types.js'; 4 + import { formatDate } from '../utils/helpers.js'; 5 + 6 + /** 7 + * Spotify streaming history record 8 + */ 9 + export interface SpotifyRecord { 10 + ts: string; 11 + platform: string; 12 + ms_played: number; 13 + conn_country: string; 14 + master_metadata_track_name: string | null; 15 + master_metadata_album_artist_name: string | null; 16 + master_metadata_album_album_name: string | null; 17 + spotify_track_uri: string | null; 18 + episode_name: string | null; 19 + episode_show_name: string | null; 20 + spotify_episode_uri: string | null; 21 + reason_start: string; 22 + reason_end: string; 23 + shuffle: boolean; 24 + skipped: boolean; 25 + offline: boolean; 26 + offline_timestamp: number | null; 27 + incognito_mode: boolean; 28 + } 29 + 30 + /** 31 + * Parse Spotify JSON export 32 + * Supports both single files and directories with multiple JSON files 33 + */ 34 + export function parseSpotifyJson(filePathOrDir: string): SpotifyRecord[] { 35 + console.log(`Reading Spotify export: ${filePathOrDir}`); 36 + 37 + const stats = fs.statSync(filePathOrDir); 38 + let allRecords: SpotifyRecord[] = []; 39 + 40 + if (stats.isDirectory()) { 41 + // Read all JSON files in the directory 42 + const files = fs.readdirSync(filePathOrDir) 43 + .filter(f => f.endsWith('.json') && f.startsWith('Streaming_History_Audio')) 44 + .map(f => path.join(filePathOrDir, f)); 45 + 46 + console.log(`Found ${files.length} Spotify JSON files in directory`); 47 + 48 + for (const file of files) { 49 + const fileContent = fs.readFileSync(file, 'utf-8'); 50 + const records = JSON.parse(fileContent) as SpotifyRecord[]; 51 + allRecords = allRecords.concat(records); 52 + console.log(` ${path.basename(file)}: ${records.length} records`); 53 + } 54 + } else { 55 + // Single file 56 + const fileContent = fs.readFileSync(filePathOrDir, 'utf-8'); 57 + allRecords = JSON.parse(fileContent) as SpotifyRecord[]; 58 + } 59 + 60 + // Filter out records without track names (podcasts, audiobooks, etc.) 61 + const trackRecords = allRecords.filter(r => 62 + r.master_metadata_track_name && 63 + r.master_metadata_album_artist_name 64 + ); 65 + 66 + console.log(`✓ Parsed ${trackRecords.length} track records (filtered ${allRecords.length - trackRecords.length} non-music records)\n`); 67 + return trackRecords; 68 + } 69 + 70 + /** 71 + * Convert Spotify record to ATProto play record 72 + */ 73 + export function convertSpotifyToPlayRecord(spotifyRecord: SpotifyRecord, config: Config): PlayRecord { 74 + const { RECORD_TYPE, CLIENT_AGENT } = config; 75 + 76 + // Spotify timestamp is already in ISO 8601 format 77 + const playedTime = spotifyRecord.ts; 78 + 79 + // Build artists array 80 + const artists: PlayRecord['artists'] = []; 81 + if (spotifyRecord.master_metadata_album_artist_name) { 82 + artists.push({ 83 + artistName: spotifyRecord.master_metadata_album_artist_name, 84 + }); 85 + } 86 + 87 + // Build the play record 88 + const playRecord: PlayRecord = { 89 + $type: RECORD_TYPE, 90 + trackName: spotifyRecord.master_metadata_track_name || 'Unknown Track', 91 + artists, 92 + playedTime, 93 + submissionClientAgent: CLIENT_AGENT, 94 + musicServiceBaseDomain: 'spotify.com', 95 + originUrl: '', 96 + }; 97 + 98 + // Add optional fields 99 + if (spotifyRecord.master_metadata_album_album_name) { 100 + playRecord.releaseName = spotifyRecord.master_metadata_album_album_name; 101 + } 102 + 103 + // Generate Spotify URL if we have the track URI 104 + if (spotifyRecord.spotify_track_uri) { 105 + const trackId = spotifyRecord.spotify_track_uri.replace('spotify:track:', ''); 106 + playRecord.originUrl = `https://open.spotify.com/track/${trackId}`; 107 + } 108 + 109 + return playRecord; 110 + } 111 + 112 + /** 113 + * Sort records chronologically 114 + */ 115 + export function sortSpotifyRecords(records: PlayRecord[], reverseChronological = false): PlayRecord[] { 116 + console.log(`Sorting records ${reverseChronological ? 'newest' : 'oldest'} first...`); 117 + 118 + records.sort((a, b) => { 119 + const timeA = new Date(a.playedTime).getTime(); 120 + const timeB = new Date(b.playedTime).getTime(); 121 + return reverseChronological ? timeB - timeA : timeA - timeB; 122 + }); 123 + 124 + const firstPlay = formatDate(records[0].playedTime); 125 + const lastPlay = formatDate(records[records.length - 1].playedTime); 126 + console.log(`✓ Sorted ${records.length} records`); 127 + console.log(` First: ${firstPlay}`); 128 + console.log(` Last: ${lastPlay}\n`); 129 + 130 + return records; 131 + }
+1
src/types.ts
··· 45 45 'dry-run'?: boolean; 46 46 'reverse-chronological'?: boolean; 47 47 sync?: boolean; 48 + spotify?: boolean; 48 49 'remove-duplicates'?: boolean; 49 50 } 50 51
+33
src/utils/tid.ts
··· 1 + /** 2 + * TID (Timestamp Identifier) generation for ATProto 3 + * Based on: https://atproto.com/specs/record-key#record-key-type-tid 4 + */ 5 + 6 + const B32_CHARS = '234567abcdefghijklmnopqrstuvwxyz'; 7 + 8 + /** 9 + * Generate a TID from a Date object 10 + * TID encodes Unix microseconds in base32 11 + */ 12 + export function generateTID(date: Date): string { 13 + // Convert to Unix microseconds 14 + // JS only gives us millisecond precision, so we multiply by 1000 15 + const unixMicros = Math.floor(date.getTime() * 1000); 16 + 17 + let tid = ''; 18 + for (let i = 0; i < 13; i++) { 19 + // Extract 5 bits at a time (base32) 20 + const shift = 60 - (i * 5); 21 + const index = Math.floor(unixMicros / Math.pow(2, shift)) % 32; 22 + tid += B32_CHARS[index]; 23 + } 24 + 25 + return tid; 26 + } 27 + 28 + /** 29 + * Generate a TID from an ISO 8601 timestamp string 30 + */ 31 + export function generateTIDFromISO(isoString: string): string { 32 + return generateTID(new Date(isoString)); 33 + }