Import your Last.fm and Spotify listening history to the AT Protocol network using the
fm.teal.alpha.feed.play lexicon.
feat: add Spotify import support and TID-based record keys
- Add support for Spotify Extended Streaming History JSON exports
- Accept single files or directories
- Filter out podcasts and non-music content
- Introduce --spotify CLI flag and update help text
- Generate TID-based rkeys from playedTime for chronological ordering
- Update importer metadata and client agent to v0.3.0
- Clarify MusicBrainz support as Last.fm-only
- Expand README with Spotify setup, examples, and record schema
+281
-36
+82
-20
README.md
+82
-20
README.md
···
7
7
## Features
8
8
9
9
- ✅ **Batch Operations**: Uses `com.atproto.repo.applyWrites` for efficient batch publishing (up to 200 records per call)
10
-
10
+
- ✅ **Spotify Support**: Import from Spotify Extended Streaming History (JSON format)
11
+
- ✅ **TID-based Record Keys**: Records use timestamp-based identifiers for chronological ordering
11
12
- ✅ **Re-Sync Mode**: Check existing Teal records and only import new scrobbles (no duplicates!)
12
13
- ✅ **Rate Limiting**: Automatically limits imports to 1K records per day to prevent rate limiting your entire PDS
13
14
- ✅ **Multi-Day Imports**: Large imports (>1K records) automatically span multiple days with 24-hour pauses
···
19
20
- ✅ **Batch Processing**: Configurable batching with rate limit safety
20
21
- ✅ **Progress Tracking**: Real-time progress with time estimates
21
22
- ✅ **Error Handling**: Continues on errors with detailed reporting
22
-
- ✅ **MusicBrainz Support**: Preserves MusicBrainz IDs when available
23
+
- ✅ **MusicBrainz Support**: Preserves MusicBrainz IDs when available (Last.fm only)
23
24
- ✅ **Chronological Ordering**: Processes oldest first (or newest with `-r` flag)
24
25
25
26
## Important: Rate Limits
···
82
83
For automation or scripting, provide all parameters via flags:
83
84
84
85
```bash
85
-
# Full automation
86
+
# Full automation (Last.fm)
86
87
npm start -- -f lastfm.csv -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y
88
+
89
+
# Import from Spotify (single file)
90
+
npm start -- -f Streaming_History_Audio_2021-2023_0.json --spotify -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y
91
+
92
+
# Import from Spotify (directory with multiple files - recommended)
93
+
npm start -- -f '/path/to/Spotify Extended Streaming History' --spotify -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y
87
94
88
95
# Preview without publishing
89
96
npm start -- -f lastfm.csv --dry-run
90
97
98
+
# Preview Spotify import
99
+
npm start -- -f '/path/to/Spotify Extended Streaming History' --spotify --dry-run
100
+
91
101
# Custom batch settings (advanced users)
92
102
npm start -- -f lastfm.csv -i alice.bsky.social -b 20 -d 3000
93
103
···
100
110
| Option | Short | Description | Default |
101
111
|--------|-------|-------------|---------|
102
112
| `--help` | `-h` | Show help message | - |
103
-
| `--file <path>` | `-f` | Path to Last.fm CSV export file | (prompted) |
113
+
| `--file <path>` | `-f` | Path to Last.fm CSV or Spotify JSON file/directory | (prompted) |
114
+
| `--spotify` | | Import from Spotify JSON export instead of Last.fm | false |
104
115
| `--identifier <id>` | `-i` | ATProto handle or DID | (prompted) |
105
116
| `--password <pass>` | `-p` | ATProto app password | (prompted) |
106
117
| `--batch-size <num>` | `-b` | Records per batch | Auto-calculated |
···
124
135
125
136
⚠️ Lower delays increase speed but risk hitting rate limits. The automatic calculation is recommended.
126
137
127
-
## Getting Your Last.fm Data
138
+
## Getting Your Data
139
+
140
+
### Last.fm Export
128
141
129
142
1. Go to <https://mainstream.ghan.nl/export.html>
130
143
2. Request your data export in CSV format
131
144
3. Download the CSV file when ready
132
145
4. Use the CSV file path with this script
133
146
147
+
### Spotify Export
148
+
149
+
1. Go to your [Spotify Privacy Settings](https://www.spotify.com/account/privacy/)
150
+
2. Scroll down to "Download your data" and request your data
151
+
3. Select "Extended streaming history" (this can take up to 30 days)
152
+
4. When ready, download and extract the ZIP file
153
+
5. Use either:
154
+
- A single JSON file: `Streaming_History_Audio_2021-2023_0.json`
155
+
- The entire extracted directory (recommended)
156
+
157
+
**Note**: Spotify exports include multiple JSON files. The importer automatically:
158
+
- Reads all `Streaming_History_Audio_*.json` files in a directory
159
+
- Filters out podcasts, audiobooks, and other non-music content
160
+
- Combines all music tracks into a single import
161
+
134
162
## What Gets Imported
135
163
136
-
Each Last.fm scrobble becomes an `fm.teal.alpha.feed.play` record with:
164
+
Each scrobble (from Last.fm or Spotify) becomes an `fm.teal.alpha.feed.play` record with:
137
165
138
166
### Required Fields
139
167
- **trackName**: The name of the track
140
-
- **artists**: Array of artist objects (requires `artistName`, optional `artistMbId`)
168
+
- **artists**: Array of artist objects (requires `artistName`, optional `artistMbId` for Last.fm)
141
169
- **playedTime**: ISO 8601 timestamp of when you listened
142
-
- **submissionClientAgent**: Identifies this importer (`lastfm-importer/v0.2.0`)
143
-
- **musicServiceBaseDomain**: Always set to `last.fm`
170
+
- **submissionClientAgent**: Identifies this importer (`lastfm-importer/v0.3.0`)
171
+
- **musicServiceBaseDomain**: Set to `last.fm` or `spotify.com` depending on source
144
172
145
173
### Optional Fields (when available)
146
174
- **releaseName**: Album/release name
147
-
- **releaseMbId**: MusicBrainz release ID
148
-
- **recordingMbId**: MusicBrainz recording/track ID
149
-
- **originUrl**: Link to the track on Last.fm
175
+
- **releaseMbId**: MusicBrainz release ID (Last.fm only)
176
+
- **recordingMbId**: MusicBrainz recording/track ID (Last.fm only)
177
+
- **originUrl**: Link to the track on Last.fm or Spotify
150
178
151
-
### Example Record
179
+
### Example Records
152
180
181
+
**Last.fm Record:**
153
182
```json
154
183
{
155
184
"$type": "fm.teal.alpha.feed.play",
···
165
194
"recordingMbId": "3a390ad3-fe56-45f2-a073-bebc45d6bde1",
166
195
"playedTime": "2025-11-13T23:49:36Z",
167
196
"originUrl": "https://www.last.fm/music/Cjbeards/_/Paint+My+Masterpiece",
168
-
"submissionClientAgent": "lastfm-importer/v0.2.0",
197
+
"submissionClientAgent": "lastfm-importer/v0.3.0",
169
198
"musicServiceBaseDomain": "last.fm"
199
+
}
200
+
```
201
+
202
+
**Spotify Record:**
203
+
```json
204
+
{
205
+
"$type": "fm.teal.alpha.feed.play",
206
+
"trackName": "Don't Give Up",
207
+
"artists": [
208
+
{
209
+
"artistName": "Chicane"
210
+
}
211
+
],
212
+
"releaseName": "Twenty",
213
+
"playedTime": "2021-09-09T10:34:08Z",
214
+
"originUrl": "https://open.spotify.com/track/3gZqDJkMZipOYCRjlHWgOV",
215
+
"submissionClientAgent": "lastfm-importer/v0.3.0",
216
+
"musicServiceBaseDomain": "spotify.com"
170
217
}
171
218
```
172
219
···
285
332
5. Shows clear schedule before starting
286
333
287
334
### Record Processing
288
-
1. Parses CSV using `csv-parse` library
289
-
2. Sorts records chronologically (or reverse if `-r` flag)
290
-
3. Converts Last.fm format to `fm.teal.alpha.feed.play` schema
291
-
4. Validates required fields
292
-
5. Publishes in batches using `com.atproto.repo.applyWrites` (up to 200 records per call, the PDS maximum)
335
+
1. Parses input file(s):
336
+
- **Last.fm**: CSV using `csv-parse` library
337
+
- **Spotify**: JSON files (single or multiple in directory)
338
+
2. Filters data:
339
+
- **Spotify**: Automatically removes podcasts, audiobooks, and non-music content
340
+
3. Converts to `fm.teal.alpha.feed.play` schema
341
+
4. Sorts records chronologically (or reverse if `-r` flag)
342
+
5. Generates TID-based record keys from `playedTime` for chronological ordering
343
+
6. Validates required fields
344
+
7. Publishes in batches using `com.atproto.repo.applyWrites` (up to 200 records per call, the PDS maximum)
293
345
294
346
**Note:** The batch publishing uses `applyWrites` instead of individual `createRecord` calls for dramatically improved performance (up to 20x faster).
295
347
296
348
### Data Mapping
349
+
350
+
**Last.fm:**
297
351
- **Track info**: Direct mapping from CSV columns
298
352
- **Timestamps**: Converts Unix timestamps to ISO 8601
299
353
- **MusicBrainz IDs**: Preserved when present in CSV
300
354
- **URLs**: Generated from artist/track names
301
355
- **Artists**: Wrapped in array format with optional MBID
356
+
357
+
**Spotify:**
358
+
- **Track info**: Extracted from JSON fields
359
+
- **Timestamps**: Already in ISO 8601 format (`ts` field)
360
+
- **URLs**: Generated from `spotify_track_uri` field
361
+
- **Artists**: Extracted from `master_metadata_album_artist_name`
362
+
- **Albums**: Extracted from `master_metadata_album_album_name`
363
+
- **Filtering**: Non-music content automatically excluded
302
364
303
365
## Lexicon Reference
304
366
···
357
419
358
420
---
359
421
360
-
**Note**: This tool is for personal use. Respect Last.fm's terms of service and rate limits when exporting your data.
422
+
**Note**: This tool is for personal use. Respect the terms of service and rate limits when exporting your data.
+2
-2
package-lock.json
+2
-2
package-lock.json
···
1
1
{
2
2
"name": "lastfm-importer",
3
-
"version": "0.2.0",
3
+
"version": "0.3.0",
4
4
"lockfileVersion": 3,
5
5
"requires": true,
6
6
"packages": {
7
7
"": {
8
8
"name": "lastfm-importer",
9
-
"version": "0.2.0",
9
+
"version": "0.3.0",
10
10
"license": "AGPL-3.0-only",
11
11
"dependencies": {
12
12
"@atproto/api": "^0.13.35",
+1
-1
package.json
+1
-1
package.json
+1
-1
src/config.ts
+1
-1
src/config.ts
···
17
17
export const RECORD_TYPE = 'fm.teal.alpha.feed.play';
18
18
19
19
// Client agent
20
-
export const CLIENT_AGENT = 'lastfm-importer/v0.2.0';
20
+
export const CLIENT_AGENT = 'lastfm-importer/v0.3.0';
21
21
22
22
// Default batch configuration - aggressive defaults for maximum speed
23
23
// Will dynamically adjust based on success/failure
+27
-11
src/lib/cli.ts
+27
-11
src/lib/cli.ts
···
2
2
import { AtpAgent } from '@atproto/api'; // Use AtpAgent for consistency
3
3
import type { PlayRecord, Config, CommandLineArgs, PublishResult } from '../types.js';
4
4
import { login } from './auth.js';
5
-
import { parseLastFmCsv, convertToPlayRecord, sortRecords } from '../lib/csv.js';
5
+
import { parseLastFmCsv, convertToPlayRecord, sortRecords } from '../lib/csv.js';
6
+
import { parseSpotifyJson, convertSpotifyToPlayRecord, sortSpotifyRecords } from '../lib/spotify.js';
6
7
import { publishRecordsWithApplyWrites } from './publisher.js';
7
8
import { prompt } from '../utils/input.js';
8
9
import config from '../config.js';
···
14
15
*/
15
16
export function showHelp(): void {
16
17
console.log(`
17
-
Last.fm to ATProto Importer v0.2.0
18
+
Last.fm to ATProto Importer v0.3.0
18
19
19
20
Usage: npm start [options]
20
21
21
22
Options:
22
23
-h, --help Show this help message
23
-
-f, --file <path> Path to Last.fm CSV export file
24
+
-f, --file <path> Path to Last.fm CSV or Spotify JSON export file/directory
24
25
-i, --identifier <id> ATProto handle or DID
25
26
-p, --password <pass> ATProto app password
26
27
-b, --batch-size <num> Number of records per batch (auto-calculated if not set)
···
29
30
-n, --dry-run Preview records without publishing
30
31
-r, --reverse-chronological Process newest first (default: oldest first)
31
32
-s, --sync Re-sync mode: check existing Teal records and only import new ones
33
+
--spotify Import from Spotify JSON export instead of Last.fm CSV
32
34
--remove-duplicates Remove duplicate records from Teal (keeps first occurrence)
33
35
`);
34
36
}
···
49
51
'dry-run': { type: 'boolean', short: 'n', default: false },
50
52
'reverse-chronological': { type: 'boolean', short: 'r', default: false },
51
53
sync: { type: 'boolean', short: 's', default: false },
54
+
spotify: { type: 'boolean', default: false },
52
55
'remove-duplicates': { type: 'boolean', default: false },
53
56
} as const;
54
57
···
131
134
}
132
135
133
136
// 2. Parse and Prepare Records
134
-
// This function is assumed to read the file path in args.file
135
-
const csvRecords = parseLastFmCsv(args.file);
137
+
const useSpotify = args.spotify ?? false;
138
+
let records: PlayRecord[];
139
+
let rawRecordCount: number;
136
140
137
-
// This function maps the raw CSV records to the standardized PlayRecord structure
138
-
let records: PlayRecord[] = csvRecords.map(record => convertToPlayRecord(record, cfg));
141
+
if (useSpotify) {
142
+
console.log('📀 Importing from Spotify export...\n');
143
+
const spotifyRecords = parseSpotifyJson(args.file);
144
+
rawRecordCount = spotifyRecords.length;
145
+
records = spotifyRecords.map(record => convertSpotifyToPlayRecord(record, cfg));
146
+
} else {
147
+
console.log('📀 Importing from Last.fm CSV export...\n');
148
+
const csvRecords = parseLastFmCsv(args.file);
149
+
rawRecordCount = csvRecords.length;
150
+
records = csvRecords.map(record => convertToPlayRecord(record, cfg));
151
+
}
139
152
140
153
// 2.5. Sync Mode: Fetch existing records and filter duplicates
141
154
if (syncMode && agent) {
155
+
const originalRecords = [...records]; // Save before filtering
142
156
const existingRecords = await fetchExistingRecords(agent, cfg);
143
157
records = filterNewRecords(records, existingRecords);
144
158
···
147
161
process.exit(0);
148
162
}
149
163
150
-
displaySyncStats(csvRecords.map(r => convertToPlayRecord(r, cfg)), existingRecords, records);
164
+
displaySyncStats(originalRecords, existingRecords, records);
151
165
}
152
166
153
167
const totalRecords = records.length;
154
168
155
169
const reverseChronological = args['reverse-chronological'] ?? false;
156
-
const sortedRecords = sortRecords(records, reverseChronological);
170
+
const sortedRecords = useSpotify
171
+
? sortSpotifyRecords(records, reverseChronological)
172
+
: sortRecords(records, reverseChronological);
157
173
158
174
// 3. Determine Batching parameters
159
175
let batchDelay = cfg.DEFAULT_BATCH_DELAY;
···
193
209
// 5. Confirmation Prompt
194
210
if (!dryRun && !(args.yes ?? false)) {
195
211
if (syncMode) {
196
-
console.log(`\nReady to publish ${totalRecords.toLocaleString()} NEW records (${csvRecords.length - totalRecords} duplicates skipped).`);
212
+
console.log(`\nReady to publish ${totalRecords.toLocaleString()} NEW records (${rawRecordCount - totalRecords} duplicates skipped).`);
197
213
} else {
198
214
console.log(`\nReady to publish ${totalRecords.toLocaleString()} records.`);
199
215
}
···
224
240
console.log(`\n🎉 ${syncMode ? 'Sync' : 'Import'} Complete!`);
225
241
console.log(`Total records processed: ${result.successCount.toLocaleString()} (${result.errorCount.toLocaleString()} failed)`);
226
242
if (syncMode) {
227
-
console.log(`Duplicates skipped: ${csvRecords.length - totalRecords}`);
243
+
console.log(`Duplicates skipped: ${rawRecordCount - totalRecords}`);
228
244
}
229
245
}
230
246
+3
-1
src/lib/publisher.ts
+3
-1
src/lib/publisher.ts
···
7
7
displayRateLimitInfo,
8
8
calculateRateLimitedBatches,
9
9
} from '../utils/rate-limiter.js';
10
+
import { generateTIDFromISO } from '../utils/tid.js';
10
11
import type { PlayRecord, Config, PublishResult } from '../types.js';
11
12
12
13
/**
···
79
80
80
81
const batchStartTime = Date.now();
81
82
82
-
// Build writes array for applyWrites
83
+
// Build writes array for applyWrites with TID-based rkeys
83
84
const writes = batch.map((record) => ({
84
85
$type: 'com.atproto.repo.applyWrites#create',
85
86
collection: RECORD_TYPE,
87
+
rkey: generateTIDFromISO(record.playedTime),
86
88
value: record,
87
89
}));
88
90
+131
src/lib/spotify.ts
+131
src/lib/spotify.ts
···
1
+
import * as fs from 'fs';
2
+
import * as path from 'path';
3
+
import type { PlayRecord, Config } from '../types.js';
4
+
import { formatDate } from '../utils/helpers.js';
5
+
6
+
/**
7
+
* Spotify streaming history record
8
+
*/
9
+
export interface SpotifyRecord {
10
+
ts: string;
11
+
platform: string;
12
+
ms_played: number;
13
+
conn_country: string;
14
+
master_metadata_track_name: string | null;
15
+
master_metadata_album_artist_name: string | null;
16
+
master_metadata_album_album_name: string | null;
17
+
spotify_track_uri: string | null;
18
+
episode_name: string | null;
19
+
episode_show_name: string | null;
20
+
spotify_episode_uri: string | null;
21
+
reason_start: string;
22
+
reason_end: string;
23
+
shuffle: boolean;
24
+
skipped: boolean;
25
+
offline: boolean;
26
+
offline_timestamp: number | null;
27
+
incognito_mode: boolean;
28
+
}
29
+
30
+
/**
31
+
* Parse Spotify JSON export
32
+
* Supports both single files and directories with multiple JSON files
33
+
*/
34
+
export function parseSpotifyJson(filePathOrDir: string): SpotifyRecord[] {
35
+
console.log(`Reading Spotify export: ${filePathOrDir}`);
36
+
37
+
const stats = fs.statSync(filePathOrDir);
38
+
let allRecords: SpotifyRecord[] = [];
39
+
40
+
if (stats.isDirectory()) {
41
+
// Read all JSON files in the directory
42
+
const files = fs.readdirSync(filePathOrDir)
43
+
.filter(f => f.endsWith('.json') && f.startsWith('Streaming_History_Audio'))
44
+
.map(f => path.join(filePathOrDir, f));
45
+
46
+
console.log(`Found ${files.length} Spotify JSON files in directory`);
47
+
48
+
for (const file of files) {
49
+
const fileContent = fs.readFileSync(file, 'utf-8');
50
+
const records = JSON.parse(fileContent) as SpotifyRecord[];
51
+
allRecords = allRecords.concat(records);
52
+
console.log(` ${path.basename(file)}: ${records.length} records`);
53
+
}
54
+
} else {
55
+
// Single file
56
+
const fileContent = fs.readFileSync(filePathOrDir, 'utf-8');
57
+
allRecords = JSON.parse(fileContent) as SpotifyRecord[];
58
+
}
59
+
60
+
// Filter out records without track names (podcasts, audiobooks, etc.)
61
+
const trackRecords = allRecords.filter(r =>
62
+
r.master_metadata_track_name &&
63
+
r.master_metadata_album_artist_name
64
+
);
65
+
66
+
console.log(`✓ Parsed ${trackRecords.length} track records (filtered ${allRecords.length - trackRecords.length} non-music records)\n`);
67
+
return trackRecords;
68
+
}
69
+
70
+
/**
71
+
* Convert Spotify record to ATProto play record
72
+
*/
73
+
export function convertSpotifyToPlayRecord(spotifyRecord: SpotifyRecord, config: Config): PlayRecord {
74
+
const { RECORD_TYPE, CLIENT_AGENT } = config;
75
+
76
+
// Spotify timestamp is already in ISO 8601 format
77
+
const playedTime = spotifyRecord.ts;
78
+
79
+
// Build artists array
80
+
const artists: PlayRecord['artists'] = [];
81
+
if (spotifyRecord.master_metadata_album_artist_name) {
82
+
artists.push({
83
+
artistName: spotifyRecord.master_metadata_album_artist_name,
84
+
});
85
+
}
86
+
87
+
// Build the play record
88
+
const playRecord: PlayRecord = {
89
+
$type: RECORD_TYPE,
90
+
trackName: spotifyRecord.master_metadata_track_name || 'Unknown Track',
91
+
artists,
92
+
playedTime,
93
+
submissionClientAgent: CLIENT_AGENT,
94
+
musicServiceBaseDomain: 'spotify.com',
95
+
originUrl: '',
96
+
};
97
+
98
+
// Add optional fields
99
+
if (spotifyRecord.master_metadata_album_album_name) {
100
+
playRecord.releaseName = spotifyRecord.master_metadata_album_album_name;
101
+
}
102
+
103
+
// Generate Spotify URL if we have the track URI
104
+
if (spotifyRecord.spotify_track_uri) {
105
+
const trackId = spotifyRecord.spotify_track_uri.replace('spotify:track:', '');
106
+
playRecord.originUrl = `https://open.spotify.com/track/${trackId}`;
107
+
}
108
+
109
+
return playRecord;
110
+
}
111
+
112
+
/**
113
+
* Sort records chronologically
114
+
*/
115
+
export function sortSpotifyRecords(records: PlayRecord[], reverseChronological = false): PlayRecord[] {
116
+
console.log(`Sorting records ${reverseChronological ? 'newest' : 'oldest'} first...`);
117
+
118
+
records.sort((a, b) => {
119
+
const timeA = new Date(a.playedTime).getTime();
120
+
const timeB = new Date(b.playedTime).getTime();
121
+
return reverseChronological ? timeB - timeA : timeA - timeB;
122
+
});
123
+
124
+
const firstPlay = formatDate(records[0].playedTime);
125
+
const lastPlay = formatDate(records[records.length - 1].playedTime);
126
+
console.log(`✓ Sorted ${records.length} records`);
127
+
console.log(` First: ${firstPlay}`);
128
+
console.log(` Last: ${lastPlay}\n`);
129
+
130
+
return records;
131
+
}
+1
src/types.ts
+1
src/types.ts
+33
src/utils/tid.ts
+33
src/utils/tid.ts
···
1
+
/**
2
+
* TID (Timestamp Identifier) generation for ATProto
3
+
* Based on: https://atproto.com/specs/record-key#record-key-type-tid
4
+
*/
5
+
6
+
const B32_CHARS = '234567abcdefghijklmnopqrstuvwxyz';
7
+
8
+
/**
9
+
* Generate a TID from a Date object
10
+
* TID encodes Unix microseconds in base32
11
+
*/
12
+
export function generateTID(date: Date): string {
13
+
// Convert to Unix microseconds
14
+
// JS only gives us millisecond precision, so we multiply by 1000
15
+
const unixMicros = Math.floor(date.getTime() * 1000);
16
+
17
+
let tid = '';
18
+
for (let i = 0; i < 13; i++) {
19
+
// Extract 5 bits at a time (base32)
20
+
const shift = 60 - (i * 5);
21
+
const index = Math.floor(unixMicros / Math.pow(2, shift)) % 32;
22
+
tid += B32_CHARS[index];
23
+
}
24
+
25
+
return tid;
26
+
}
27
+
28
+
/**
29
+
* Generate a TID from an ISO 8601 timestamp string
30
+
*/
31
+
export function generateTIDFromISO(isoString: string): string {
32
+
return generateTID(new Date(isoString));
33
+
}