Import your Last.fm and Spotify listening history to the AT Protocol network using the
fm.teal.alpha.feed.play lexicon.
rewrite rate-limiting layer
+1415
-1212
+47
-13
README.md
+47
-13
README.md
···
12
12
This importer automatically protects your PDS by:
13
13
- Limiting imports to **1,000 records per day** (with 75% safety margin)
14
14
- Calculating optimal batch sizes and delays
15
-
- Pausing 24 hours between days for large imports
15
+
- Automatically waiting for rate limit resets when limits are hit
16
16
- Providing clear progress tracking and time estimates
17
17
18
18
For more details, see the [Bluesky Rate Limits Documentation](https://docs.bsky.app/blog/rate-limits-pds-v3).
···
58
58
- ✅ **Input Deduplication**: Removes duplicate entries within the source file before submission
59
59
- ✅ **Batch Operations**: Uses `com.atproto.repo.applyWrites` for efficient batch publishing (up to 200 records per call)
60
60
- ✅ **Rate Limiting**: Automatic daily limits prevent PDS rate limiting
61
-
- ✅ **Multi-Day Imports**: Large imports automatically span multiple days with 24-hour pauses
61
+
- ✅ **Automatic Rate Limiting**: Waits for limit resets when daily/hourly limits are reached
62
+
- ✅ **Multi-Day Imports**: Large imports automatically span multiple days with automatic waits
62
63
- ✅ **Resume Support**: Safe to stop (Ctrl+C) and restart - continues from where it left off
63
64
- ✅ **Graceful Cancellation**: Press Ctrl+C to stop after the current batch completes
64
65
···
411
412
3. Calculates optimal batch size and delay to spread records evenly
412
413
4. Enforces minimum delay between batches
413
414
5. Shows clear schedule before starting
415
+
6. Logs waiting periods when rate limits are hit with duration
416
+
417
+
**Example rate-limit wait logging:**
418
+
```
419
+
ℹ Rate limit (hourly), waiting 23m 45s for reset
420
+
ℹ Rate limit (daily), waiting 1h 12m 30s for reset
421
+
```
414
422
415
423
### Multi-Day Imports
416
424
···
418
426
1. **Calculates a schedule**: Splits your import across multiple days
419
427
2. **Shows the plan**: Displays which records will be imported each day
420
428
3. **Processes Day 1**: Imports the first batch of records
421
-
4. **Pauses 24 hours**: Waits a full day before continuing
429
+
4. **Waits for reset**: When limits are reached, waits for the hourly/daily reset
422
430
5. **Repeats**: Continues until all records are imported
423
431
424
432
**Example output for a 20,000 record import:**
425
433
```
426
-
📊 Rate Limiting Information:
427
-
Total records: 20,000
428
-
Daily limit: 7,500 records/day
429
-
Estimated duration: 3 days
430
-
Batch size: 200 records
431
-
Batch delay: 11.52s
434
+
=== Batch Configuration ===
435
+
ℹ Using auto-calculated batch size: 200 records
436
+
ℹ Batch delay: 11520ms
437
+
438
+
=== Import Configuration ===
439
+
ℹ Total records: 20,000
440
+
ℹ Batch size: 200 records
441
+
ℹ Batch delay: 11520ms
442
+
ℹ Duration: 3 days (7,500 records/day limit)
443
+
⚠️ Large import will span multiple days with automatic rate-limit waits
444
+
445
+
=== Publishing Records ===
446
+
→ Processed batch 1-200 (0.0s, 173.2 rec/s, 2m 0s remaining)
447
+
→ Processed batch 201-400 (2.0s, 100.0 rec/s, 1m 58s remaining)
448
+
...
449
+
ℹ Rate limit (hourly), waiting 45m 0s for reset
450
+
→ Resuming after rate limit reset
451
+
→ Processed batch 7801-8000 (45m 0s, 2.9 rec/s, 4h 35m 50s remaining)
432
452
```
433
453
434
454
**Important notes:**
···
446
466
- **Yellow (⚠️)**: Warnings
447
467
- **Red (✗)**: Errors
448
468
- **Bold Red (🛑)**: Fatal errors
469
+
- **Blue (ℹ)**: Informational messages (including rate-limit waits)
470
+
471
+
### Rate-Limit Wait Messages
472
+
473
+
When the importer hits a rate limit and needs to wait, it logs a clear message:
474
+
475
+
```
476
+
ℹ Rate limit (hourly), waiting 23m 45s for reset
477
+
ℹ Rate limit (daily), waiting 1h 12m 30s for reset
478
+
```
479
+
480
+
These messages use `formatDuration()` for human-readable duration display (e.g., `23m 45s`, `1h 12m 30s`). The wait reason indicates which limit was hit:
481
+
- **hourly**: Hourly rate limit reached
482
+
- **daily**: Daily rate limit reached
449
483
450
484
### Verbosity Levels
451
485
···
471
505
- **Network errors**: Failed records are logged but don't stop the import
472
506
- **Invalid data**: Skipped with error messages
473
507
- **Authentication issues**: Clear error messages with suggested fixes
474
-
- **Rate limit hits**: Automatic adjustment and retry logic
508
+
- **Rate limit hits**: Automatic adjustment with logged backoff duration and retry logic
475
509
- **Ctrl+C handling**: Gracefully stops after current batch
476
510
477
511
## Troubleshooting
···
489
523
### Performance Issues
490
524
491
525
**"Rate limit exceeded"**
492
-
- The importer should prevent this automatically
493
-
- If you see this, wait 24 hours before retrying
526
+
- The importer handles this automatically by waiting for reset
527
+
- Progress messages show wait duration when rate limits are hit
494
528
- Consider reducing batch size with `-b` flag
495
529
496
530
**Import seems stuck**
497
531
- Check progress messages - large imports take time
498
-
- Multi-day imports pause for 24 hours between days
532
+
- Rate-limit waits may occur between days
499
533
- You can safely stop (Ctrl+C) and resume later
500
534
- Use `--verbose` flag to see detailed progress
501
535
+1
-1
package.json
+1
-1
package.json
+2
-2
src/config.ts
+2
-2
src/config.ts
···
20
20
export const RECORD_TYPE = 'fm.teal.alpha.feed.play';
21
21
22
22
// Build client agent string
23
-
export function buildClientAgent(_debug = false) {
23
+
export function buildClientAgent() {
24
24
// Always return just the version, regardless of debug mode
25
25
// The debug parameter is kept for backwards compatibility but unused
26
26
return 'malachite/v0.6.2';
···
43
43
const config: Config = {
44
44
RECORD_TYPE,
45
45
MIN_RECORDS_FOR_SCALING: 20,
46
-
BASE_BATCH_SIZE: 200, // Match DEFAULT_BATCH_SIZE for consistency
46
+
BASE_BATCH_SIZE: 200, // Match DEFAULT_BATCH_SIZE for consistency
47
47
SCALING_FACTOR: 1.5,
48
48
DEFAULT_BATCH_SIZE,
49
49
DEFAULT_BATCH_DELAY,
+1
-1
src/index.ts
+1
-1
src/index.ts
+95
src/lib/atproto/auth.ts
+95
src/lib/atproto/auth.ts
···
1
+
import { Agent, CredentialSession } from '@atproto/api';
2
+
import { prompt } from '../../utils/input.js';
3
+
import * as ui from '../../utils/ui.js';
4
+
import { PassthroughAgent } from './passthrough-agent.js';
5
+
6
+
export interface ResolvedIdentity {
7
+
did: string;
8
+
handle: string;
9
+
pds: string;
10
+
signing_key: string;
11
+
}
12
+
13
+
export interface LoginOptions {
14
+
identifier?: string;
15
+
password?: string;
16
+
}
17
+
18
+
async function resolveIdentifier(identifier: string): Promise<ResolvedIdentity> {
19
+
ui.startSpinner(`Resolving identifier: ${identifier}`);
20
+
21
+
try {
22
+
const response = await fetch(
23
+
`${process.env.IDENTITY_RESOLVER_URL ?? 'https://slingshot.microcosm.blue/xrpc/com.bad-example.identity.resolveMiniDoc'}?identifier=${encodeURIComponent(identifier)}`,
24
+
);
25
+
26
+
if (!response.ok) {
27
+
throw new Error(`Failed to resolve identifier: ${response.status} ${response.statusText}`);
28
+
}
29
+
30
+
const data = (await response.json()) as ResolvedIdentity;
31
+
32
+
if (!data.did || !data.pds || !data.signing_key) {
33
+
throw new Error('Invalid response from identity resolver: missing required fields');
34
+
}
35
+
36
+
ui.succeedSpinner(`Resolved to PDS: ${data.pds}`);
37
+
return data;
38
+
} catch (error) {
39
+
ui.failSpinner('Failed to resolve identifier');
40
+
throw error;
41
+
}
42
+
}
43
+
44
+
export async function login(options: LoginOptions = {}): Promise<PassthroughAgent> {
45
+
const { identifier: inputIdentifier, password: inputPassword } = options;
46
+
47
+
ui.header('ATProto Login');
48
+
49
+
const identifier = inputIdentifier ?? (await prompt('Handle or DID: '));
50
+
ui.keyValue('Handle or DID', identifier);
51
+
52
+
const password = inputPassword ?? (await prompt('App password: ', true));
53
+
ui.keyValue('App password', '[hidden]');
54
+
55
+
console.log('');
56
+
57
+
try {
58
+
const resolved = await resolveIdentifier(identifier);
59
+
60
+
ui.startSpinner('Logging in...');
61
+
const session = new CredentialSession(new URL(resolved.pds));
62
+
63
+
await session.login({
64
+
identifier: resolved.did,
65
+
password: password,
66
+
});
67
+
68
+
const agent = new Agent(session);
69
+
70
+
ui.succeedSpinner('Logged in successfully!');
71
+
ui.keyValue('DID', agent.did || 'unknown');
72
+
ui.keyValue('Handle', resolved.handle || 'unknown');
73
+
console.log('');
74
+
75
+
return new PassthroughAgent(agent);
76
+
} catch (error) {
77
+
const err = error as Error;
78
+
ui.failSpinner('Login failed');
79
+
80
+
if (err.message.includes('Failed to resolve identifier')) {
81
+
throw new Error('Handle not found. Please check your AT Protocol handle.');
82
+
} else if (err.message.includes('AuthFactorTokenRequired')) {
83
+
throw new Error('Two-factor authentication required. Please use your app password.');
84
+
} else if (
85
+
err.message.includes('AccountTakedown') ||
86
+
err.message.includes('AccountSuspended')
87
+
) {
88
+
throw new Error('Account is suspended or has been taken down.');
89
+
} else if (err.message.includes('InvalidCredentials')) {
90
+
throw new Error('Invalid credentials. Please check your handle and app password.');
91
+
} else {
92
+
throw new Error(`Login failed: ${err.message || 'Unknown error'}`);
93
+
}
94
+
}
95
+
}
+10
src/lib/atproto/index.ts
+10
src/lib/atproto/index.ts
···
1
+
export { login } from './auth.js';
2
+
export { MockAgent } from './mock-agent.js';
3
+
export { PassthroughAgent } from './passthrough-agent.js';
4
+
export { RateLimitedAgent, CancellationError, type RateLimitHooks } from './rate-limited-agent.js';
5
+
export type {
6
+
AgentWithAccess,
7
+
AtprotoAgentInterface,
8
+
RateLimitOptions,
9
+
RateLimitStatus,
10
+
} from './types.js';
+112
src/lib/atproto/mock-agent.ts
+112
src/lib/atproto/mock-agent.ts
···
1
+
import type {
2
+
ApplyWritesParams,
3
+
ApplyWritesResponse,
4
+
AtprotoAgentInterface,
5
+
DeleteRecordParams,
6
+
DeleteRecordResponse,
7
+
ListRecordsParams,
8
+
ListRecordsResponse,
9
+
} from './types.js';
10
+
11
+
export interface MockAgentConfig {
12
+
did?: string;
13
+
delay?: number;
14
+
shouldFail?: boolean;
15
+
failOnCallNumber?: number;
16
+
rateLimitErrorOnCallNumber?: number;
17
+
consumedPointsOnError?: number;
18
+
}
19
+
20
+
export class MockAgent implements AtprotoAgentInterface {
21
+
private config: Required<MockAgentConfig>;
22
+
private callCounter: number;
23
+
24
+
constructor(config: MockAgentConfig = {}) {
25
+
this.config = {
26
+
did: config.did ?? 'did:test:123',
27
+
delay: config.delay ?? 0,
28
+
shouldFail: config.shouldFail ?? false,
29
+
failOnCallNumber: config.failOnCallNumber ?? -1,
30
+
rateLimitErrorOnCallNumber: config.rateLimitErrorOnCallNumber ?? -1,
31
+
consumedPointsOnError: config.consumedPointsOnError ?? 10,
32
+
};
33
+
this.callCounter = 0;
34
+
}
35
+
36
+
resetCallCounter(): void {
37
+
this.callCounter = 0;
38
+
}
39
+
40
+
private async simulateDelay(): Promise<void> {
41
+
if (this.config.delay > 0) {
42
+
await new Promise((resolve) => setTimeout(resolve, this.config.delay));
43
+
}
44
+
}
45
+
46
+
private checkAndThrow(): void {
47
+
this.callCounter++;
48
+
49
+
if (this.callCounter === this.config.rateLimitErrorOnCallNumber) {
50
+
const resetTime = Math.floor(Date.now() / 1000) + 60;
51
+
const limit = 100;
52
+
const remaining = limit - this.config.consumedPointsOnError;
53
+
const error = new Error('Rate Limit Exceeded') as any;
54
+
error.status = 429;
55
+
error.error = 'RateLimitExceeded';
56
+
error.headers = {
57
+
'ratelimit-remaining': String(remaining),
58
+
'ratelimit-limit': String(limit),
59
+
'ratelimit-reset': String(resetTime),
60
+
'ratelimit-policy': '100;w=86400',
61
+
};
62
+
throw error;
63
+
}
64
+
65
+
if (this.config.shouldFail && this.callCounter === this.config.failOnCallNumber) {
66
+
throw new Error('Mock error');
67
+
}
68
+
}
69
+
70
+
async listRecords(params: ListRecordsParams): Promise<ListRecordsResponse> {
71
+
await this.simulateDelay();
72
+
this.checkAndThrow();
73
+
74
+
return {
75
+
records: [
76
+
{
77
+
uri: `at://${params.repo ?? this.config.did}/fm.teal.alpha.feed.play/3klj4`,
78
+
cid: 'bafyreic',
79
+
value: { $type: 'fm.teal.alpha.feed.play' },
80
+
},
81
+
],
82
+
};
83
+
}
84
+
85
+
async applyWrites(params: ApplyWritesParams): Promise<ApplyWritesResponse> {
86
+
await this.simulateDelay();
87
+
this.checkAndThrow();
88
+
89
+
return {
90
+
data: {
91
+
results: params.writes.map((_, i) => ({
92
+
uri: `at://${params.repo ?? this.config.did}/fm.teal.alpha.feed.play/${i}`,
93
+
})),
94
+
},
95
+
};
96
+
}
97
+
98
+
async deleteRecord(params: DeleteRecordParams): Promise<DeleteRecordResponse> {
99
+
await this.simulateDelay();
100
+
this.checkAndThrow();
101
+
102
+
return {
103
+
data: {
104
+
uri: `at://${params.repo ?? this.config.did}/${params.collection}/${params.rkey}`,
105
+
},
106
+
};
107
+
}
108
+
109
+
getDid(): string {
110
+
return this.config.did;
111
+
}
112
+
}
+67
src/lib/atproto/passthrough-agent.ts
+67
src/lib/atproto/passthrough-agent.ts
···
1
+
import type { Agent } from '@atproto/api';
2
+
import type {
3
+
AgentWithAccess,
4
+
ApplyWritesParams,
5
+
ApplyWritesResponse,
6
+
DeleteRecordParams,
7
+
DeleteRecordResponse,
8
+
ListRecordsParams,
9
+
ListRecordsResponse,
10
+
} from './types.js';
11
+
12
+
export class PassthroughAgent implements AgentWithAccess {
13
+
private _agent: Agent;
14
+
15
+
constructor(agent: Agent) {
16
+
this._agent = agent;
17
+
}
18
+
19
+
get agent(): Agent {
20
+
return this._agent;
21
+
}
22
+
23
+
private getRepo(params: { repo?: string }): string {
24
+
return params.repo ?? this.getDid();
25
+
}
26
+
27
+
async listRecords(params: ListRecordsParams): Promise<ListRecordsResponse> {
28
+
const response = await this._agent.com.atproto.repo.listRecords({
29
+
repo: this.getRepo(params),
30
+
collection: params.collection,
31
+
limit: params.limit,
32
+
cursor: params.cursor,
33
+
});
34
+
35
+
return {
36
+
records: response.data.records,
37
+
cursor: response.data.cursor,
38
+
};
39
+
}
40
+
41
+
async applyWrites(params: ApplyWritesParams): Promise<ApplyWritesResponse> {
42
+
const response = await this._agent.com.atproto.repo.applyWrites({
43
+
repo: this.getRepo(params),
44
+
writes: params.writes as any,
45
+
});
46
+
47
+
return {
48
+
data: response.data as ApplyWritesResponse['data'],
49
+
};
50
+
}
51
+
52
+
async deleteRecord(params: DeleteRecordParams): Promise<DeleteRecordResponse> {
53
+
const response = await this._agent.com.atproto.repo.deleteRecord({
54
+
repo: this.getRepo(params),
55
+
collection: params.collection,
56
+
rkey: params.rkey,
57
+
});
58
+
59
+
return {
60
+
data: response.data as DeleteRecordResponse['data'],
61
+
};
62
+
}
63
+
64
+
getDid(): string {
65
+
return this._agent.did ?? '';
66
+
}
67
+
}
+305
src/lib/atproto/rate-limited-agent.ts
+305
src/lib/atproto/rate-limited-agent.ts
···
1
+
import type {
2
+
ApplyWritesParams,
3
+
ApplyWritesResponse,
4
+
AtprotoAgentInterface,
5
+
DeleteRecordParams,
6
+
DeleteRecordResponse,
7
+
ListRecordsParams,
8
+
ListRecordsResponse,
9
+
RateLimitOptions,
10
+
RateLimitStatus,
11
+
} from './types.js';
12
+
13
+
export interface RateLimitHooks {
14
+
onWaitForReset?: (
15
+
status: RateLimitStatus,
16
+
waitTimeMs: number,
17
+
reason: 'daily' | 'hourly',
18
+
) => void;
19
+
onRateLimitBackoff?: (status: RateLimitStatus, backoffMs: number) => void;
20
+
onRateLimitError?: (status: RateLimitStatus, error: unknown) => void;
21
+
}
22
+
23
+
export class RateLimitedAgent implements AtprotoAgentInterface {
24
+
private delegate: AtprotoAgentInterface;
25
+
private config: Required<Omit<RateLimitOptions, 'signal'>>;
26
+
private signal?: AbortSignal;
27
+
private hooks?: RateLimitHooks;
28
+
29
+
private dailyPointsUsed: number = 0;
30
+
private hourlyPointsUsed: number = 0;
31
+
private dailyResetTime: Date;
32
+
private hourlyResetTime: Date;
33
+
34
+
private consecutiveFailures: number = 0;
35
+
private readonly baseDelayMs: number = 1000;
36
+
private readonly maxBackoffMs: number = 60000;
37
+
38
+
constructor(delegate: AtprotoAgentInterface, options?: RateLimitOptions & RateLimitHooks) {
39
+
this.delegate = delegate;
40
+
this.signal = options?.signal;
41
+
this.hooks = options;
42
+
43
+
this.config = {
44
+
pointsPerDay: options?.pointsPerDay ?? 30000,
45
+
safetyMargin: options?.safetyMargin ?? 0.75,
46
+
maxPointsPerMinute: options?.maxPointsPerMinute ?? 20,
47
+
pointsPerRead: options?.pointsPerRead ?? 1,
48
+
pointsPerWrite: options?.pointsPerWrite ?? 3,
49
+
pointsPerDelete: options?.pointsPerDelete ?? 1,
50
+
};
51
+
52
+
const now = new Date();
53
+
this.dailyResetTime = this.getNextDailyReset(now);
54
+
this.hourlyResetTime = this.getNextHourlyReset(now);
55
+
56
+
if (this.signal) {
57
+
this.signal.addEventListener('abort', () => {
58
+
this.consecutiveFailures = Infinity;
59
+
});
60
+
}
61
+
}
62
+
63
+
private getNextDailyReset(date: Date): Date {
64
+
const reset = new Date(date);
65
+
reset.setHours(24, 0, 0, 0);
66
+
if (reset <= date) {
67
+
reset.setDate(reset.getDate() + 1);
68
+
}
69
+
return reset;
70
+
}
71
+
72
+
private getNextHourlyReset(date: Date): Date {
73
+
const reset = new Date(date);
74
+
reset.setMinutes(0, 0, 0);
75
+
if (reset <= date) {
76
+
reset.setHours(reset.getHours() + 1);
77
+
}
78
+
return reset;
79
+
}
80
+
81
+
private get usableDailyPoints(): number {
82
+
return Math.floor(this.config.pointsPerDay * this.config.safetyMargin);
83
+
}
84
+
85
+
private get usableHourlyPoints(): number {
86
+
return Math.floor((this.config.pointsPerDay * this.config.safetyMargin) / 24);
87
+
}
88
+
89
+
private get dailyPointsRemaining(): number {
90
+
return this.usableDailyPoints - this.dailyPointsUsed;
91
+
}
92
+
93
+
private get hourlyPointsRemaining(): number {
94
+
return this.usableHourlyPoints - this.hourlyPointsUsed;
95
+
}
96
+
97
+
private checkAndResetTimers(): void {
98
+
const now = new Date();
99
+
100
+
if (now >= this.dailyResetTime) {
101
+
this.dailyPointsUsed = 0;
102
+
this.dailyResetTime = this.getNextDailyReset(now);
103
+
}
104
+
105
+
if (now >= this.hourlyResetTime) {
106
+
this.hourlyPointsUsed = 0;
107
+
this.hourlyResetTime = this.getNextHourlyReset(now);
108
+
}
109
+
}
110
+
111
+
private checkCancelled(): void {
112
+
if (this.signal?.aborted) {
113
+
throw new CancellationError('Operation cancelled');
114
+
}
115
+
}
116
+
117
+
private getCurrentStatus(): RateLimitStatus {
118
+
this.checkAndResetTimers();
119
+
return {
120
+
dailyPointsRemaining: Math.max(0, this.dailyPointsRemaining),
121
+
hourlyPointsRemaining: Math.max(0, this.hourlyPointsRemaining),
122
+
dailyResetAt: this.dailyResetTime,
123
+
hourlyResetAt: this.hourlyResetTime,
124
+
};
125
+
}
126
+
127
+
private async waitForReset(targetPoints: number): Promise<void> {
128
+
this.checkCancelled();
129
+
130
+
const status = this.getCurrentStatus();
131
+
const now = new Date();
132
+
133
+
let resetTime: Date;
134
+
let reason: 'daily' | 'hourly';
135
+
136
+
if (status.dailyPointsRemaining < targetPoints && status.hourlyPointsRemaining < targetPoints) {
137
+
resetTime =
138
+
status.dailyResetAt < status.hourlyResetAt ? status.dailyResetAt : status.hourlyResetAt;
139
+
reason = status.dailyResetAt < status.hourlyResetAt ? 'daily' : 'hourly';
140
+
} else if (status.dailyPointsRemaining < targetPoints) {
141
+
resetTime = status.dailyResetAt;
142
+
reason = 'daily';
143
+
} else {
144
+
resetTime = status.hourlyResetAt;
145
+
reason = 'hourly';
146
+
}
147
+
148
+
const waitTime = Math.max(0, resetTime.getTime() - now.getTime());
149
+
if (waitTime > 0) {
150
+
this.hooks?.onWaitForReset?.(status, waitTime, reason);
151
+
await new Promise((resolve, reject) => {
152
+
const timeout = setTimeout(resolve, waitTime);
153
+
this.signal?.addEventListener('abort', () => {
154
+
clearTimeout(timeout);
155
+
reject(new CancellationError('Operation cancelled'));
156
+
});
157
+
});
158
+
}
159
+
160
+
this.checkAndResetTimers();
161
+
}
162
+
163
+
private calculateBackoff(): number {
164
+
const multiplier = 2 ** this.consecutiveFailures;
165
+
return Math.min(this.baseDelayMs * multiplier, this.maxBackoffMs);
166
+
}
167
+
168
+
private resetBackoff(): void {
169
+
this.consecutiveFailures = 0;
170
+
}
171
+
172
+
private extractPointsFromError(
173
+
error: unknown,
174
+
): { dailyConsumed: number; hourlyConsumed: number } | null {
175
+
const err = error as {
176
+
headers?: Record<string, string>;
177
+
status?: number;
178
+
message?: string;
179
+
};
180
+
const headers = err.headers;
181
+
182
+
if (!headers) {
183
+
return null;
184
+
}
185
+
186
+
const remainingHeader = headers['ratelimit-remaining'] ?? headers['RateLimit-Remaining'];
187
+
const limitHeader = headers['ratelimit-limit'] ?? headers['RateLimit-Limit'];
188
+
189
+
if (remainingHeader && limitHeader) {
190
+
const remaining = parseInt(remainingHeader, 10);
191
+
const limit = parseInt(limitHeader, 10);
192
+
const consumed = limit - remaining;
193
+
return { dailyConsumed: consumed, hourlyConsumed: consumed };
194
+
}
195
+
196
+
return null;
197
+
}
198
+
199
+
private isRateLimitError(error: unknown): boolean {
200
+
const err = error as { status?: number; message?: string; error?: string };
201
+
const status = err.status;
202
+
const message = err.message ?? '';
203
+
const xrpcError = err.error ?? '';
204
+
return (
205
+
status === 429 ||
206
+
message.includes('rate limit') ||
207
+
message.includes('too many requests') ||
208
+
message.includes('RatelimitExceeded') ||
209
+
xrpcError === 'RateLimitExceeded'
210
+
);
211
+
}
212
+
213
+
private async executeWithRateLimit<T>(operation: () => Promise<T>, points: number): Promise<T> {
214
+
while (true) {
215
+
this.checkCancelled();
216
+
const status = this.getCurrentStatus();
217
+
218
+
if (status.dailyPointsRemaining < points) {
219
+
await this.waitForReset(points);
220
+
continue;
221
+
}
222
+
223
+
if (status.hourlyPointsRemaining < points) {
224
+
await this.waitForReset(points);
225
+
continue;
226
+
}
227
+
228
+
try {
229
+
const result = await operation();
230
+
this.dailyPointsUsed += points;
231
+
this.hourlyPointsUsed += points;
232
+
this.resetBackoff();
233
+
return result;
234
+
} catch (error) {
235
+
if (error instanceof CancellationError) {
236
+
throw error;
237
+
}
238
+
239
+
if (this.isRateLimitError(error)) {
240
+
const statusBeforeError = this.getCurrentStatus();
241
+
this.hooks?.onRateLimitError?.(statusBeforeError, error);
242
+
243
+
const consumedPoints = this.extractPointsFromError(error);
244
+
if (consumedPoints !== null) {
245
+
this.dailyPointsUsed = Math.max(this.dailyPointsUsed, consumedPoints.dailyConsumed);
246
+
this.hourlyPointsUsed = Math.max(this.hourlyPointsUsed, consumedPoints.hourlyConsumed);
247
+
}
248
+
249
+
this.consecutiveFailures++;
250
+
const backoff = this.calculateBackoff();
251
+
const statusAfterUpdate = this.getCurrentStatus();
252
+
this.hooks?.onRateLimitBackoff?.(statusAfterUpdate, backoff);
253
+
254
+
await new Promise((resolve, reject) => {
255
+
const timeout = setTimeout(resolve, backoff);
256
+
this.signal?.addEventListener('abort', () => {
257
+
clearTimeout(timeout);
258
+
reject(new CancellationError('Operation cancelled'));
259
+
});
260
+
});
261
+
this.checkAndResetTimers();
262
+
continue;
263
+
}
264
+
265
+
throw error;
266
+
}
267
+
}
268
+
}
269
+
270
+
async listRecords(params: ListRecordsParams): Promise<ListRecordsResponse> {
271
+
return this.executeWithRateLimit(
272
+
() => this.delegate.listRecords(params),
273
+
this.config.pointsPerRead,
274
+
);
275
+
}
276
+
277
+
async applyWrites(params: ApplyWritesParams): Promise<ApplyWritesResponse> {
278
+
const writeCount = params.writes.length;
279
+
const points = writeCount * this.config.pointsPerWrite;
280
+
281
+
return this.executeWithRateLimit(() => this.delegate.applyWrites(params), points);
282
+
}
283
+
284
+
async deleteRecord(params: DeleteRecordParams): Promise<DeleteRecordResponse> {
285
+
return this.executeWithRateLimit(
286
+
() => this.delegate.deleteRecord(params),
287
+
this.config.pointsPerDelete,
288
+
);
289
+
}
290
+
291
+
getRateLimitStatus(): RateLimitStatus {
292
+
return this.getCurrentStatus();
293
+
}
294
+
295
+
getDid(): string {
296
+
return this.delegate.getDid();
297
+
}
298
+
}
299
+
300
+
export class CancellationError extends Error {
301
+
constructor(message: string) {
302
+
super(message);
303
+
this.name = 'CancellationError';
304
+
}
305
+
}
+73
src/lib/atproto/types.ts
+73
src/lib/atproto/types.ts
···
1
+
import type { Agent } from '@atproto/api';
2
+
3
+
export interface ListRecordsParams {
4
+
repo?: string;
5
+
collection: string;
6
+
limit?: number;
7
+
cursor?: string;
8
+
}
9
+
10
+
export interface ListRecordsResponse {
11
+
records: Array<{
12
+
uri: string;
13
+
cid: string;
14
+
value: unknown;
15
+
}>;
16
+
cursor?: string;
17
+
}
18
+
19
+
export interface ApplyWritesParams {
20
+
repo?: string;
21
+
writes: Array<{
22
+
$type: string;
23
+
collection: string;
24
+
rkey: string;
25
+
value: unknown;
26
+
}>;
27
+
}
28
+
29
+
export interface ApplyWritesResponse {
30
+
data: {
31
+
results?: Array<Record<string, unknown>>;
32
+
};
33
+
}
34
+
35
+
export interface DeleteRecordParams {
36
+
repo?: string;
37
+
collection: string;
38
+
rkey: string;
39
+
}
40
+
41
+
export interface DeleteRecordResponse {
42
+
data: {
43
+
uri: string;
44
+
};
45
+
}
46
+
47
+
export interface AtprotoAgentInterface {
48
+
listRecords(params: ListRecordsParams): Promise<ListRecordsResponse>;
49
+
applyWrites(params: ApplyWritesParams): Promise<ApplyWritesResponse>;
50
+
deleteRecord(params: DeleteRecordParams): Promise<DeleteRecordResponse>;
51
+
getDid(): string;
52
+
}
53
+
54
+
export interface AgentWithAccess extends AtprotoAgentInterface {
55
+
readonly agent: Agent;
56
+
}
57
+
58
+
export interface RateLimitStatus {
59
+
dailyPointsRemaining: number;
60
+
hourlyPointsRemaining: number;
61
+
dailyResetAt: Date;
62
+
hourlyResetAt: Date;
63
+
}
64
+
65
+
export interface RateLimitOptions {
66
+
pointsPerDay?: number;
67
+
safetyMargin?: number;
68
+
maxPointsPerMinute?: number;
69
+
pointsPerRead?: number;
70
+
pointsPerWrite?: number;
71
+
pointsPerDelete?: number;
72
+
signal?: AbortSignal;
73
+
}
-105
src/lib/auth.ts
-105
src/lib/auth.ts
···
1
-
import { AtpAgent } from '@atproto/api';
2
-
import { prompt } from '../utils/input.js';
3
-
import * as ui from '../utils/ui.js';
4
-
5
-
interface ResolvedIdentity {
6
-
did: string;
7
-
handle: string;
8
-
pds: string;
9
-
signing_key: string;
10
-
}
11
-
12
-
/**
13
-
* Resolves an AT Protocol identifier (handle or DID) to get PDS information
14
-
*/
15
-
async function resolveIdentifier(identifier: string): Promise<ResolvedIdentity> {
16
-
ui.startSpinner(`Resolving identifier: ${identifier}`);
17
-
18
-
try {
19
-
const response = await fetch(
20
-
`https://slingshot.microcosm.blue/xrpc/com.bad-example.identity.resolveMiniDoc?identifier=${encodeURIComponent(identifier)}`
21
-
);
22
-
23
-
if (!response.ok) {
24
-
throw new Error(`Failed to resolve identifier: ${response.status} ${response.statusText}`);
25
-
}
26
-
27
-
const data = await response.json() as ResolvedIdentity;
28
-
29
-
if (!data.did || !data.pds) {
30
-
throw new Error('Invalid response from identity resolver');
31
-
}
32
-
33
-
ui.succeedSpinner(`Resolved to PDS: ${data.pds}`);
34
-
return data;
35
-
} catch (error) {
36
-
ui.failSpinner('Failed to resolve identifier');
37
-
throw error;
38
-
}
39
-
}
40
-
41
-
/**
42
-
* Login to ATProto using Slingshot resolver
43
-
*/
44
-
export async function login(
45
-
identifier: string | undefined,
46
-
password: string | undefined,
47
-
_resolverUrl?: string // Keep parameter for backwards compatibility but don't use it
48
-
): Promise<AtpAgent> {
49
-
ui.header('ATProto Login');
50
-
51
-
// Prompt for missing credentials
52
-
if (!identifier) {
53
-
identifier = await prompt('Handle or DID: ');
54
-
} else {
55
-
ui.keyValue('Handle or DID', identifier);
56
-
}
57
-
58
-
if (!password) {
59
-
password = await prompt('App password: ', true);
60
-
} else {
61
-
ui.keyValue('App password', '[hidden]');
62
-
}
63
-
64
-
console.log('');
65
-
66
-
try {
67
-
// Resolve the identifier to get PDS and other info
68
-
const resolved = await resolveIdentifier(identifier);
69
-
70
-
// Initialize the agent with the resolved PDS URL
71
-
ui.startSpinner('Logging in...');
72
-
const agent = new AtpAgent({
73
-
service: resolved.pds,
74
-
});
75
-
76
-
// Attempt to login using the resolved DID for more reliable authentication
77
-
await agent.login({
78
-
identifier: resolved.did,
79
-
password: password,
80
-
});
81
-
82
-
ui.succeedSpinner('Logged in successfully!');
83
-
ui.keyValue('DID', agent.session?.did || 'unknown');
84
-
ui.keyValue('Handle', agent.session?.handle || 'unknown');
85
-
console.log('');
86
-
87
-
return agent;
88
-
} catch (error) {
89
-
const err = error as Error;
90
-
ui.failSpinner('Login failed');
91
-
92
-
// Provide more specific error messages
93
-
if (err.message.includes('Failed to resolve identifier')) {
94
-
throw new Error('Handle not found. Please check your AT Protocol handle.');
95
-
} else if (err.message.includes('AuthFactorTokenRequired')) {
96
-
throw new Error('Two-factor authentication required. Please use your app password.');
97
-
} else if (err.message.includes('AccountTakedown') || err.message.includes('AccountSuspended')) {
98
-
throw new Error('Account is suspended or has been taken down.');
99
-
} else if (err.message.includes('InvalidCredentials')) {
100
-
throw new Error('Invalid credentials. Please check your handle and app password.');
101
-
} else {
102
-
throw new Error(`Login failed: ${err.message || 'Unknown error'}`);
103
-
}
104
-
}
105
-
}
+93
-42
src/lib/cli.ts
+93
-42
src/lib/cli.ts
···
1
1
#!/usr/bin/env node
2
2
import { parseArgs } from 'node:util';
3
-
import { AtpAgent } from '@atproto/api';
4
-
import type { PlayRecord, Config, CommandLineArgs, PublishResult } from '../types.js';
5
-
import { login } from './auth.js';
6
-
import { parseLastFmCsv, convertToPlayRecord, sortRecords } from '../lib/csv.js';
7
-
import { parseSpotifyJson, convertSpotifyToPlayRecord, sortSpotifyRecords } from '../lib/spotify.js';
8
-
import { parseCombinedExports } from '../lib/merge.js';
9
-
import { publishRecordsWithApplyWrites } from './publisher.js';
10
-
import { prompt } from '../utils/input.js';
11
3
import config from '../config.js';
12
-
import { calculateOptimalBatchSize } from '../utils/helpers.js';
13
-
import { fetchExistingRecords, filterNewRecords, displaySyncStats, removeDuplicates, deduplicateInputRecords } from './sync.js';
14
-
import { Logger, LogLevel, setGlobalLogger, log } from '../utils/logger.js';
15
-
import { registerKillswitch } from '../utils/killswitch.js';
16
-
import { clearCache, clearAllCaches } from '../utils/teal-cache.js';
4
+
import { convertToPlayRecord, parseLastFmCsv, sortRecords } from '../lib/csv.js';
5
+
import { parseCombinedExports } from '../lib/merge.js';
17
6
import {
18
-
loadImportState,
7
+
convertSpotifyToPlayRecord,
8
+
parseSpotifyJson,
9
+
sortSpotifyRecords,
10
+
} from '../lib/spotify.js';
11
+
import type { CommandLineArgs, Config, PlayRecord, PublishResult } from '../types.js';
12
+
import { calculateOptimalBatchSize, formatDuration } from '../utils/helpers.js';
13
+
import {
14
+
clearImportState,
19
15
createImportState,
20
16
displayResumeInfo,
21
-
clearImportState,
22
-
ImportState,
17
+
type ImportState,
18
+
loadImportState,
23
19
} from '../utils/import-state.js';
20
+
import { prompt } from '../utils/input.js';
21
+
import { Logger, LogLevel, log, setGlobalLogger } from '../utils/logger.js';
22
+
import { clearAllCaches, clearCache } from '../utils/teal-cache.js';
23
+
import { login } from './atproto/auth.js';
24
+
import { RateLimitedAgent, type RateLimitHooks } from './atproto/rate-limited-agent.js';
25
+
import type { AtprotoAgentInterface } from './atproto/types.js';
26
+
import { publishRecordsWithApplyWrites } from './publisher.js';
27
+
import {
28
+
deduplicateInputRecords,
29
+
displaySyncStats,
30
+
fetchExistingRecords,
31
+
filterNewRecords,
32
+
removeDuplicates,
33
+
} from './sync.js';
24
34
25
35
/**
26
36
* Show help message
27
37
*/
28
38
export function showHelp(): void {
29
39
console.log(`
30
-
${'\x1b[1m'}Last.fm to ATProto Importer v0.6.2${'\x1b[0m'}
40
+
${'\x1b[1m'}Last.fm to ATProto Importer v0.7.0${'\x1b[0m'}
31
41
32
42
${'\x1b[1m'}USAGE:${'\x1b[0m'}
33
43
npm start [options]
···
197
207
*/
198
208
export async function runCLI(): Promise<void> {
199
209
try {
200
-
registerKillswitch();
210
+
const abortController = new AbortController();
201
211
const args = parseCommandLineArgs();
202
212
const cfg = config as Config;
203
-
let agent: AtpAgent | null = null;
213
+
let agent: AtprotoAgentInterface | null = null;
214
+
215
+
process.on('SIGINT', () => {
216
+
abortController.abort();
217
+
});
204
218
205
219
const logger = new Logger(
206
-
args.quiet ? LogLevel.WARN :
207
-
args.verbose ? LogLevel.DEBUG :
208
-
LogLevel.INFO
220
+
args.quiet ? LogLevel.WARN : args.verbose ? LogLevel.DEBUG : LogLevel.INFO,
209
221
);
210
222
setGlobalLogger(logger);
211
223
···
227
239
}
228
240
log.section('Clear Cache');
229
241
log.info('Authenticating to identify cache...');
230
-
agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent;
231
-
const did = agent.session?.did;
242
+
const did = (await login({ identifier: args.handle, password: args.password })).getDid();
232
243
if (!did) {
233
244
throw new Error('Failed to get DID from session');
234
245
}
···
246
257
247
258
if (mode === 'combined') {
248
259
if (!args.input || !args['spotify-input']) {
249
-
throw new Error('Combined mode requires both --input (Last.fm) and --spotify-input (Spotify)');
260
+
throw new Error(
261
+
'Combined mode requires both --input (Last.fm) and --spotify-input (Spotify)',
262
+
);
250
263
}
251
264
} else if (mode !== 'deduplicate' && !args.input) {
252
265
throw new Error('Missing required argument: --input <path>');
···
257
270
throw new Error('Deduplicate mode requires --handle and --password');
258
271
}
259
272
log.section('Remove Duplicate Records');
260
-
agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent;
273
+
const rateLimitHooks: RateLimitHooks = {
274
+
onWaitForReset: (_status, waitTimeMs, reason) => {
275
+
log.info(`Rate limit (${reason}), waiting ${formatDuration(waitTimeMs)} for reset`);
276
+
},
277
+
onRateLimitBackoff: (_status, backoffMs) => {
278
+
log.debug(`Rate limited by server, backing off ${formatDuration(backoffMs)}`);
279
+
},
280
+
};
281
+
agent = new RateLimitedAgent(
282
+
await login({ identifier: args.handle, password: args.password }),
283
+
{
284
+
safetyMargin: args.aggressive ? 0.9 : 0.75,
285
+
signal: abortController.signal,
286
+
...rateLimitHooks,
287
+
},
288
+
);
261
289
const result = await removeDuplicates(agent, cfg, true);
262
290
if (result.totalDuplicates === 0) {
263
291
return;
264
292
}
265
293
if (!dryRun && !args.yes) {
266
-
log.warn(`This will permanently delete ${result.totalDuplicates} duplicate records from Teal.`);
294
+
log.warn(
295
+
`This will permanently delete ${result.totalDuplicates} duplicate records from Teal.`,
296
+
);
267
297
log.info('The first occurrence of each duplicate will be kept.');
268
298
log.blank();
269
299
const answer = await prompt('Are you sure you want to continue? (y/N) ');
···
271
301
log.info('Duplicate removal cancelled by user.');
272
302
process.exit(0);
273
303
}
274
-
await removeDuplicates(agent, cfg, false);
304
+
await removeDuplicates(agent!, cfg, false);
275
305
log.success('Duplicate removal complete!');
276
306
} else if (dryRun) {
277
307
log.info('DRY RUN: No records were actually removed.');
···
284
314
throw new Error('Missing required arguments: --handle and --password');
285
315
}
286
316
log.debug('Authenticating...');
287
-
agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent;
317
+
const rateLimitHooks: RateLimitHooks = {
318
+
onWaitForReset: (_status, waitTimeMs, reason) => {
319
+
log.info(`Rate limit (${reason}), waiting ${formatDuration(waitTimeMs)} for reset`);
320
+
},
321
+
onRateLimitBackoff: (_status, backoffMs) => {
322
+
log.debug(`Rate limited by server, backing off ${formatDuration(backoffMs)}`);
323
+
},
324
+
};
325
+
agent = new RateLimitedAgent(
326
+
await login({ identifier: args.handle, password: args.password }),
327
+
{
328
+
safetyMargin: args.aggressive ? 0.9 : 0.75,
329
+
signal: abortController.signal,
330
+
...rateLimitHooks,
331
+
},
332
+
);
288
333
log.debug('Authentication successful');
289
334
290
335
log.section('Loading Records');
291
336
let records: PlayRecord[];
292
337
let rawRecordCount: number;
293
-
const isDebug = args.verbose ?? false;
294
338
295
339
if (mode === 'combined') {
296
340
log.info('Merging Last.fm and Spotify exports...');
297
-
records = parseCombinedExports(args.input!, args['spotify-input']!, cfg, isDebug);
341
+
records = parseCombinedExports(args.input!, args['spotify-input']!, cfg);
298
342
rawRecordCount = records.length;
299
343
} else if (mode === 'spotify') {
300
344
log.info('Importing from Spotify export...');
301
345
const spotifyRecords = parseSpotifyJson(args.input!);
302
346
rawRecordCount = spotifyRecords.length;
303
-
records = spotifyRecords.map(record => convertSpotifyToPlayRecord(record, cfg, isDebug));
347
+
records = spotifyRecords.map((record) => convertSpotifyToPlayRecord(record, cfg));
304
348
} else {
305
349
log.info('Importing from Last.fm CSV export...');
306
350
const csvRecords = parseLastFmCsv(args.input!);
307
351
rawRecordCount = csvRecords.length;
308
-
records = csvRecords.map(record => convertToPlayRecord(record, cfg, isDebug));
352
+
records = csvRecords.map((record) => convertToPlayRecord(record, cfg));
309
353
}
310
354
311
355
log.success(`Loaded ${rawRecordCount.toLocaleString()} records`);
···
346
390
347
391
if (mode !== 'combined') {
348
392
log.debug(`Sorting records (reverse: ${args.reverse})...`);
349
-
records = mode === 'spotify'
350
-
? sortSpotifyRecords(records, args.reverse ?? false)
351
-
: sortRecords(records, args.reverse ?? false);
393
+
records =
394
+
mode === 'spotify'
395
+
? sortSpotifyRecords(records, args.reverse ?? false)
396
+
: sortRecords(records, args.reverse ?? false);
352
397
}
353
398
354
399
log.section('Batch Configuration');
355
400
let batchDelay = cfg.DEFAULT_BATCH_DELAY;
356
401
if (args['batch-delay']) {
357
402
const delay = parseInt(args['batch-delay'], 10);
358
-
if (isNaN(delay)) {
403
+
if (Number.isNaN(delay)) {
359
404
throw new Error(`Invalid batch delay: ${args['batch-delay']}`);
360
405
}
361
406
batchDelay = Math.max(delay, cfg.MIN_BATCH_DELAY);
···
367
412
let batchSize: number;
368
413
if (args['batch-size']) {
369
414
batchSize = parseInt(args['batch-size'], 10);
370
-
if (isNaN(batchSize) || batchSize <= 0) {
415
+
if (Number.isNaN(batchSize) || batchSize <= 0) {
371
416
throw new Error(`Invalid batch size: ${args['batch-size']}`);
372
417
}
373
418
log.info(`Using manual batch size: ${batchSize} records`);
···
391
436
const recordsPerDay = cfg.RECORDS_PER_DAY_LIMIT * safetyMargin;
392
437
const estimatedDays = Math.ceil(totalRecords / recordsPerDay);
393
438
if (estimatedDays > 1) {
394
-
log.info(`Duration: ${estimatedDays} days (${recordsPerDay.toLocaleString()} records/day limit)`);
439
+
log.info(
440
+
`Duration: ${estimatedDays} days (${recordsPerDay.toLocaleString()} records/day limit)`,
441
+
);
395
442
log.warn('Large import will span multiple days with automatic pauses');
396
443
}
397
444
log.blank();
···
432
479
if (!dryRun && !args.yes) {
433
480
const modeLabel = mode === 'combined' ? 'merged' : mode === 'sync' ? 'new' : '';
434
481
const skippedInfo = mode === 'sync' ? ` (${rawRecordCount - totalRecords} skipped)` : '';
435
-
log.raw(`Ready to publish ${totalRecords.toLocaleString()} ${modeLabel} records${skippedInfo}`);
482
+
log.raw(
483
+
`Ready to publish ${totalRecords.toLocaleString()} ${modeLabel} records${skippedInfo}`,
484
+
);
436
485
const answer = await prompt('Continue? (y/N) ');
437
486
if (answer.toLowerCase() !== 'y') {
438
487
log.info('Cancelled by user.');
···
450
499
cfg,
451
500
dryRun,
452
501
mode === 'sync' || mode === 'combined',
453
-
importState
502
+
importState,
454
503
);
455
504
456
505
log.blank();
···
462
511
} else {
463
512
const modeLabel = mode === 'combined' ? 'Combined' : mode === 'sync' ? 'Sync' : 'Import';
464
513
log.success(`${modeLabel} complete!`);
465
-
log.info(`Processed: ${result.successCount.toLocaleString()} (${result.errorCount.toLocaleString()} failed)`);
514
+
log.info(
515
+
`Processed: ${result.successCount.toLocaleString()} (${result.errorCount.toLocaleString()} failed)`,
516
+
);
466
517
if (mode === 'sync' || mode === 'combined') {
467
518
const skipped = rawRecordCount - totalRecords;
468
519
if (skipped > 0) {
+6
-6
src/lib/csv.ts
+6
-6
src/lib/csv.ts
···
1
-
import * as fs from 'fs';
1
+
import { readFileSync } from 'node:fs';
2
2
import { parse } from 'csv-parse/sync';
3
-
import type { LastFmCsvRecord, PlayRecord, Config } from '../types.js';
3
+
import { buildClientAgent } from '../config.js';
4
+
import type { Config, LastFmCsvRecord, PlayRecord } from '../types.js';
4
5
import { formatDate } from '../utils/helpers.js';
5
-
import { buildClientAgent } from '../config.js';
6
6
7
7
/**
8
8
* Parse Last.fm CSV export
9
9
*/
10
10
export function parseLastFmCsv(filePath: string): LastFmCsvRecord[] {
11
11
console.log(`Reading CSV file: ${filePath}`);
12
-
const fileContent = fs.readFileSync(filePath, 'utf-8');
12
+
const fileContent = readFileSync(filePath, 'utf-8');
13
13
14
14
const records = parse(fileContent, {
15
15
columns: true,
···
24
24
/**
25
25
* Convert Last.fm CSV record to ATProto play record
26
26
*/
27
-
export function convertToPlayRecord(csvRecord: LastFmCsvRecord, config: Config, debug = false): PlayRecord {
27
+
export function convertToPlayRecord(csvRecord: LastFmCsvRecord, config: Config): PlayRecord {
28
28
const { RECORD_TYPE } = config;
29
29
30
30
// Parse the timestamp
···
49
49
trackName: csvRecord.track,
50
50
artists,
51
51
playedTime,
52
-
submissionClientAgent: buildClientAgent(debug),
52
+
submissionClientAgent: buildClientAgent(),
53
53
musicServiceBaseDomain: 'last.fm',
54
54
originUrl: '',
55
55
};
+22
-25
src/lib/merge.ts
+22
-25
src/lib/merge.ts
···
22
22
return str
23
23
.toLowerCase()
24
24
.replace(/[^\w\s]/g, '') // Remove punctuation
25
-
.replace(/\s+/g, ' ') // Normalize whitespace
25
+
.replace(/\s+/g, ' ') // Normalize whitespace
26
26
.trim();
27
27
}
28
28
···
41
41
}
42
42
43
43
// Check normalized track and artist
44
-
return (
45
-
a.normalizedTrack === b.normalizedTrack &&
46
-
a.normalizedArtist === b.normalizedArtist
47
-
);
44
+
return a.normalizedTrack === b.normalizedTrack && a.normalizedArtist === b.normalizedArtist;
48
45
}
49
46
50
47
/**
···
53
50
*/
54
51
function chooseBetterRecord(a: NormalizedRecord, b: NormalizedRecord): PlayRecord {
55
52
// Prefer Last.fm if it has any MusicBrainz IDs
56
-
const aHasMbIds = a.source === 'lastfm' && (
57
-
a.original.recordingMbId ||
58
-
a.original.releaseMbId ||
59
-
(a.original.artists[0]?.artistMbId)
60
-
);
53
+
const aHasMbIds =
54
+
a.source === 'lastfm' &&
55
+
(a.original.recordingMbId || a.original.releaseMbId || a.original.artists[0]?.artistMbId);
61
56
62
-
const bHasMbIds = b.source === 'lastfm' && (
63
-
b.original.recordingMbId ||
64
-
b.original.releaseMbId ||
65
-
(b.original.artists[0]?.artistMbId)
66
-
);
57
+
const bHasMbIds =
58
+
b.source === 'lastfm' &&
59
+
(b.original.recordingMbId || b.original.releaseMbId || b.original.artists[0]?.artistMbId);
67
60
68
61
if (aHasMbIds && !bHasMbIds) return a.original;
69
62
if (bHasMbIds && !aHasMbIds) return b.original;
···
81
74
*/
82
75
function mergeRecords(
83
76
lastfmRecords: PlayRecord[],
84
-
spotifyRecords: PlayRecord[]
77
+
spotifyRecords: PlayRecord[],
85
78
): { merged: PlayRecord[]; stats: MergeStats } {
86
79
log.info('Merging Last.fm and Spotify exports...');
87
80
log.blank();
88
-
81
+
89
82
const stats: MergeStats = {
90
83
lastfmTotal: lastfmRecords.length,
91
84
spotifyTotal: spotifyRecords.length,
···
96
89
};
97
90
98
91
// Normalize all records
99
-
const normalizedLastFm: NormalizedRecord[] = lastfmRecords.map(record => ({
92
+
const normalizedLastFm: NormalizedRecord[] = lastfmRecords.map((record) => ({
100
93
original: record,
101
94
normalizedTrack: normalizeString(record.trackName),
102
95
normalizedArtist: normalizeString(record.artists[0]?.artistName || ''),
···
104
97
source: 'lastfm' as const,
105
98
}));
106
99
107
-
const normalizedSpotify: NormalizedRecord[] = spotifyRecords.map(record => ({
100
+
const normalizedSpotify: NormalizedRecord[] = spotifyRecords.map((record) => ({
108
101
original: record,
109
102
normalizedTrack: normalizeString(record.trackName),
110
103
normalizedArtist: normalizeString(record.artists[0]?.artistName || ''),
···
128
121
129
122
if (seen.has(key)) {
130
123
// Find the existing record to compare
131
-
const existingIndex = uniqueRecords.findIndex(r => {
124
+
const existingIndex = uniqueRecords.findIndex((r) => {
132
125
const normalized: NormalizedRecord = {
133
126
original: r,
134
127
normalizedTrack: normalizeString(r.trackName),
···
144
137
const existing: NormalizedRecord = {
145
138
original: uniqueRecords[existingIndex],
146
139
normalizedTrack: normalizeString(uniqueRecords[existingIndex].trackName),
147
-
normalizedArtist: normalizeString(uniqueRecords[existingIndex].artists[0]?.artistName || ''),
140
+
normalizedArtist: normalizeString(
141
+
uniqueRecords[existingIndex].artists[0]?.artistName || '',
142
+
),
148
143
timestamp: new Date(uniqueRecords[existingIndex].playedTime).getTime(),
149
-
source: uniqueRecords[existingIndex].musicServiceBaseDomain === 'last.fm' ? 'lastfm' : 'spotify',
144
+
source:
145
+
uniqueRecords[existingIndex].musicServiceBaseDomain === 'last.fm'
146
+
? 'lastfm'
147
+
: 'spotify',
150
148
};
151
149
152
150
uniqueRecords[existingIndex] = chooseBetterRecord(existing, record);
···
216
214
lastfmPath: string,
217
215
spotifyPath: string,
218
216
config: Config,
219
-
debug = false
220
217
): PlayRecord[] {
221
218
log.section('Combined Import Mode');
222
219
log.blank();
···
224
221
// Parse Last.fm
225
222
log.info('Parsing Last.fm export...');
226
223
const lastfmCsvRecords = parseLastFmCsv(lastfmPath);
227
-
const lastfmRecords = lastfmCsvRecords.map(r => convertToPlayRecord(r, config, debug));
224
+
const lastfmRecords = lastfmCsvRecords.map((r) => convertToPlayRecord(r, config));
228
225
229
226
// Parse Spotify
230
227
log.info('Parsing Spotify export...');
231
228
const spotifyJsonRecords = parseSpotifyJson(spotifyPath);
232
-
const spotifyRecords = spotifyJsonRecords.map(r => convertSpotifyToPlayRecord(r, config, debug));
229
+
const spotifyRecords = spotifyJsonRecords.map((r) => convertSpotifyToPlayRecord(r, config));
233
230
234
231
// Merge and deduplicate
235
232
const { merged, stats } = mergeRecords(lastfmRecords, spotifyRecords);
+48
-234
src/lib/publisher.ts
+48
-234
src/lib/publisher.ts
···
1
-
import type { AtpAgent } from '@atproto/api';
1
+
import type { AtprotoAgentInterface } from './atproto/types.js';
2
+
import { CancellationError } from './atproto/rate-limited-agent.js';
2
3
import { formatDuration, formatDate } from '../utils/helpers.js';
3
-
import { isImportCancelled } from '../utils/killswitch.js';
4
-
import {
5
-
calculateDailySchedule,
6
-
displayRateLimitWarning,
7
-
displayRateLimitInfo,
8
-
calculateRateLimitedBatches,
9
-
} from '../utils/rate-limiter.js';
10
4
import { generateTIDFromISO } from '../utils/tid.js';
11
5
import type { PlayRecord, Config, PublishResult } from '../types.js';
12
6
import { log } from '../utils/logger.js';
···
17
11
getResumeStartIndex,
18
12
} from '../utils/import-state.js';
19
13
20
-
/**
21
-
* Maximum operations allowed per applyWrites call
22
-
* PDS allows up to 200 operations per call. Each create operation costs 3 rate limit points.
23
-
* We use the full limit for maximum performance.
24
-
* See: https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/api/com/atproto/repo/applyWrites.ts
25
-
*/
26
14
const MAX_APPLY_WRITES_OPS = 200;
27
15
28
-
/**
29
-
* Publish records using com.atproto.repo.applyWrites for efficient batching
30
-
* with adaptive rate limiting and stateful resume support
31
-
*/
32
16
export async function publishRecordsWithApplyWrites(
33
-
agent: AtpAgent | null,
17
+
agent: AtprotoAgentInterface | null,
34
18
records: PlayRecord[],
35
19
batchSize: number,
36
20
batchDelay: number,
37
21
config: Config,
38
22
dryRun = false,
39
23
syncMode = false,
40
-
importState: ImportState | null = null
24
+
importState: ImportState | null = null,
41
25
): Promise<PublishResult> {
42
26
const { RECORD_TYPE } = config;
43
27
const totalRecords = records.length;
44
28
45
29
if (dryRun) {
46
-
return handleDryRun(records, batchSize, batchDelay, config, syncMode);
30
+
return handleDryRun(records, batchSize, syncMode);
47
31
}
48
32
49
33
if (!agent) {
50
34
throw new Error('Agent is required for publishing');
51
35
}
52
36
53
-
// Start with aggressive settings
54
-
let currentBatchSize = Math.min(batchSize, MAX_APPLY_WRITES_OPS);
55
-
let currentBatchDelay = batchDelay;
56
-
57
-
// Adaptive rate limiting state
58
-
let consecutiveSuccesses = 0;
59
-
let consecutiveFailures = 0;
60
-
const MAX_CONSECUTIVE_FAILURES = 3;
61
-
62
-
log.section('Conservative Adaptive Import');
63
-
log.info(`Initial batch size: ${currentBatchSize} records (conservative)`);
64
-
log.info(`Initial delay: ${currentBatchDelay}ms (2 seconds - very safe)`);
65
-
log.info(`Will automatically adjust based on server response`);
66
-
log.info(`Using conservative settings to protect your PDS`);
67
-
log.blank();
68
-
log.info(`Publishing ${totalRecords.toLocaleString()} records using adaptive batching...`);
37
+
log.section('Publishing Records');
38
+
log.info(`Publishing ${totalRecords.toLocaleString()} records...`);
69
39
log.warn('Press Ctrl+C to stop gracefully after current batch');
70
40
log.blank();
71
41
···
73
43
let errorCount = 0;
74
44
const startTime = Date.now();
75
45
76
-
// Resume from saved state if available
77
46
let startIndex = importState ? getResumeStartIndex(importState) : 0;
78
47
if (importState && startIndex > 0) {
79
-
log.info(`Resuming from record ${startIndex + 1} (${(startIndex / totalRecords * 100).toFixed(1)}% complete)`);
48
+
log.info(
49
+
`Resuming from record ${startIndex + 1} (${((startIndex / totalRecords) * 100).toFixed(1)}% complete)`,
50
+
);
80
51
log.blank();
81
52
}
82
53
83
54
let i = startIndex;
84
55
while (i < totalRecords) {
85
-
// Check killswitch before processing batch
86
-
if (isImportCancelled()) {
87
-
return handleCancellation(successCount, errorCount, totalRecords);
88
-
}
89
-
90
-
const batch = records.slice(i, Math.min(i + currentBatchSize, totalRecords));
91
-
const batchNum = Math.floor(i / currentBatchSize) + 1;
56
+
const batch = records.slice(i, Math.min(i + batchSize, totalRecords));
57
+
const batchNum = Math.floor(i / batchSize) + 1;
92
58
const progress = ((i / totalRecords) * 100).toFixed(1);
93
59
94
60
log.progress(
95
-
`[${progress}%] Batch ${batchNum} (records ${i + 1}-${Math.min(i + currentBatchSize, totalRecords)}) [size: ${currentBatchSize}, delay: ${currentBatchDelay}ms]`
61
+
`[${progress}%] Batch ${batchNum} (records ${i + 1}-${Math.min(i + batchSize, totalRecords)})`,
96
62
);
97
63
98
-
const batchStartTime = Date.now();
99
-
100
-
// Build writes array for applyWrites with TID-based rkeys
101
64
const writes = await Promise.all(
102
65
batch.map(async (record) => ({
103
66
$type: 'com.atproto.repo.applyWrites#create',
104
67
collection: RECORD_TYPE,
105
68
rkey: await generateTIDFromISO(record.playedTime, 'inject:playlist'),
106
69
value: record,
107
-
}))
70
+
})),
108
71
);
109
72
110
73
try {
111
-
// Call applyWrites with the batch
112
-
const response = await agent.com.atproto.repo.applyWrites({
113
-
repo: agent.session?.did || '',
114
-
writes: writes as any,
115
-
});
74
+
await agent.applyWrites({ writes: writes as any });
75
+
successCount += batch.length;
116
76
117
-
// Success!
118
-
const batchSuccessCount = response.data.results?.length || batch.length;
119
-
successCount += batchSuccessCount;
120
-
consecutiveSuccesses++;
121
-
consecutiveFailures = 0;
122
-
123
-
const batchDuration = Date.now() - batchStartTime;
124
-
log.debug(`Batch complete in ${batchDuration}ms (${batchSuccessCount} successful)`);
125
-
126
-
// Save state after successful batch
127
77
if (importState) {
128
-
updateImportState(importState, i + batch.length - 1, batchSuccessCount, 0);
129
-
}
130
-
131
-
// Speed up if we're doing well (after 5 consecutive successes)
132
-
if (consecutiveSuccesses >= 5 && currentBatchDelay > config.MIN_BATCH_DELAY) {
133
-
const oldDelay = currentBatchDelay;
134
-
currentBatchDelay = Math.max(
135
-
config.MIN_BATCH_DELAY,
136
-
Math.floor(currentBatchDelay * 0.8)
137
-
);
138
-
if (oldDelay !== currentBatchDelay) {
139
-
log.info(`⚡ Speeding up! Delay: ${oldDelay}ms → ${currentBatchDelay}ms`);
140
-
}
141
-
consecutiveSuccesses = 0;
78
+
updateImportState(importState, i + batch.length - 1, batch.length, 0);
142
79
}
143
80
144
81
i += batch.length;
145
-
146
82
} catch (error) {
147
-
const err = error as any;
148
-
const isRateLimitError =
149
-
err.status === 429 ||
150
-
err.message?.includes('rate limit') ||
151
-
err.message?.includes('too many requests');
152
-
153
-
consecutiveFailures++;
154
-
consecutiveSuccesses = 0;
83
+
if (error instanceof CancellationError) {
84
+
return handleCancellation(successCount, errorCount, totalRecords, importState);
85
+
}
155
86
156
-
if (isRateLimitError) {
157
-
log.warn('Rate limit hit! Backing off...');
158
-
159
-
// Exponential backoff
160
-
const backoffMultiplier = Math.pow(2, consecutiveFailures);
161
-
currentBatchDelay = Math.min(
162
-
currentBatchDelay * backoffMultiplier,
163
-
60000 // Max 60 seconds
164
-
);
165
-
166
-
// Also reduce batch size
167
-
currentBatchSize = Math.max(
168
-
Math.floor(currentBatchSize / 2),
169
-
10 // Minimum 10 records
170
-
);
171
-
172
-
log.info(`📉 Adjusted: batch size → ${currentBatchSize}, delay → ${currentBatchDelay}ms`);
173
-
log.info(`⏳ Waiting ${currentBatchDelay}ms before retry...`);
174
-
175
-
await new Promise((resolve) => setTimeout(resolve, currentBatchDelay));
176
-
177
-
// Don't advance i, retry this batch
178
-
continue;
179
-
180
-
} else {
181
-
// Other error - log and continue
182
-
errorCount += batch.length;
183
-
log.error(`Batch failed: ${err.message}`);
184
-
185
-
// Log failed records
186
-
batch.slice(0, 3).forEach((record) => {
187
-
log.debug(`Failed: ${record.trackName} by ${record.artists[0]?.artistName}`);
188
-
});
189
-
if (batch.length > 3) {
190
-
log.debug(`... and ${batch.length - 3} more failed`);
191
-
}
87
+
const err = error as Error;
88
+
errorCount += batch.length;
89
+
log.error(`Batch failed: ${err.message}`);
192
90
193
-
// Save state with errors
194
-
if (importState) {
195
-
updateImportState(importState, i + batch.length - 1, 0, batch.length);
196
-
}
91
+
batch.slice(0, 3).forEach((record) => {
92
+
log.debug(`Failed: ${record.trackName} by ${record.artists[0]?.artistName}`);
93
+
});
94
+
if (batch.length > 3) {
95
+
log.debug(`... and ${batch.length - 3} more failed`);
96
+
}
197
97
198
-
// If too many consecutive failures, slow down
199
-
if (consecutiveFailures >= MAX_CONSECUTIVE_FAILURES) {
200
-
currentBatchDelay = Math.min(currentBatchDelay * 2, 10000);
201
-
currentBatchSize = Math.max(Math.floor(currentBatchSize / 2), 10);
202
-
log.warn(`📉 Multiple failures: adjusted to ${currentBatchSize} records, ${currentBatchDelay}ms delay`);
203
-
}
204
-
205
-
i += batch.length; // Skip failed batch
98
+
if (importState) {
99
+
updateImportState(importState, i + batch.length - 1, 0, batch.length);
206
100
}
101
+
102
+
i += batch.length;
207
103
}
208
104
209
105
const elapsed = formatDuration(Date.now() - startTime);
210
106
const recordsPerSecond = successCount / ((Date.now() - startTime) / 1000);
211
107
const remainingRecords = totalRecords - i;
212
108
const estimatedRemaining = remainingRecords / Math.max(recordsPerSecond, 1);
213
-
109
+
214
110
log.debug(
215
-
`Elapsed: ${elapsed} | Speed: ${recordsPerSecond.toFixed(1)} rec/s | Remaining: ~${formatDuration(estimatedRemaining * 1000)}`
111
+
`Elapsed: ${elapsed} | Speed: ${recordsPerSecond.toFixed(1)} rec/s | Remaining: ~${formatDuration(estimatedRemaining * 1000)}`,
216
112
);
217
113
log.blank();
218
114
219
-
// Check again before waiting
220
-
if (isImportCancelled()) {
221
-
return handleCancellation(successCount, errorCount, totalRecords);
222
-
}
223
-
224
-
// Wait before next batch (except for last batch)
225
115
if (i < totalRecords) {
226
-
await new Promise((resolve) => setTimeout(resolve, currentBatchDelay));
116
+
await new Promise((resolve) => setTimeout(resolve, batchDelay));
227
117
}
228
118
}
229
119
230
-
// Mark import as complete
231
120
if (importState) {
232
121
completeImport(importState);
233
122
log.debug('Import state saved as completed');
···
236
125
return { successCount, errorCount, cancelled: false };
237
126
}
238
127
239
-
240
-
241
-
/**
242
-
* Handle dry run mode
243
-
*/
244
-
function handleDryRun(
245
-
records: PlayRecord[],
246
-
batchSize: number,
247
-
batchDelay: number,
248
-
config: Config,
249
-
syncMode: boolean
250
-
): PublishResult {
128
+
function handleDryRun(records: PlayRecord[], batchSize: number, syncMode: boolean): PublishResult {
251
129
const totalRecords = records.length;
252
130
253
-
// Calculate rate limiting info
254
-
const rateLimitParams = calculateRateLimitedBatches(totalRecords, config);
255
-
256
-
if (rateLimitParams.needsRateLimiting) {
257
-
displayRateLimitWarning();
258
-
batchSize = rateLimitParams.batchSize;
259
-
batchDelay = rateLimitParams.batchDelay;
260
-
261
-
// Ensure batch size doesn't exceed applyWrites limit
262
-
batchSize = Math.min(batchSize, MAX_APPLY_WRITES_OPS);
263
-
264
-
displayRateLimitInfo(
265
-
totalRecords,
266
-
batchSize,
267
-
batchDelay,
268
-
rateLimitParams.estimatedDays,
269
-
rateLimitParams.recordsPerDay
270
-
);
271
-
272
-
if (rateLimitParams.estimatedDays > 1) {
273
-
// Only show daily schedule in verbose/debug mode
274
-
if (log.getLevel() <= 0) { // DEBUG level
275
-
const dailySchedule = calculateDailySchedule(
276
-
totalRecords,
277
-
batchSize,
278
-
batchDelay,
279
-
rateLimitParams.recordsPerDay
280
-
);
281
-
282
-
console.log('📅 Multi-Day Import Schedule:\n');
283
-
dailySchedule.forEach((day) => {
284
-
console.log(` Day ${day.day}:`);
285
-
console.log(` Records ${day.recordsStart + 1}-${day.recordsEnd} (${day.recordsCount} total)`);
286
-
if (day.pauseAfter) {
287
-
console.log(` → Pause 24h after completion`);
288
-
}
289
-
});
290
-
console.log('');
291
-
}
292
-
}
293
-
}
294
-
295
131
log.section(`DRY RUN MODE ${syncMode ? '(SYNC)' : ''}`);
296
132
if (syncMode) {
297
133
log.info('Sync mode: Only new records will be published');
298
134
}
299
135
log.info(`Total: ${totalRecords.toLocaleString()} records`);
300
136
log.info(`Batch: ${Math.min(batchSize, MAX_APPLY_WRITES_OPS)} records per call`);
301
-
302
-
if (rateLimitParams.estimatedDays > 1) {
303
-
log.info(`Duration: ${rateLimitParams.estimatedDays} days with automatic pauses`);
304
-
} else {
305
-
log.info(`Time: ~${formatDuration(Math.ceil(totalRecords / batchSize) * batchDelay)}`);
306
-
}
307
137
log.blank();
308
138
309
-
// Show first 5 records as preview
310
139
const previewCount = Math.min(5, totalRecords);
311
140
log.info(`Preview (first ${previewCount} records):`);
312
141
log.blank();
···
314
143
for (let i = 0; i < previewCount; i++) {
315
144
const record = records[i];
316
145
const artistName = record.artists[0]?.artistName || 'Unknown Artist';
317
-
146
+
318
147
log.raw(`${i + 1}. ${artistName} - ${record.trackName}`);
319
-
320
-
// Album/Release
148
+
321
149
if (record.releaseName) {
322
150
log.raw(` Album: ${record.releaseName}`);
323
151
}
324
-
325
-
// Timestamp
152
+
326
153
log.raw(` Played: ${formatDate(record.playedTime, true)}`);
327
-
328
-
// Source and URL
329
154
log.raw(` Source: ${record.musicServiceBaseDomain}`);
330
155
log.raw(` URL: ${record.originUrl}`);
331
-
332
-
// MusicBrainz IDs (if available)
333
-
const mbids: string[] = [];
334
-
if (record.artists[0]?.artistMbId) mbids.push(`Artist: ${record.artists[0].artistMbId}`);
335
-
if (record.recordingMbId) mbids.push(`Track: ${record.recordingMbId}`);
336
-
if (record.releaseMbId) mbids.push(`Album: ${record.releaseMbId}`);
337
-
338
-
if (mbids.length > 0) {
339
-
log.raw(` MusicBrainz IDs: ${mbids.join(', ')}`);
340
-
}
341
-
342
-
// Record metadata
343
-
log.raw(` Record Type: ${record.$type}`);
344
-
log.raw(` Client: ${record.submissionClientAgent}`);
345
-
346
156
log.blank();
347
157
}
348
158
···
358
168
return { successCount: totalRecords, errorCount: 0, cancelled: false };
359
169
}
360
170
361
-
/**
362
-
* Handle cancellation
363
-
*/
364
171
function handleCancellation(
365
172
successCount: number,
366
173
errorCount: number,
367
-
totalRecords: number
174
+
totalRecords: number,
175
+
importState: ImportState | null,
368
176
): PublishResult {
369
177
log.blank();
370
178
log.warn('Import cancelled by user');
179
+
180
+
if (importState) {
181
+
updateImportState(importState, successCount - 1, 0, 0);
182
+
log.info('Import state saved - you can resume later');
183
+
}
184
+
371
185
log.info(`Processed: ${successCount.toLocaleString()}/${totalRecords.toLocaleString()} records`);
372
186
log.info(`Remaining: ${(totalRecords - successCount).toLocaleString()} records`);
373
187
return { successCount, errorCount, cancelled: true };
+24
-16
src/lib/spotify.ts
+24
-16
src/lib/spotify.ts
···
34
34
*/
35
35
export function parseSpotifyJson(filePathOrDir: string): SpotifyRecord[] {
36
36
console.log(`Reading Spotify export: ${filePathOrDir}`);
37
-
37
+
38
38
const stats = fs.statSync(filePathOrDir);
39
39
let allRecords: SpotifyRecord[] = [];
40
-
40
+
41
41
if (stats.isDirectory()) {
42
42
// Read all JSON files in the directory
43
-
const files = fs.readdirSync(filePathOrDir)
44
-
.filter(f => f.endsWith('.json') && f.startsWith('Streaming_History_Audio'))
45
-
.map(f => path.join(filePathOrDir, f));
46
-
43
+
const files = fs
44
+
.readdirSync(filePathOrDir)
45
+
.filter((f) => f.endsWith('.json') && f.startsWith('Streaming_History_Audio'))
46
+
.map((f) => path.join(filePathOrDir, f));
47
+
47
48
console.log(`Found ${files.length} Spotify JSON files in directory`);
48
-
49
+
49
50
for (const file of files) {
50
51
const fileContent = fs.readFileSync(file, 'utf-8');
51
52
const records = JSON.parse(fileContent) as SpotifyRecord[];
···
57
58
const fileContent = fs.readFileSync(filePathOrDir, 'utf-8');
58
59
allRecords = JSON.parse(fileContent) as SpotifyRecord[];
59
60
}
60
-
61
+
61
62
// Filter out records without track names (podcasts, audiobooks, etc.)
62
-
const trackRecords = allRecords.filter(r =>
63
-
r.master_metadata_track_name &&
64
-
r.master_metadata_album_artist_name
63
+
const trackRecords = allRecords.filter(
64
+
(r) => r.master_metadata_track_name && r.master_metadata_album_artist_name,
65
65
);
66
-
67
-
console.log(`✓ Parsed ${trackRecords.length} track records (filtered ${allRecords.length - trackRecords.length} non-music records)\n`);
66
+
67
+
console.log(
68
+
`✓ Parsed ${trackRecords.length} track records (filtered ${allRecords.length - trackRecords.length} non-music records)\n`,
69
+
);
68
70
return trackRecords;
69
71
}
70
72
71
73
/**
72
74
* Convert Spotify record to ATProto play record
73
75
*/
74
-
export function convertSpotifyToPlayRecord(spotifyRecord: SpotifyRecord, config: Config, debug = false): PlayRecord {
76
+
export function convertSpotifyToPlayRecord(
77
+
spotifyRecord: SpotifyRecord,
78
+
config: Config,
79
+
): PlayRecord {
75
80
const { RECORD_TYPE } = config;
76
81
77
82
// Spotify timestamp is already in ISO 8601 format
···
91
96
trackName: spotifyRecord.master_metadata_track_name || 'Unknown Track',
92
97
artists,
93
98
playedTime,
94
-
submissionClientAgent: buildClientAgent(debug),
99
+
submissionClientAgent: buildClientAgent(),
95
100
musicServiceBaseDomain: 'spotify.com',
96
101
originUrl: '',
97
102
};
···
113
118
/**
114
119
* Sort records chronologically
115
120
*/
116
-
export function sortSpotifyRecords(records: PlayRecord[], reverseChronological = false): PlayRecord[] {
121
+
export function sortSpotifyRecords(
122
+
records: PlayRecord[],
123
+
reverseChronological = false,
124
+
): PlayRecord[] {
117
125
console.log(`Sorting records ${reverseChronological ? 'newest' : 'oldest'} first...`);
118
126
119
127
records.sort((a, b) => {
+87
-215
src/lib/sync.ts
+87
-215
src/lib/sync.ts
···
1
-
import type { AtpAgent } from '@atproto/api';
1
+
import type { AtprotoAgentInterface } from './atproto/types.js';
2
2
import type { PlayRecord, Config } from '../types.js';
3
3
import { formatDate, formatDateRange } from '../utils/helpers.js';
4
4
import * as ui from '../utils/ui.js';
5
5
import { log } from '../utils/logger.js';
6
-
import { isImportCancelled } from '../utils/killswitch.js';
7
6
import { isCacheValid, loadCache, saveCache, getCacheInfo } from '../utils/teal-cache.js';
8
7
9
8
interface ExistingRecord {
···
17
16
records: ExistingRecord[];
18
17
}
19
18
20
-
/**
21
-
* Fetch all existing play records from Teal
22
-
* Returns a Map where each key can have multiple records (for duplicate detection)
23
-
*/
19
+
const DEFAULT_BATCH_SIZE = 100;
20
+
24
21
export async function fetchExistingRecords(
25
-
agent: AtpAgent,
22
+
agent: AtprotoAgentInterface,
26
23
config: Config,
27
-
forceRefresh: boolean = false
24
+
forceRefresh: boolean = false,
28
25
): Promise<Map<string, ExistingRecord>> {
29
26
log.section('Checking Existing Records');
30
27
const { RECORD_TYPE } = config;
31
-
const did = agent.session?.did;
28
+
const did = agent.getDid();
32
29
33
30
if (!did) {
34
31
throw new Error('No authenticated session found');
35
32
}
36
33
37
-
// Check cache first (unless force refresh)
38
34
if (!forceRefresh && isCacheValid(did)) {
39
35
const cacheInfo = getCacheInfo(did);
40
-
log.info(`📂 Loading from cache (${cacheInfo.age!.toFixed(1)}h old, ${cacheInfo.records!.toLocaleString()} records)...`);
41
-
36
+
log.info(
37
+
`📂 Loading from cache (${cacheInfo.age!.toFixed(1)}h old, ${cacheInfo.records!.toLocaleString()} records)...`,
38
+
);
39
+
42
40
const cached = loadCache(did);
43
41
if (cached) {
44
-
// Convert cached records to the format we need
45
42
const existingRecords = new Map<string, ExistingRecord>();
46
43
for (const [, record] of cached.entries()) {
47
44
const playRecord = record.value as PlayRecord;
48
45
const key = createRecordKey(playRecord);
49
46
existingRecords.set(key, record as ExistingRecord);
50
47
}
51
-
48
+
52
49
log.success(`✓ Loaded ${existingRecords.size.toLocaleString()} records from cache`);
53
50
log.blank();
54
51
return existingRecords;
55
52
}
56
53
}
57
54
58
-
// Cache miss or force refresh - fetch from Teal
59
55
if (forceRefresh) {
60
56
log.info('🔄 Force refresh - fetching from Teal...');
61
57
} else {
62
58
log.info('Fetching records from Teal to avoid duplicates...');
63
59
}
64
-
60
+
65
61
const existingRecords = new Map<string, ExistingRecord>();
66
62
const cacheMap = new Map<string, { uri: string; cid: string; value: any }>();
67
63
let cursor: string | undefined = undefined;
68
64
let totalFetched = 0;
69
65
const startTime = Date.now();
70
66
71
-
// Adaptive batch sizing
72
-
let batchSize = 25; // Start conservative
73
-
let consecutiveFastRequests = 0;
74
-
let consecutiveSlowRequests = 0;
75
-
const TARGET_LATENCY_MS = 2000; // Target 2s per request
76
-
const MIN_BATCH_SIZE = 10;
77
-
const MAX_BATCH_SIZE = 100; // AT Protocol maximum
78
-
let requestCount = 0;
79
-
80
67
try {
81
-
// Fetch records in batches using listRecords with adaptive sizing
82
68
do {
83
-
// Check for cancellation
84
-
if (isImportCancelled()) {
85
-
log.warn('Fetch cancelled by user');
86
-
throw new Error('Operation cancelled by user');
87
-
}
88
-
89
-
requestCount++;
90
-
const requestStart = Date.now();
91
-
92
-
log.debug(`Request #${requestCount}: Fetching batch of ${batchSize}...`);
93
-
94
-
const response = await agent.com.atproto.repo.listRecords({
69
+
const response = await agent.listRecords({
95
70
repo: did,
96
71
collection: RECORD_TYPE,
97
-
limit: batchSize,
72
+
limit: DEFAULT_BATCH_SIZE,
98
73
cursor: cursor,
99
74
});
100
75
101
-
const requestLatency = Date.now() - requestStart;
102
-
const records = response.data.records;
103
-
104
-
log.debug(`Request #${requestCount}: Got ${records.length} records in ${requestLatency}ms`);
76
+
const records = response.records;
105
77
106
-
// Batch process records for better performance
107
78
for (const record of records) {
108
79
const playRecord = record.value as unknown as PlayRecord;
109
80
const key = createRecordKey(playRecord);
···
113
84
value: playRecord,
114
85
};
115
86
existingRecords.set(key, existingRecord);
116
-
// Also store for cache (using URI as key for cache)
117
87
cacheMap.set(record.uri, existingRecord);
118
88
}
119
89
120
90
totalFetched += records.length;
121
-
cursor = response.data.cursor;
91
+
cursor = response.cursor;
122
92
123
-
// Adaptive batch size adjustment based on latency
124
-
if (requestLatency < TARGET_LATENCY_MS) {
125
-
// Request was fast - try to increase batch size
126
-
consecutiveFastRequests++;
127
-
consecutiveSlowRequests = 0;
128
-
129
-
if (consecutiveFastRequests >= 3 && batchSize < MAX_BATCH_SIZE) {
130
-
const oldSize = batchSize;
131
-
batchSize = Math.min(MAX_BATCH_SIZE, Math.floor(batchSize * 1.5));
132
-
if (oldSize !== batchSize) {
133
-
log.info(`⚡ Network performing well - increased batch size: ${oldSize} → ${batchSize}`);
134
-
}
135
-
consecutiveFastRequests = 0;
136
-
}
137
-
} else {
138
-
// Request was slow - decrease batch size
139
-
consecutiveSlowRequests++;
140
-
consecutiveFastRequests = 0;
141
-
142
-
if (consecutiveSlowRequests >= 2 && batchSize > MIN_BATCH_SIZE) {
143
-
const oldSize = batchSize;
144
-
batchSize = Math.max(MIN_BATCH_SIZE, Math.floor(batchSize * 0.7));
145
-
log.info(`🐌 Network slow - decreased batch size: ${oldSize} → ${batchSize}`);
146
-
consecutiveSlowRequests = 0;
147
-
}
148
-
}
149
-
150
-
// Show progress every 250 records or every request if less than 1000 total
151
93
const showProgress = totalFetched % 250 === 0 && totalFetched > 0;
152
94
if (showProgress || totalFetched < 1000) {
153
95
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
154
-
const rate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0);
155
-
log.progress(`Fetched ${totalFetched.toLocaleString()} records (${rate} rec/s, batch: ${batchSize}, ${elapsed}s)...`);
96
+
const rate = ((totalFetched / (Date.now() - startTime)) * 1000).toFixed(0);
97
+
log.progress(
98
+
`Fetched ${totalFetched.toLocaleString()} records (${rate} rec/s, ${elapsed}s)...`,
99
+
);
156
100
}
157
101
} while (cursor);
158
102
159
103
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
160
-
const avgRate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0);
161
-
log.success(`Found ${existingRecords.size.toLocaleString()} existing records in ${elapsed}s (avg ${avgRate} rec/s)`);
162
-
163
-
// Save to cache
164
-
log.debug('Saving records to cache...');
104
+
const avgRate = ((totalFetched / (Date.now() - startTime)) * 1000).toFixed(0);
105
+
log.success(
106
+
`Found ${existingRecords.size.toLocaleString()} existing records in ${elapsed}s (avg ${avgRate} rec/s)`,
107
+
);
108
+
165
109
saveCache(did, cacheMap);
166
-
110
+
167
111
log.blank();
168
112
return existingRecords;
169
113
} catch (error) {
···
173
117
}
174
118
}
175
119
176
-
/**
177
-
* Fetch all existing play records as an array (for duplicate detection)
178
-
* This version keeps ALL records, including duplicates
179
-
*/
180
120
export async function fetchAllRecords(
181
-
agent: AtpAgent,
182
-
config: Config
121
+
agent: AtprotoAgentInterface,
122
+
config: Config,
183
123
): Promise<ExistingRecord[]> {
184
124
const { RECORD_TYPE } = config;
185
-
const did = agent.session?.did;
125
+
const did = agent.getDid();
186
126
187
127
if (!did) {
188
128
throw new Error('No authenticated session found');
···
194
134
let totalFetched = 0;
195
135
const startTime = Date.now();
196
136
197
-
// Adaptive batch sizing
198
-
let batchSize = 25; // Start conservative
199
-
let consecutiveFastRequests = 0;
200
-
let consecutiveSlowRequests = 0;
201
-
const TARGET_LATENCY_MS = 2000; // Target 2s per request
202
-
const MIN_BATCH_SIZE = 10;
203
-
const MAX_BATCH_SIZE = 100; // AT Protocol maximum
204
-
let requestCount = 0;
205
-
206
137
try {
207
-
// Fetch records in batches using listRecords with adaptive sizing
208
138
do {
209
-
// Check for cancellation
210
-
if (isImportCancelled()) {
211
-
ui.failSpinner('Fetch cancelled by user');
212
-
throw new Error('Operation cancelled by user');
213
-
}
214
-
215
-
requestCount++;
216
-
const requestStart = Date.now();
217
-
218
-
const response = await agent.com.atproto.repo.listRecords({
139
+
const response = await agent.listRecords({
219
140
repo: did,
220
141
collection: RECORD_TYPE,
221
-
limit: batchSize,
142
+
limit: DEFAULT_BATCH_SIZE,
222
143
cursor: cursor,
223
144
});
224
145
225
-
const requestLatency = Date.now() - requestStart;
226
-
const records = response.data.records;
146
+
const records = response.records;
227
147
for (const record of records) {
228
148
const playRecord = record.value as unknown as PlayRecord;
229
149
allRecords.push({
···
234
154
}
235
155
236
156
totalFetched += records.length;
237
-
cursor = response.data.cursor;
238
-
239
-
// Adaptive batch size adjustment based on latency
240
-
if (requestLatency < TARGET_LATENCY_MS) {
241
-
// Request was fast - try to increase batch size
242
-
consecutiveFastRequests++;
243
-
consecutiveSlowRequests = 0;
244
-
245
-
if (consecutiveFastRequests >= 3 && batchSize < MAX_BATCH_SIZE) {
246
-
batchSize = Math.min(MAX_BATCH_SIZE, Math.floor(batchSize * 1.5));
247
-
consecutiveFastRequests = 0;
248
-
}
249
-
} else {
250
-
// Request was slow - decrease batch size
251
-
consecutiveSlowRequests++;
252
-
consecutiveFastRequests = 0;
157
+
cursor = response.cursor;
253
158
254
-
if (consecutiveSlowRequests >= 2 && batchSize > MIN_BATCH_SIZE) {
255
-
batchSize = Math.max(MIN_BATCH_SIZE, Math.floor(batchSize * 0.7));
256
-
consecutiveSlowRequests = 0;
257
-
}
258
-
}
259
-
260
-
// Update spinner with progress every 250 records or every request if less than 1000 total
261
159
const showProgress = totalFetched % 250 === 0 && totalFetched > 0;
262
160
if (showProgress || totalFetched < 1000) {
263
161
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
264
-
const rate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0);
265
-
ui.updateSpinner(`Fetching records... ${totalFetched.toLocaleString()} found (${rate} rec/s, batch: ${batchSize}, ${elapsed}s)`);
162
+
const rate = ((totalFetched / (Date.now() - startTime)) * 1000).toFixed(0);
163
+
ui.updateSpinner(
164
+
`Fetching records... ${totalFetched.toLocaleString()} found (${rate} rec/s, ${elapsed}s)`,
165
+
);
266
166
}
267
167
} while (cursor);
268
168
269
169
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
270
-
const avgRate = (totalFetched / (Date.now() - startTime) * 1000).toFixed(0);
271
-
ui.succeedSpinner(`Found ${allRecords.length.toLocaleString()} total records in ${elapsed}s (avg ${avgRate} rec/s)`);
170
+
const avgRate = ((totalFetched / (Date.now() - startTime)) * 1000).toFixed(0);
171
+
ui.succeedSpinner(
172
+
`Found ${allRecords.length.toLocaleString()} total records in ${elapsed}s (avg ${avgRate} rec/s)`,
173
+
);
272
174
return allRecords;
273
175
} catch (error) {
274
176
ui.failSpinner('Failed to fetch existing records');
···
276
178
}
277
179
}
278
180
279
-
/**
280
-
* Create a unique key for a play record based on its essential properties
281
-
* This is used to identify duplicates
282
-
*/
283
181
export function createRecordKey(record: PlayRecord): string {
284
182
const artist = record.artists[0]?.artistName || '';
285
183
const track = record.trackName;
286
184
const timestamp = record.playedTime;
287
185
288
-
// Normalize strings to handle case and whitespace differences
289
186
const normalizedArtist = artist.toLowerCase().trim();
290
187
const normalizedTrack = track.toLowerCase().trim();
291
188
292
189
return `${normalizedArtist}|||${normalizedTrack}|||${timestamp}`;
293
190
}
294
191
295
-
/**
296
-
* Deduplicate input records before submission
297
-
* Keeps the first occurrence of each duplicate
298
-
*/
299
-
export function deduplicateInputRecords(records: PlayRecord[]): { unique: PlayRecord[]; duplicates: number } {
192
+
export function deduplicateInputRecords(records: PlayRecord[]): {
193
+
unique: PlayRecord[];
194
+
duplicates: number;
195
+
} {
300
196
const seen = new Map<string, PlayRecord>();
301
197
const duplicates: PlayRecord[] = [];
302
198
···
311
207
312
208
return {
313
209
unique: Array.from(seen.values()),
314
-
duplicates: duplicates.length
210
+
duplicates: duplicates.length,
315
211
};
316
212
}
317
213
318
-
/**
319
-
* Filter out records that already exist in Teal
320
-
*/
321
214
export function filterNewRecords(
322
215
lastfmRecords: PlayRecord[],
323
-
existingRecords: Map<string, ExistingRecord>
216
+
existingRecords: Map<string, ExistingRecord>,
324
217
): PlayRecord[] {
325
218
log.section('Identifying New Records');
326
219
···
341
234
log.info(`New: ${newRecords.length.toLocaleString()} to import`);
342
235
log.blank();
343
236
344
-
// Show some examples of duplicates if any (only in verbose mode)
345
-
if (log.getLevel() <= 0 && duplicates.length > 0) { // DEBUG level
237
+
if (log.getLevel() <= 0 && duplicates.length > 0) {
346
238
const exampleCount = Math.min(3, duplicates.length);
347
239
log.debug('Examples of existing records (skipped):');
348
240
duplicates.slice(0, exampleCount).forEach((record, i) => {
···
358
250
return newRecords;
359
251
}
360
252
361
-
/**
362
-
* Get time range of records
363
-
*/
364
253
export function getRecordTimeRange(records: PlayRecord[]): { earliest: Date; latest: Date } | null {
365
254
if (records.length === 0) {
366
255
return null;
367
256
}
368
257
369
-
const times = records.map(r => new Date(r.playedTime).getTime());
258
+
const times = records.map((r) => new Date(r.playedTime).getTime());
370
259
const earliest = new Date(Math.min(...times));
371
260
const latest = new Date(Math.max(...times));
372
261
373
262
return { earliest, latest };
374
263
}
375
264
376
-
/**
377
-
* Display sync statistics
378
-
*/
379
265
export function displaySyncStats(
380
266
lastfmRecords: PlayRecord[],
381
267
existingRecords: Map<string, ExistingRecord>,
382
-
newRecords: PlayRecord[]
268
+
newRecords: PlayRecord[],
383
269
): void {
384
270
const lastfmRange = getRecordTimeRange(lastfmRecords);
385
-
const existingArray = Array.from(existingRecords.values()).map(r => r.value);
271
+
const existingArray = Array.from(existingRecords.values()).map((r) => r.value);
386
272
const existingRange = getRecordTimeRange(existingArray);
387
273
388
274
log.section('Sync Statistics');
···
404
290
log.blank();
405
291
}
406
292
407
-
/**
408
-
* Find duplicate records in the existing records
409
-
* Returns groups of duplicates (where each group has 2+ records with the same key)
410
-
*/
411
-
export function findDuplicates(
412
-
allRecords: ExistingRecord[]
413
-
): DuplicateGroup[] {
293
+
export function findDuplicates(allRecords: ExistingRecord[]): DuplicateGroup[] {
414
294
const keyGroups = new Map<string, ExistingRecord[]>();
415
-
416
-
// Group records by their key
295
+
417
296
for (const record of allRecords) {
418
297
const key = createRecordKey(record.value);
419
298
if (!keyGroups.has(key)) {
···
421
300
}
422
301
keyGroups.get(key)!.push(record);
423
302
}
424
-
425
-
// Filter to only groups with duplicates (2+ records)
303
+
426
304
const duplicates: DuplicateGroup[] = [];
427
305
for (const [key, records] of keyGroups) {
428
306
if (records.length > 1) {
429
307
duplicates.push({ key, records });
430
308
}
431
309
}
432
-
310
+
433
311
return duplicates;
434
312
}
435
313
436
-
/**
437
-
* Remove duplicate records from Teal, keeping only the first occurrence
438
-
*/
439
314
export async function removeDuplicates(
440
-
agent: AtpAgent,
315
+
agent: AtprotoAgentInterface,
441
316
config: Config,
442
-
dryRun: boolean = false
317
+
dryRun: boolean = false,
443
318
): Promise<{ totalDuplicates: number; recordsRemoved: number }> {
444
319
ui.header('Checking for Duplicate Records');
445
-
446
-
// Fetch ALL records (including duplicates)
320
+
447
321
const allRecords = await fetchAllRecords(agent, config);
448
-
322
+
449
323
ui.startSpinner('Analyzing records for duplicates...');
450
324
const duplicateGroups = findDuplicates(allRecords);
451
-
325
+
452
326
if (duplicateGroups.length === 0) {
453
327
ui.succeedSpinner('No duplicates found!');
454
328
return { totalDuplicates: 0, recordsRemoved: 0 };
455
329
}
456
-
330
+
457
331
ui.stopSpinner();
458
-
459
-
// Count total duplicate records (excluding the one we keep per group)
460
-
const totalDuplicates = duplicateGroups.reduce((sum, group) => sum + (group.records.length - 1), 0);
461
-
462
-
ui.warning(`Found ${duplicateGroups.length.toLocaleString()} duplicate groups (${totalDuplicates.toLocaleString()} records to remove)`);
332
+
333
+
const totalDuplicates = duplicateGroups.reduce(
334
+
(sum, group) => sum + (group.records.length - 1),
335
+
0,
336
+
);
337
+
338
+
ui.warning(
339
+
`Found ${duplicateGroups.length.toLocaleString()} duplicate groups (${totalDuplicates.toLocaleString()} records to remove)`,
340
+
);
463
341
console.log('');
464
-
465
-
// Show examples
342
+
466
343
const exampleCount = Math.min(5, duplicateGroups.length);
467
344
ui.subheader('Examples of Duplicates:');
468
345
for (let i = 0; i < exampleCount; i++) {
469
346
const group = duplicateGroups[i];
470
347
const firstRecord = group.records[0].value;
471
348
console.log(` ${i + 1}. ${firstRecord.artists[0]?.artistName} - ${firstRecord.trackName}`);
472
-
console.log(` ${formatDate(firstRecord.playedTime, true)} · ${group.records.length - 1} duplicate(s)`);
349
+
console.log(
350
+
` ${formatDate(firstRecord.playedTime, true)} · ${group.records.length - 1} duplicate(s)`,
351
+
);
473
352
}
474
-
353
+
475
354
if (duplicateGroups.length > exampleCount) {
476
355
console.log(` ... and ${duplicateGroups.length - exampleCount} more groups`);
477
356
}
478
357
console.log('');
479
-
358
+
480
359
if (dryRun) {
481
360
ui.info('DRY RUN: No records were removed.');
482
361
return { totalDuplicates, recordsRemoved: 0 };
483
362
}
484
-
485
-
// Remove duplicates (keep first, delete rest)
363
+
486
364
console.log('');
487
365
const progressBar = ui.createProgressBar(totalDuplicates, 'Removing duplicates');
488
366
let recordsRemoved = 0;
489
367
const startTime = Date.now();
490
-
368
+
491
369
for (const group of duplicateGroups) {
492
-
// Keep the first record, delete the rest
493
370
const toDelete = group.records.slice(1);
494
-
371
+
495
372
for (const record of toDelete) {
496
373
try {
497
-
await agent.com.atproto.repo.deleteRecord({
498
-
repo: agent.session?.did || '',
374
+
await agent.deleteRecord({
375
+
repo: agent.getDid() || '',
499
376
collection: record.value.$type,
500
377
rkey: record.uri.split('/').pop()!,
501
378
});
502
379
recordsRemoved++;
503
-
504
-
// Update progress bar
380
+
505
381
const elapsed = (Date.now() - startTime) / 1000;
506
382
const speed = recordsRemoved / Math.max(elapsed, 0.1);
507
383
progressBar.update(recordsRemoved, { speed });
508
-
509
-
} catch (error) {
510
-
// Silently continue on errors
511
-
}
512
-
513
-
// Small delay between deletions
514
-
await new Promise(resolve => setTimeout(resolve, 100));
384
+
} catch (error) {}
385
+
386
+
await new Promise((resolve) => setTimeout(resolve, 100));
515
387
}
516
388
}
517
-
389
+
518
390
progressBar.stop();
519
391
console.log('');
520
392
ui.success(`Removed ${recordsRemoved.toLocaleString()} duplicate records`);
521
393
ui.info(`Kept ${duplicateGroups.length.toLocaleString()} unique records`);
522
-
394
+
523
395
return { totalDuplicates, recordsRemoved };
524
396
}
+185
src/tests/atproto/rate-limited-agent.test.ts
+185
src/tests/atproto/rate-limited-agent.test.ts
···
1
+
import assert from 'node:assert';
2
+
import { describe, it } from 'node:test';
3
+
import { MockAgent } from '../../lib/atproto/mock-agent.js';
4
+
import { RateLimitedAgent, CancellationError } from '../../lib/atproto/rate-limited-agent.js';
5
+
6
+
describe('RateLimitedAgent', () => {
7
+
describe('point tracking', () => {
8
+
it('should track daily and hourly points separately', () => {
9
+
const mock = new MockAgent({ did: 'did:test:123' });
10
+
const agent = new RateLimitedAgent(mock, {
11
+
pointsPerDay: 100,
12
+
safetyMargin: 1.0,
13
+
pointsPerWrite: 3,
14
+
pointsPerRead: 1,
15
+
pointsPerDelete: 1,
16
+
});
17
+
18
+
const status = agent.getRateLimitStatus();
19
+
assert.strictEqual(status.dailyPointsRemaining, 100);
20
+
assert.strictEqual(status.hourlyPointsRemaining, 4);
21
+
});
22
+
23
+
it('should consume points on writes', async () => {
24
+
const mock = new MockAgent({ did: 'did:test:123' });
25
+
const agent = new RateLimitedAgent(mock, {
26
+
pointsPerDay: 10000,
27
+
safetyMargin: 1.0,
28
+
pointsPerWrite: 3,
29
+
pointsPerRead: 1,
30
+
pointsPerDelete: 1,
31
+
});
32
+
33
+
await agent.applyWrites({
34
+
writes: [
35
+
{ $type: 'create', collection: 'test', rkey: '1', value: {} },
36
+
{ $type: 'create', collection: 'test', rkey: '2', value: {} },
37
+
{ $type: 'create', collection: 'test', rkey: '3', value: {} },
38
+
],
39
+
});
40
+
41
+
const status = agent.getRateLimitStatus();
42
+
assert.strictEqual(status.dailyPointsRemaining, 9991);
43
+
});
44
+
45
+
it('should consume 1 point per read', async () => {
46
+
const mock = new MockAgent({ did: 'did:test:123' });
47
+
const agent = new RateLimitedAgent(mock, {
48
+
pointsPerDay: 10000,
49
+
safetyMargin: 1.0,
50
+
pointsPerWrite: 3,
51
+
pointsPerRead: 1,
52
+
pointsPerDelete: 1,
53
+
});
54
+
55
+
await agent.listRecords({ collection: 'test' });
56
+
57
+
const status = agent.getRateLimitStatus();
58
+
assert.strictEqual(status.dailyPointsRemaining, 9999);
59
+
});
60
+
61
+
it('should consume 1 point per delete', async () => {
62
+
const mock = new MockAgent({ did: 'did:test:123' });
63
+
const agent = new RateLimitedAgent(mock, {
64
+
pointsPerDay: 10000,
65
+
safetyMargin: 1.0,
66
+
pointsPerWrite: 3,
67
+
pointsPerRead: 1,
68
+
pointsPerDelete: 1,
69
+
});
70
+
71
+
await agent.deleteRecord({ collection: 'test', rkey: 'abc123' });
72
+
73
+
const status = agent.getRateLimitStatus();
74
+
assert.strictEqual(status.dailyPointsRemaining, 9999);
75
+
});
76
+
});
77
+
78
+
describe('reset timers', () => {
79
+
it('should reset daily points at midnight', () => {
80
+
const mock = new MockAgent({ did: 'did:test:123' });
81
+
const agent = new RateLimitedAgent(mock, {
82
+
pointsPerDay: 100,
83
+
safetyMargin: 1.0,
84
+
pointsPerWrite: 3,
85
+
});
86
+
87
+
const status = agent.getRateLimitStatus();
88
+
assert.ok(status.dailyResetAt > new Date());
89
+
});
90
+
91
+
it('should reset hourly points at the next hour', () => {
92
+
const mock = new MockAgent({ did: 'did:test:123' });
93
+
const agent = new RateLimitedAgent(mock, {
94
+
pointsPerDay: 100,
95
+
safetyMargin: 1.0,
96
+
pointsPerWrite: 3,
97
+
});
98
+
99
+
const status = agent.getRateLimitStatus();
100
+
assert.ok(status.hourlyResetAt > new Date());
101
+
assert.ok(status.hourlyResetAt <= new Date(Date.now() + 60 * 60 * 1000));
102
+
});
103
+
});
104
+
105
+
describe('getDid', () => {
106
+
it('should return DID from delegate', () => {
107
+
const mock = new MockAgent({ did: 'did:test:abc123' });
108
+
const agent = new RateLimitedAgent(mock);
109
+
110
+
assert.strictEqual(agent.getDid(), 'did:test:abc123');
111
+
});
112
+
});
113
+
114
+
describe('passthrough behavior', () => {
115
+
it('should pass through listRecords calls correctly', async () => {
116
+
const mock = new MockAgent({ did: 'did:test:123' });
117
+
const agent = new RateLimitedAgent(mock);
118
+
119
+
const response = await agent.listRecords({ collection: 'test' });
120
+
assert.ok(Array.isArray(response.records));
121
+
assert.strictEqual(response.records.length, 1);
122
+
});
123
+
124
+
it('should pass through applyWrites calls correctly', async () => {
125
+
const mock = new MockAgent({ did: 'did:test:123' });
126
+
const agent = new RateLimitedAgent(mock);
127
+
128
+
const response = await agent.applyWrites({
129
+
writes: [
130
+
{ $type: 'create', collection: 'test', rkey: '1', value: {} },
131
+
{ $type: 'create', collection: 'test', rkey: '2', value: {} },
132
+
],
133
+
});
134
+
135
+
assert.ok(response.data.results);
136
+
assert.strictEqual(response.data.results!.length, 2);
137
+
});
138
+
139
+
it('should pass through deleteRecord calls correctly', async () => {
140
+
const mock = new MockAgent({ did: 'did:test:123' });
141
+
const agent = new RateLimitedAgent(mock);
142
+
143
+
const response = await agent.deleteRecord({
144
+
collection: 'test',
145
+
rkey: 'abc123',
146
+
});
147
+
148
+
assert.ok(response.data.uri.includes('abc123'));
149
+
});
150
+
});
151
+
152
+
describe('cancellation', () => {
153
+
it('should throw CancellationError when signal is aborted', async () => {
154
+
const mock = new MockAgent({ did: 'did:test:123' });
155
+
const abortController = new AbortController();
156
+
const agent = new RateLimitedAgent(mock, {
157
+
signal: abortController.signal,
158
+
});
159
+
160
+
abortController.abort();
161
+
162
+
await assert.rejects(
163
+
async () => agent.listRecords({ collection: 'test' }),
164
+
CancellationError,
165
+
);
166
+
});
167
+
168
+
it('should cancel during waitForReset', async () => {
169
+
const mock = new MockAgent({ did: 'did:test:123', delay: 100 });
170
+
const abortController = new AbortController();
171
+
const agent = new RateLimitedAgent(mock, {
172
+
pointsPerDay: 1,
173
+
safetyMargin: 1.0,
174
+
signal: abortController.signal,
175
+
});
176
+
177
+
setTimeout(() => abortController.abort(), 10);
178
+
179
+
await assert.rejects(
180
+
async () => agent.listRecords({ collection: 'test' }),
181
+
CancellationError,
182
+
);
183
+
});
184
+
});
185
+
});
+18
-24
src/tests/tid-integration.test.ts
+18
-24
src/tests/tid-integration.test.ts
···
1
1
/**
2
2
* Integration test for TID generation in the importer
3
-
*
3
+
*
4
4
* Tests the full flow with actual CSV data
5
5
*/
6
6
···
27
27
resetTidClock();
28
28
initTidClock({ mode: 'dry-run', seed });
29
29
30
-
const dates = [
31
-
'2005-01-01T00:00:00Z',
32
-
'2010-01-01T00:00:00Z',
33
-
'2015-01-01T00:00:00Z',
34
-
];
30
+
const dates = ['2005-01-01T00:00:00Z', '2010-01-01T00:00:00Z', '2015-01-01T00:00:00Z'];
35
31
36
-
const tids1 = await Promise.all(dates.map(d => generateTIDFromISO(d, 'test')));
32
+
const tids1 = await Promise.all(dates.map((d) => generateTIDFromISO(d, 'test')));
37
33
38
34
// Reset and regenerate
39
35
resetTidClock();
40
36
initTidClock({ mode: 'dry-run', seed });
41
37
42
-
const tids2 = await Promise.all(dates.map(d => generateTIDFromISO(d, 'test')));
38
+
const tids2 = await Promise.all(dates.map((d) => generateTIDFromISO(d, 'test')));
43
39
44
40
// Should be identical
45
41
assert.deepStrictEqual(tids1, tids2);
···
56
52
}));
57
53
58
54
const tids = await Promise.all(
59
-
records.map(r => generateTIDFromISO(r.playedTime, 'batch:test'))
55
+
records.map((r) => generateTIDFromISO(r.playedTime, 'batch:test')),
60
56
);
61
57
62
58
// All should be valid
···
82
78
83
79
// Simulate Last.fm scrobbles from different eras
84
80
const historicalDates = [
85
-
'2005-03-15T14:30:00Z', // Very old
81
+
'2005-03-15T14:30:00Z', // Very old
86
82
'2010-06-20T08:15:00Z',
87
83
'2015-11-05T19:45:00Z',
88
84
'2020-01-01T00:00:00Z',
89
-
'2024-12-25T12:00:00Z', // Recent
85
+
'2024-12-25T12:00:00Z', // Recent
90
86
];
91
87
92
-
const tids = await Promise.all(
93
-
historicalDates.map(d => generateTIDFromISO(d, 'historical'))
94
-
);
88
+
const tids = await Promise.all(historicalDates.map((d) => generateTIDFromISO(d, 'historical')));
95
89
96
90
// All valid
97
91
for (const tid of tids) {
···
112
106
113
107
const tids = await Promise.all(
114
108
Array.from({ length: 100 }, (_, i) =>
115
-
generateTIDFromISO(new Date(2020, 0, 1, 0, 0, i).toISOString(), 'audit-test')
116
-
)
109
+
generateTIDFromISO(new Date(2020, 0, 1, 0, 0, i).toISOString(), 'audit-test'),
110
+
),
117
111
);
118
112
119
113
const report = auditTids(tids);
···
133
127
it('should detect problems in TID audit', async () => {
134
128
// Create a bad TID list
135
129
const badTids = [
136
-
'3jzfcijpj2z2a', // Valid
137
-
'3jzfcijpj2z2b', // Valid
138
-
'3jzfcijpj2z2a', // Duplicate!
139
-
'0000000000000', // Invalid format
140
-
'3jzfcijpj2z2c', // Valid
141
-
'3jzfcijpj2z2b', // Out of order
130
+
'3jzfcijpj2z2a', // Valid
131
+
'3jzfcijpj2z2b', // Valid
132
+
'3jzfcijpj2z2a', // Duplicate!
133
+
'0000000000000', // Invalid format
134
+
'3jzfcijpj2z2c', // Valid
135
+
'3jzfcijpj2z2b', // Out of order
142
136
];
143
137
144
138
const report = auditTids(badTids);
···
170
164
Array.from({ length: batchSize }, (_, recordIdx) => {
171
165
const date = new Date(2020, 0, 1, batchIdx, recordIdx).toISOString();
172
166
return generateTIDFromISO(date, `batch-${batchIdx}`);
173
-
})
167
+
}),
174
168
);
175
-
})
169
+
}),
176
170
);
177
171
178
172
const flatTids = allTids.flat();
+56
-71
src/tests/tid.test.ts
+56
-71
src/tests/tid.test.ts
···
1
1
/**
2
2
* Unit tests for TID generation
3
-
*
3
+
*
4
4
* Tests cover:
5
5
* - Format validation
6
6
* - Monotonicity (single-threaded and concurrent)
···
33
33
34
34
describe('TID Format Validation', () => {
35
35
it('should validate correct TID format', () => {
36
-
const validTids = [
37
-
'3jzfcijpj2z2a',
38
-
'7777777777777',
39
-
'3zzzzzzzzzzzz',
40
-
];
36
+
const validTids = ['3jzfcijpj2z2a', '7777777777777', '3zzzzzzzzzzzz'];
41
37
42
38
for (const tid of validTids) {
43
39
assert.strictEqual(validateTid(tid), true, `Should validate: ${tid}`);
···
46
42
47
43
it('should reject invalid TID format', () => {
48
44
const invalidTids = [
49
-
'3jzfcijpj2z21', // Invalid character (1)
50
-
'0000000000000', // Invalid character (0)
51
-
'3jzfcijpj2z2aa', // Too long
52
-
'3jzfcijpj2z2', // Too short
53
-
'3jzf-cij-pj2z-2a', // Dashes not allowed
54
-
'zzzzzzzzzzzzz', // High bit violation
55
-
'kjzfcijpj2z2a', // High bit violation
45
+
'3jzfcijpj2z21', // Invalid character (1)
46
+
'0000000000000', // Invalid character (0)
47
+
'3jzfcijpj2z2aa', // Too long
48
+
'3jzfcijpj2z2', // Too short
49
+
'3jzf-cij-pj2z-2a', // Dashes not allowed
50
+
'zzzzzzzzzzzzz', // High bit violation
51
+
'kjzfcijpj2z2a', // High bit violation
56
52
];
57
53
58
54
for (const tid of invalidTids) {
···
71
67
});
72
68
73
69
it('should throw on ensureValidTid for invalid input', () => {
74
-
assert.throws(
75
-
() => ensureValidTid('invalid'),
76
-
InvalidTidError
77
-
);
70
+
assert.throws(() => ensureValidTid('invalid'), InvalidTidError);
78
71
});
79
72
});
80
73
···
106
99
});
107
100
108
101
it('should encode clock ID correctly', async () => {
109
-
const clock = new TidClock(
110
-
new FakeClock(1000000000000000),
111
-
new SilentTidLogger(),
112
-
{ clockId: 15 }
113
-
);
102
+
const clock = new TidClock(new FakeClock(1000000000000000), new SilentTidLogger(), {
103
+
clockId: 15,
104
+
});
114
105
const tid = await clock.next();
115
106
116
107
const clockId = decodeTidClockId(tid);
···
152
143
const clock = new TidClock(fakeClock, new SilentTidLogger());
153
144
154
145
const tid1 = await clock.next();
155
-
146
+
156
147
// Move clock backwards
157
148
fakeClock.set(999999000000000);
158
-
149
+
159
150
const tid2 = await clock.next();
160
151
161
152
// Should still be monotonic
···
167
158
const clock = new TidClock(fakeClock, new SilentTidLogger());
168
159
169
160
const tids: string[] = [];
170
-
161
+
171
162
// Generate some TIDs
172
163
for (let i = 0; i < 5; i++) {
173
164
tids.push(await clock.next());
···
239
230
it('should generate deterministic TIDs with same seed', async () => {
240
231
const seed = 1000000000000000;
241
232
242
-
const clock1 = new TidClock(new FakeClock(seed), new SilentTidLogger(), { clockId: 10 });
243
-
const clock2 = new TidClock(new FakeClock(seed), new SilentTidLogger(), { clockId: 10 });
233
+
const clock1 = new TidClock(new FakeClock(seed), new SilentTidLogger(), {
234
+
clockId: 10,
235
+
});
236
+
const clock2 = new TidClock(new FakeClock(seed), new SilentTidLogger(), {
237
+
clockId: 10,
238
+
});
244
239
245
240
const tids1: string[] = [];
246
241
const tids2: string[] = [];
···
261
256
new Date('2020-12-31T23:59:59Z'),
262
257
];
263
258
264
-
const clock1 = new TidClock(new FakeClock(0), new SilentTidLogger(), { clockId: 5 });
265
-
const clock2 = new TidClock(new FakeClock(0), new SilentTidLogger(), { clockId: 5 });
259
+
const clock1 = new TidClock(new FakeClock(0), new SilentTidLogger(), {
260
+
clockId: 5,
261
+
});
262
+
const clock2 = new TidClock(new FakeClock(0), new SilentTidLogger(), {
263
+
clockId: 5,
264
+
});
266
265
267
-
const tids1 = await Promise.all(dates.map(d => clock1.fromDate(d)));
268
-
const tids2 = await Promise.all(dates.map(d => clock2.fromDate(d)));
266
+
const tids1 = await Promise.all(dates.map((d) => clock1.fromDate(d)));
267
+
const tids2 = await Promise.all(dates.map((d) => clock2.fromDate(d)));
269
268
270
269
assert.deepStrictEqual(tids1, tids2);
271
270
});
···
288
287
});
289
288
290
289
it('should persist state to disk', async () => {
291
-
const clock = new TidClock(
292
-
new FakeClock(1000000000000000),
293
-
new SilentTidLogger(),
294
-
{ statePath }
295
-
);
290
+
const clock = new TidClock(new FakeClock(1000000000000000), new SilentTidLogger(), {
291
+
statePath,
292
+
});
296
293
297
294
await clock.next();
298
295
await clock.next();
···
308
305
it('should restore state from disk', async () => {
309
306
// Use a unique state file path for this test
310
307
const restoreStatePath = path.join(tempDir, 'tid-state-restore.json');
311
-
308
+
312
309
// Create first clock and generate some TIDs
313
-
const clock1 = new TidClock(
314
-
new FakeClock(1000000000000000),
315
-
new SilentTidLogger(),
316
-
{ statePath: restoreStatePath, clockId: 10 }
317
-
);
310
+
const clock1 = new TidClock(new FakeClock(1000000000000000), new SilentTidLogger(), {
311
+
statePath: restoreStatePath,
312
+
clockId: 10,
313
+
});
318
314
319
315
await clock1.next();
320
316
const tid2 = await clock1.next();
321
317
322
318
// Create new clock with same state file
323
-
const clock2 = new TidClock(
324
-
new FakeClock(1000000000000000),
325
-
new SilentTidLogger(),
326
-
{ statePath: restoreStatePath }
327
-
);
319
+
const clock2 = new TidClock(new FakeClock(1000000000000000), new SilentTidLogger(), {
320
+
statePath: restoreStatePath,
321
+
});
328
322
329
323
const tid3 = await clock2.next();
330
324
331
325
// Should continue from where clock1 left off
332
326
assert.strictEqual(compareTids(tid2, tid3), -1);
333
-
327
+
334
328
const state = clock2.getState();
335
329
assert.strictEqual(state.generatedCount, 3);
336
330
});
···
355
349
new Date('2025-01-01T00:00:00Z'),
356
350
];
357
351
358
-
const tids = await Promise.all(dates.map(d => clock.fromDate(d)));
352
+
const tids = await Promise.all(dates.map((d) => clock.fromDate(d)));
359
353
360
354
// Should still be monotonic despite out-of-order input
361
355
assert.strictEqual(areMonotonic(tids), true);
···
379
373
describe('TID Collision Detection', () => {
380
374
it('should detect duplicate TIDs (should never happen)', async () => {
381
375
const clock = new TidClock(new FakeClock(1000000000000000), new SilentTidLogger());
382
-
376
+
383
377
// Generate a TID
384
378
await clock.next();
385
-
379
+
386
380
// Try to force a duplicate by manually resetting state (testing collision detection)
387
381
// This simulates what would happen if there was a bug in generation
388
382
const state = clock.getState();
389
-
383
+
390
384
// Create a new clock with the exact same state
391
-
const clock2 = new TidClock(
392
-
new FakeClock(state.lastTimestampUs),
393
-
new SilentTidLogger(),
394
-
{
395
-
clockId: state.clockId,
396
-
initialState: {
397
-
lastTimestampUs: state.lastTimestampUs - 1, // Trick it into generating same timestamp
398
-
generatedCount: 0,
399
-
}
400
-
}
401
-
);
402
-
385
+
const clock2 = new TidClock(new FakeClock(state.lastTimestampUs), new SilentTidLogger(), {
386
+
clockId: state.clockId,
387
+
initialState: {
388
+
lastTimestampUs: state.lastTimestampUs - 1, // Trick it into generating same timestamp
389
+
generatedCount: 0,
390
+
},
391
+
});
392
+
403
393
// First TID from clock2 might be a duplicate
404
394
// But the clock should handle it via sequence increment
405
395
const tid = await clock2.next();
···
420
410
});
421
411
422
412
it('should sort TIDs correctly', () => {
423
-
const tids = [
424
-
'7777777777777',
425
-
'3jzfcijpj2z2a',
426
-
'3zzzzzzzzzzzz',
427
-
'3jzfcijpj2z2b',
428
-
];
413
+
const tids = ['7777777777777', '3jzfcijpj2z2a', '3zzzzzzzzzzzz', '3jzfcijpj2z2b'];
429
414
430
415
const sorted = [...tids].sort(compareTids);
431
-
416
+
432
417
assert.deepStrictEqual(sorted, [
433
418
'3jzfcijpj2z2a',
434
419
'3jzfcijpj2z2b',
+10
-9
src/types.ts
+10
-9
src/types.ts
···
1
1
import { AtpAgent as Agent } from '@atproto/api';
2
+
export type { AtprotoAgentInterface } from './lib/atproto/types.js';
2
3
3
4
/**
4
5
* Type alias for the ATProto Agent, used for clarity in the project.
···
37
38
export interface CommandLineArgs {
38
39
// Help
39
40
help?: boolean;
40
-
41
+
41
42
// Authentication
42
43
handle?: string;
43
44
password?: string;
44
-
45
+
45
46
// Input
46
47
input?: string;
47
48
'spotify-input'?: string;
48
-
49
+
49
50
// Mode
50
51
mode?: string;
51
-
52
+
52
53
// Batch configuration
53
54
'batch-size'?: string;
54
55
'batch-delay'?: string;
55
-
56
+
56
57
// Import options
57
58
reverse?: boolean;
58
59
yes?: boolean;
···
61
62
fresh?: boolean;
62
63
'clear-cache'?: boolean;
63
64
'clear-all-caches'?: boolean;
64
-
65
+
65
66
// Output
66
67
verbose?: boolean;
67
68
quiet?: boolean;
68
-
69
+
69
70
// Legacy flags (for backwards compatibility)
70
71
file?: string;
71
72
'spotify-file'?: string;
···
90
91
SCALING_FACTOR: number;
91
92
DEFAULT_BATCH_DELAY: number;
92
93
93
-
DEFAULT_BATCH_SIZE: number; // from rate limiter
94
-
MIN_BATCH_DELAY: number; // from rate limiter
94
+
DEFAULT_BATCH_SIZE: number; // from rate limiter
95
+
MIN_BATCH_DELAY: number; // from rate limiter
95
96
RECORDS_PER_DAY_LIMIT: number;
96
97
SAFETY_MARGIN: number;
97
98
AGGRESSIVE_SAFETY_MARGIN: number;
+19
-43
src/utils/helpers.ts
+19
-43
src/utils/helpers.ts
···
8
8
*/
9
9
function getUserLocale(): string {
10
10
// Try to get locale from environment variables
11
-
const envLang = process.env.LANG?.split('.')[0] ||
12
-
process.env.LC_ALL?.split('.')[0];
11
+
const envLang = process.env.LANG?.split('.')[0] || process.env.LC_ALL?.split('.')[0];
13
12
14
13
// Filter out invalid locales (like "C" or "POSIX")
15
14
if (envLang && envLang !== 'C' && envLang !== 'POSIX') {
···
40
39
export function formatDate(date: string | Date, includeTime: boolean = false): string {
41
40
const dateObj = typeof date === 'string' ? new Date(date) : date;
42
41
const locale = getUserLocale();
43
-
42
+
44
43
if (includeTime) {
45
44
return dateObj.toLocaleString(locale, {
46
45
year: 'numeric',
···
48
47
day: 'numeric',
49
48
hour: '2-digit',
50
49
minute: '2-digit',
51
-
timeZoneName: 'short'
50
+
timeZoneName: 'short',
52
51
});
53
52
} else {
54
53
return dateObj.toLocaleDateString(locale, {
55
54
year: 'numeric',
56
55
month: 'short',
57
-
day: 'numeric'
56
+
day: 'numeric',
58
57
});
59
58
}
60
59
}
···
76
75
const seconds = Math.floor(milliseconds / 1000);
77
76
const minutes = Math.floor(seconds / 60);
78
77
const hours = Math.floor(minutes / 60);
79
-
78
+
80
79
if (hours > 0) {
81
80
const mins = minutes % 60;
82
81
return `${hours}h ${mins}m`;
···
92
91
* Calculate optimal batch size based on total records and rate limits
93
92
* Uses a logarithmic scaling approach to balance throughput with API safety
94
93
*/
95
-
export function calculateOptimalBatchSize(totalRecords: number, batchDelay: number, config: Config): number {
94
+
export function calculateOptimalBatchSize(
95
+
totalRecords: number,
96
+
batchDelay: number,
97
+
config: Config,
98
+
): number {
96
99
const {
97
100
MIN_RECORDS_FOR_SCALING,
98
101
BASE_BATCH_SIZE,
99
102
MAX_BATCH_SIZE,
100
103
SCALING_FACTOR,
101
-
DEFAULT_BATCH_DELAY
104
+
DEFAULT_BATCH_DELAY,
102
105
} = config;
103
-
106
+
104
107
const delay = batchDelay || DEFAULT_BATCH_DELAY;
105
-
108
+
106
109
// For very small datasets, use minimal batches
107
110
if (totalRecords <= 50) {
108
111
return 3;
109
112
}
110
-
113
+
111
114
// For small to medium datasets, use conservative batching
112
115
if (totalRecords <= MIN_RECORDS_FOR_SCALING) {
113
116
return BASE_BATCH_SIZE;
114
117
}
115
-
118
+
116
119
// Logarithmic scaling
117
120
const logScale = Math.log2(totalRecords / MIN_RECORDS_FOR_SCALING);
118
-
const calculatedSize = Math.floor(BASE_BATCH_SIZE + (logScale * SCALING_FACTOR));
119
-
121
+
const calculatedSize = Math.floor(BASE_BATCH_SIZE + logScale * SCALING_FACTOR);
122
+
120
123
// Apply maximum cap
121
124
let optimalSize = Math.min(calculatedSize, MAX_BATCH_SIZE);
122
-
125
+
123
126
// Adjust based on batch delay
124
127
if (delay < 1500 && optimalSize > 15) {
125
128
optimalSize = Math.floor(optimalSize * 0.75);
126
129
}
127
-
130
+
128
131
// Ensure batch size is at least 3
129
132
return Math.max(3, optimalSize);
130
133
}
131
-
132
-
/**
133
-
* Logs rate limiting and batching information to the console.
134
-
* Note: This function cannot import log from logger.ts to avoid circular dependencies,
135
-
* so it uses console.log directly. The CLI controls the log level, so this output
136
-
* is appropriately controlled.
137
-
*/
138
-
export function showRateLimitInfo(
139
-
totalRecords: number,
140
-
batchSize: number,
141
-
batchDelay: number,
142
-
estimatedDays: number,
143
-
dailyLimit: number
144
-
): void {
145
-
console.log('\n📊 Rate Limiting Information:');
146
-
console.log(` Total records: ${totalRecords.toLocaleString()}`);
147
-
console.log(` Daily limit: ${dailyLimit.toLocaleString()} records/day`);
148
-
console.log(` Estimated duration: ${estimatedDays} day${estimatedDays > 1 ? 's' : ''}`);
149
-
console.log(` Batch size: ${batchSize} records`);
150
-
console.log(` Batch delay: ${(batchDelay / 1000).toFixed(1)}s`);
151
-
152
-
if (estimatedDays > 1) {
153
-
console.log('\n The import will automatically pause between days.');
154
-
console.log(' You can safely close and restart the importer - it will resume from where it left off.');
155
-
}
156
-
console.log('');
157
-
}
+24
-44
src/utils/import-state.ts
+24
-44
src/utils/import-state.ts
···
2
2
import path from 'path';
3
3
import os from 'os';
4
4
import crypto from 'crypto';
5
-
import type { PlayRecord } from '../types.js';
6
5
import { log } from './logger.js';
7
6
8
7
/**
···
28
27
*/
29
28
export function getStateFilePath(inputFile: string, mode: string): string {
30
29
const stateDir = path.join(os.homedir(), '.malachite', 'state');
31
-
30
+
32
31
// Create state directory if it doesn't exist
33
32
if (!fs.existsSync(stateDir)) {
34
33
fs.mkdirSync(stateDir, { recursive: true });
35
34
}
36
-
35
+
37
36
// Create a unique filename based on input file and mode
38
37
const hash = crypto
39
38
.createHash('md5')
40
39
.update(inputFile + mode)
41
40
.digest('hex')
42
41
.substring(0, 8);
43
-
42
+
44
43
return path.join(stateDir, `import-${hash}.json`);
45
44
}
46
45
···
51
50
if (!fs.existsSync(filePath)) {
52
51
return '';
53
52
}
54
-
53
+
55
54
const stats = fs.statSync(filePath);
56
55
if (stats.isDirectory()) {
57
56
// For directories (like Spotify exports), hash the directory name and modification time
···
60
59
.update(filePath + stats.mtime.toISOString())
61
60
.digest('hex');
62
61
}
63
-
62
+
64
63
// For files, use file size and modification time for quick comparison
65
64
return crypto
66
65
.createHash('md5')
···
73
72
*/
74
73
export function loadImportState(inputFile: string, mode: string): ImportState | null {
75
74
const stateFile = getStateFilePath(inputFile, mode);
76
-
75
+
77
76
if (!fs.existsSync(stateFile)) {
78
77
return null;
79
78
}
80
-
79
+
81
80
try {
82
81
const data = fs.readFileSync(stateFile, 'utf-8');
83
82
const state = JSON.parse(data) as ImportState;
84
-
83
+
85
84
// Check if input file has changed
86
85
const currentHash = calculateFileHash(inputFile);
87
86
if (state.inputFileHash !== currentHash) {
88
87
log.warn('Input file has changed since last import - starting fresh');
89
88
return null;
90
89
}
91
-
90
+
92
91
return state;
93
92
} catch (error) {
94
93
log.warn('Failed to load state file - starting fresh');
···
101
100
*/
102
101
export function saveImportState(state: ImportState): void {
103
102
const stateFile = getStateFilePath(state.inputFile, state.mode);
104
-
103
+
105
104
try {
106
105
state.lastUpdatedAt = new Date().toISOString();
107
106
fs.writeFileSync(stateFile, JSON.stringify(state, null, 2), 'utf-8');
···
116
115
export function createImportState(
117
116
inputFile: string,
118
117
mode: 'lastfm' | 'spotify' | 'combined' | 'sync',
119
-
totalRecords: number
118
+
totalRecords: number,
120
119
): ImportState {
121
120
return {
122
121
version: '1.0',
···
141
140
state: ImportState,
142
141
batchIndex: number,
143
142
successCount: number,
144
-
errorCount: number
143
+
errorCount: number,
145
144
): void {
146
145
state.processedRecords += successCount + errorCount;
147
146
state.successfulRecords += successCount;
148
147
state.failedRecords += errorCount;
149
-
148
+
150
149
if (successCount > 0) {
151
150
state.lastSuccessfulIndex = batchIndex;
152
151
}
153
-
152
+
154
153
saveImportState(state);
155
154
}
156
155
···
167
166
*/
168
167
export function clearImportState(inputFile: string, mode: string): void {
169
168
const stateFile = getStateFilePath(inputFile, mode);
170
-
169
+
171
170
if (fs.existsSync(stateFile)) {
172
171
fs.unlinkSync(stateFile);
173
172
}
···
180
179
const elapsed = Date.now() - new Date(state.startedAt).getTime();
181
180
const elapsedHours = Math.floor(elapsed / (1000 * 60 * 60));
182
181
const elapsedMinutes = Math.floor((elapsed % (1000 * 60 * 60)) / (1000 * 60));
183
-
182
+
184
183
const remaining = state.totalRecords - state.processedRecords;
185
184
const progress = ((state.processedRecords / state.totalRecords) * 100).toFixed(1);
186
-
185
+
187
186
log.section('Resuming Previous Import');
188
187
log.info(`Started: ${new Date(state.startedAt).toLocaleString()}`);
189
-
log.info(`Progress: ${state.processedRecords.toLocaleString()}/${state.totalRecords.toLocaleString()} (${progress}%)`);
188
+
log.info(
189
+
`Progress: ${state.processedRecords.toLocaleString()}/${state.totalRecords.toLocaleString()} (${progress}%)`,
190
+
);
190
191
log.info(`Successful: ${state.successfulRecords.toLocaleString()}`);
191
-
192
+
192
193
if (state.failedRecords > 0) {
193
194
log.warn(`Failed: ${state.failedRecords.toLocaleString()}`);
194
195
}
195
-
196
+
196
197
log.info(`Remaining: ${remaining.toLocaleString()} records`);
197
-
198
+
198
199
if (elapsedHours > 0) {
199
200
log.info(`Time elapsed: ${elapsedHours}h ${elapsedMinutes}m`);
200
201
} else if (elapsedMinutes > 0) {
201
202
log.info(`Time elapsed: ${elapsedMinutes}m`);
202
203
}
203
-
204
-
log.blank();
205
-
}
206
204
207
-
/**
208
-
* Filter records to skip already processed ones
209
-
*/
210
-
export function filterUnprocessedRecords(
211
-
records: PlayRecord[],
212
-
state: ImportState
213
-
): PlayRecord[] {
214
-
if (state.lastSuccessfulIndex < 0) {
215
-
return records;
216
-
}
217
-
218
-
// Skip records up to and including the last successful index
219
-
const startIndex = state.lastSuccessfulIndex + 1;
220
-
221
-
if (startIndex >= records.length) {
222
-
return [];
223
-
}
224
-
225
-
return records.slice(startIndex);
205
+
log.blank();
226
206
}
227
207
228
208
/**
-41
src/utils/killswitch.ts
-41
src/utils/killswitch.ts
···
1
-
let cancelled = false;
2
-
let sigintHandlerRegistered = false;
3
-
4
-
/**
5
-
* Register the SIGINT handler (should be called early in the application)
6
-
*/
7
-
export function registerKillswitch(): void {
8
-
if (sigintHandlerRegistered) {
9
-
return;
10
-
}
11
-
12
-
// Flip the killswitch when the user hits CTRL-C
13
-
process.on('SIGINT', () => {
14
-
if (cancelled) {
15
-
// If already cancelled and user presses Ctrl+C again, force exit
16
-
console.log('\n\nForce quit detected. Exiting immediately...');
17
-
process.exit(1);
18
-
}
19
-
20
-
console.log('\n\n⚠️ Ctrl+C detected — stopping after current batch completes...');
21
-
console.log('Press Ctrl+C again to force quit immediately.\n');
22
-
cancelled = true;
23
-
});
24
-
25
-
sigintHandlerRegistered = true;
26
-
}
27
-
28
-
/**
29
-
* Manually cancel the import if needed.
30
-
*/
31
-
export function cancelImport() {
32
-
cancelled = true;
33
-
}
34
-
35
-
/**
36
-
* Check whether the import should stop.
37
-
* Call this inside loops, batch processors, etc.
38
-
*/
39
-
export function isImportCancelled(): boolean {
40
-
return cancelled;
41
-
}
-161
src/utils/rate-limiter.ts
-161
src/utils/rate-limiter.ts
···
1
-
import type { Config } from '../types.js';
2
-
3
-
/**
4
-
* Calculate rate-limited batch parameters
5
-
* Ensures we don't exceed daily limits while maintaining efficiency
6
-
*/
7
-
export function calculateRateLimitedBatches(
8
-
totalRecords: number,
9
-
config: Config
10
-
): {
11
-
batchSize: number;
12
-
batchDelay: number;
13
-
estimatedDays: number;
14
-
recordsPerDay: number;
15
-
needsRateLimiting: boolean;
16
-
} {
17
-
const dailyLimit = Math.floor(config.RECORDS_PER_DAY_LIMIT * config.SAFETY_MARGIN);
18
-
19
-
// Check if we need rate limiting
20
-
const needsRateLimiting = totalRecords > dailyLimit;
21
-
22
-
if (!needsRateLimiting) {
23
-
// Can import everything in one go
24
-
return {
25
-
batchSize: config.DEFAULT_BATCH_SIZE,
26
-
batchDelay: config.DEFAULT_BATCH_DELAY,
27
-
estimatedDays: 1,
28
-
recordsPerDay: totalRecords,
29
-
needsRateLimiting: false,
30
-
};
31
-
}
32
-
33
-
// Calculate how many days needed
34
-
const estimatedDays = Math.ceil(totalRecords / dailyLimit);
35
-
const recordsPerDay = Math.floor(totalRecords / estimatedDays);
36
-
37
-
// Calculate batch parameters
38
-
// We want to spread records evenly throughout the day
39
-
const minutesPerDay = 24 * 60;
40
-
const batchesPerDay = Math.ceil(recordsPerDay / config.DEFAULT_BATCH_SIZE);
41
-
const delayBetweenBatches = Math.floor((minutesPerDay * 60 * 1000) / batchesPerDay);
42
-
43
-
// Ensure batch delay is at least minimum
44
-
const batchDelay = Math.max(delayBetweenBatches, config.MIN_BATCH_DELAY);
45
-
46
-
// Adjust batch size if needed to hit the target
47
-
const adjustedBatchSize = Math.min(
48
-
Math.ceil(recordsPerDay / Math.floor((minutesPerDay * 60 * 1000) / batchDelay)),
49
-
config.MAX_BATCH_SIZE
50
-
);
51
-
52
-
return {
53
-
batchSize: adjustedBatchSize,
54
-
batchDelay,
55
-
estimatedDays,
56
-
recordsPerDay,
57
-
needsRateLimiting: true,
58
-
};
59
-
}
60
-
61
-
/**
62
-
* Calculate daily batches and pause times
63
-
*/
64
-
export function calculateDailySchedule(
65
-
totalRecords: number,
66
-
batchSize: number,
67
-
batchDelay: number,
68
-
recordsPerDay: number
69
-
) {
70
-
const schedule = [];
71
-
72
-
// How many batches fit into a 24h window using the actual delay?
73
-
const batchesPerDay = Math.floor((24 * 60 * 60 * 1000) / batchDelay);
74
-
75
-
// Max records we could process in one day given the spacing
76
-
const maxRecordsPerDay = batchesPerDay * batchSize;
77
-
78
-
// Respect the external rate limit (recordsPerDay)
79
-
const dailyCap = Math.min(maxRecordsPerDay, recordsPerDay);
80
-
81
-
let processed = 0;
82
-
let day = 1;
83
-
84
-
while (processed < totalRecords) {
85
-
const recordsStart = processed;
86
-
const dailyCount = Math.min(dailyCap, totalRecords - processed);
87
-
const recordsEnd = recordsStart + dailyCount;
88
-
const isLastDay = recordsEnd >= totalRecords;
89
-
90
-
schedule.push({
91
-
day,
92
-
recordsStart,
93
-
recordsEnd,
94
-
recordsCount: dailyCount,
95
-
pauseAfter: !isLastDay,
96
-
pauseDuration: isLastDay ? 0 : 24 * 60 * 60 * 1000
97
-
});
98
-
99
-
processed = recordsEnd;
100
-
day++;
101
-
}
102
-
103
-
return schedule;
104
-
}
105
-
106
-
107
-
/**
108
-
* Format time duration in human-readable format
109
-
*/
110
-
export function formatTimeRemaining(ms: number): string {
111
-
const days = Math.floor(ms / (24 * 60 * 60 * 1000));
112
-
const hours = Math.floor((ms % (24 * 60 * 60 * 1000)) / (60 * 60 * 1000));
113
-
const minutes = Math.floor((ms % (60 * 60 * 1000)) / (60 * 1000));
114
-
115
-
if (days > 0) {
116
-
return `${days}d ${hours}h ${minutes}m`;
117
-
} else if (hours > 0) {
118
-
return `${hours}h ${minutes}m`;
119
-
} else if (minutes > 0) {
120
-
return `${minutes}m`;
121
-
} else {
122
-
return '< 1m';
123
-
}
124
-
}
125
-
126
-
/**
127
-
* Display rate limit warning
128
-
*/
129
-
export function displayRateLimitWarning(): void {
130
-
console.log('');
131
-
console.log('⚠️ IMPORTANT: Rate Limits');
132
-
console.log(' Exceeding 10K records/day can rate limit your ENTIRE PDS.');
133
-
console.log(' This affects ALL users on your PDS, not just your account.');
134
-
console.log(' Import automatically limits to 10K records/day with pauses.');
135
-
console.log(' See: https://docs.bsky.app/blog/rate-limits-pds-v3');
136
-
console.log('');
137
-
}
138
-
139
-
/**
140
-
* Display rate limiting info
141
-
*/
142
-
export function displayRateLimitInfo(
143
-
totalRecords: number,
144
-
batchSize: number,
145
-
batchDelay: number,
146
-
estimatedDays: number,
147
-
recordsPerDay: number
148
-
): void {
149
-
console.log('\n📊 Rate Limiting Information:');
150
-
console.log(` Total records: ${totalRecords.toLocaleString()}`);
151
-
console.log(` Daily limit: ${recordsPerDay.toLocaleString()} records/day`);
152
-
console.log(` Estimated duration: ${estimatedDays} day${estimatedDays > 1 ? 's' : ''}`);
153
-
console.log(` Batch size: ${batchSize} records`);
154
-
console.log(` Batch delay: ${(batchDelay / 1000).toFixed(1)}s`);
155
-
156
-
if (estimatedDays > 1) {
157
-
console.log('\n The import will automatically pause between days.');
158
-
console.log(' You can safely close and restart the importer - it will resume from where it left off.');
159
-
}
160
-
console.log('');
161
-
}
+33
-21
src/utils/teal-cache.ts
+33
-21
src/utils/teal-cache.ts
···
1
-
import { existsSync, readFileSync, writeFileSync, mkdirSync, readdirSync, unlinkSync } from 'node:fs';
1
+
import {
2
+
existsSync,
3
+
readFileSync,
4
+
writeFileSync,
5
+
mkdirSync,
6
+
readdirSync,
7
+
unlinkSync,
8
+
} from 'node:fs';
9
+
import { homedir } from 'node:os';
2
10
import { join } from 'node:path';
3
-
import { homedir } from 'node:os';
4
11
5
12
/**
6
13
* Cache configuration
···
42
49
*/
43
50
export function isCacheValid(did: string): boolean {
44
51
const cachePath = getCachePath(did);
45
-
52
+
46
53
if (!existsSync(cachePath)) {
47
54
return false;
48
55
}
49
-
56
+
50
57
try {
51
58
const data = readFileSync(cachePath, 'utf-8');
52
59
const cache: CacheFile = JSON.parse(data);
53
-
60
+
54
61
// Check version
55
62
if (cache.version !== CACHE_VERSION) {
56
63
return false;
57
64
}
58
-
65
+
59
66
// Check DID match
60
67
if (cache.did !== did) {
61
68
return false;
62
69
}
63
-
70
+
64
71
// Check age
65
72
const ageHours = (Date.now() - cache.timestamp) / (1000 * 60 * 60);
66
73
if (ageHours > CACHE_TTL_HOURS) {
67
74
return false;
68
75
}
69
-
76
+
70
77
return true;
71
78
} catch {
72
79
// Invalid cache file
···
78
85
* Load cached records for a given DID
79
86
* Returns null if cache doesn't exist or is invalid
80
87
*/
81
-
export function loadCache(did: string): Map<string, { uri: string; cid: string; value: any }> | null {
88
+
export function loadCache(
89
+
did: string,
90
+
): Map<string, { uri: string; cid: string; value: any }> | null {
82
91
const cachePath = getCachePath(did);
83
-
92
+
84
93
if (!isCacheValid(did)) {
85
94
return null;
86
95
}
87
-
96
+
88
97
try {
89
98
const data = readFileSync(cachePath, 'utf-8');
90
99
const cache: CacheFile = JSON.parse(data);
91
-
100
+
92
101
// Convert array back to Map
93
102
return new Map(cache.records);
94
103
} catch {
···
99
108
/**
100
109
* Save records to cache for a given DID
101
110
*/
102
-
export function saveCache(did: string, records: Map<string, { uri: string; cid: string; value: any }>): void {
111
+
export function saveCache(
112
+
did: string,
113
+
records: Map<string, { uri: string; cid: string; value: any }>,
114
+
): void {
103
115
ensureCacheDir();
104
-
116
+
105
117
const cache: CacheFile = {
106
118
version: CACHE_VERSION,
107
119
did,
108
120
timestamp: Date.now(),
109
121
records: Array.from(records.entries()),
110
122
};
111
-
123
+
112
124
const cachePath = getCachePath(did);
113
125
writeFileSync(cachePath, JSON.stringify(cache), 'utf-8');
114
126
}
···
118
130
*/
119
131
export function getCacheInfo(did: string): { age?: number; records?: number } {
120
132
const cachePath = getCachePath(did);
121
-
133
+
122
134
if (!existsSync(cachePath)) {
123
135
return {};
124
136
}
125
-
137
+
126
138
try {
127
139
const data = readFileSync(cachePath, 'utf-8');
128
140
const cache: CacheFile = JSON.parse(data);
129
-
141
+
130
142
const ageHours = (Date.now() - cache.timestamp) / (1000 * 60 * 60);
131
-
143
+
132
144
return {
133
145
age: ageHours,
134
146
records: cache.records.length,
···
143
155
*/
144
156
export function clearCache(did: string): void {
145
157
const cachePath = getCachePath(did);
146
-
158
+
147
159
if (existsSync(cachePath)) {
148
160
unlinkSync(cachePath);
149
161
}
···
156
168
if (!existsSync(CACHE_DIR)) {
157
169
return;
158
170
}
159
-
171
+
160
172
const files = readdirSync(CACHE_DIR);
161
173
for (const file of files) {
162
174
if (file.endsWith('.json')) {
+5
-10
src/utils/tid-audit.ts
+5
-10
src/utils/tid-audit.ts
···
1
1
/**
2
2
* TID Audit and Reporting Tools
3
-
*
3
+
*
4
4
* Utilities for auditing TID generation and producing reports
5
5
*/
6
6
···
111
111
lines.push('SUMMARY');
112
112
lines.push('-'.repeat(80));
113
113
lines.push(`Total TIDs: ${report.totalTids.toLocaleString()}`);
114
-
lines.push(`Valid: ${report.validTids.toLocaleString()} (${((report.validTids / report.totalTids) * 100).toFixed(1)}%)`);
114
+
lines.push(
115
+
`Valid: ${report.validTids.toLocaleString()} (${((report.validTids / report.totalTids) * 100).toFixed(1)}%)`,
116
+
);
115
117
lines.push(`Invalid: ${report.invalidTids.toLocaleString()}`);
116
118
lines.push(`Duplicates: ${report.duplicates.toLocaleString()}`);
117
119
lines.push(`Monotonic: ${report.monotonic ? '✓ YES' : '✗ NO'}`);
···
150
152
}
151
153
152
154
// Invalid TIDs
153
-
const invalidEntries = report.entries.filter(e => !e.valid || e.errors.length > 0);
155
+
const invalidEntries = report.entries.filter((e) => !e.valid || e.errors.length > 0);
154
156
if (invalidEntries.length > 0) {
155
157
lines.push('INVALID/PROBLEM TIDs');
156
158
lines.push('-'.repeat(80));
···
170
172
171
173
return lines.join('\n');
172
174
}
173
-
174
-
/**
175
-
* Format audit report as JSON
176
-
*/
177
-
export function formatAuditReportJson(report: TidAuditReport): string {
178
-
return JSON.stringify(report, null, 2);
179
-
}
+19
-64
src/utils/tid-clock.ts
+19
-64
src/utils/tid-clock.ts
···
1
1
/**
2
2
* TID (Timestamp Identifier) Clock for ATProto
3
-
*
3
+
*
4
4
* Implements spec-compliant, monotonic TID generation with:
5
5
* - AT-Protocol format validation (13 chars, base32 alphabet)
6
6
* - Strict monotonicity guarantees (even under clock drift)
···
8
8
* - Deterministic mode for dry-runs
9
9
* - Full observability (structured JSON logging)
10
10
* - Collision resistance
11
-
*
11
+
*
12
12
* Based on AT-Protocol spec: https://atproto.com/specs/tid
13
13
* Reference implementation: @atproto/common-web
14
14
*/
···
25
25
* TID validation error
26
26
*/
27
27
export class InvalidTidError extends Error {
28
-
constructor(message: string, public tid?: string) {
28
+
constructor(
29
+
message: string,
30
+
public tid?: string,
31
+
) {
29
32
super(message);
30
33
this.name = 'InvalidTidError';
31
34
}
32
35
}
33
36
34
37
/**
35
-
* TID generation modes
36
-
*/
37
-
export enum TidMode {
38
-
PRODUCTION = 'production', // Real wall-clock time
39
-
DRY_RUN = 'dry-run', // Deterministic with fixed seed
40
-
REPLAY = 'replay', // Replay from logged state
41
-
}
42
-
43
-
/**
44
38
* Clock source abstraction for testability
45
39
*/
46
40
export interface ClockSource {
···
61
55
*/
62
56
export class FakeClock implements ClockSource {
63
57
constructor(private timestamp: number) {}
64
-
58
+
65
59
now(): number {
66
60
return this.timestamp;
67
61
}
68
-
62
+
69
63
advance(microseconds: number): void {
70
64
this.timestamp += microseconds;
71
65
}
72
-
66
+
73
67
set(microseconds: number): void {
74
68
this.timestamp = microseconds;
75
69
}
···
79
73
* TID generator state (for persistence and logging)
80
74
*/
81
75
export interface TidState {
82
-
lastTimestampUs: number; // Last generated timestamp in microseconds
83
-
clockId: number; // Clock identifier (0-31 for this implementation)
84
-
generatedCount: number; // Total TIDs generated
76
+
lastTimestampUs: number; // Last generated timestamp in microseconds
77
+
clockId: number; // Clock identifier (0-31 for this implementation)
78
+
generatedCount: number; // Total TIDs generated
85
79
}
86
80
87
81
/**
···
91
85
tid: string;
92
86
timestampUs: number;
93
87
clockId: number;
94
-
generatedAt: string; // ISO8601 with microseconds
88
+
generatedAt: string; // ISO8601 with microseconds
95
89
validated: boolean;
96
90
context?: string;
97
91
}
···
106
100
}
107
101
108
102
/**
109
-
* Default console-based logger
110
-
*/
111
-
export class ConsoleTidLogger implements TidLogger {
112
-
logGenerated(metadata: TidMetadata, opId: string): void {
113
-
const entry = {
114
-
level: 'INFO',
115
-
op_id: opId,
116
-
ts: metadata.generatedAt,
117
-
event: 'tid.generated',
118
-
tid: metadata.tid,
119
-
clock_id: metadata.clockId,
120
-
wall_ts_us: metadata.timestampUs,
121
-
generator: 'tid-clock-v1',
122
-
context: metadata.context || 'unknown',
123
-
validated: metadata.validated,
124
-
};
125
-
console.log(JSON.stringify(entry));
126
-
}
127
-
128
-
logWarning(message: string, details?: any): void {
129
-
console.warn(JSON.stringify({
130
-
level: 'WARN',
131
-
event: 'tid.warning',
132
-
message,
133
-
...details,
134
-
}));
135
-
}
136
-
137
-
logError(message: string, details?: any): void {
138
-
console.error(JSON.stringify({
139
-
level: 'ERROR',
140
-
event: 'tid.error',
141
-
message,
142
-
...details,
143
-
}));
144
-
}
145
-
}
146
-
147
-
/**
148
103
* Silent logger (for production when structured logging is handled elsewhere)
149
104
*/
150
105
export class SilentTidLogger implements TidLogger {
···
170
125
statePath?: string;
171
126
clockId?: number;
172
127
initialState?: Partial<TidState>;
173
-
} = {}
128
+
} = {},
174
129
) {
175
130
this.clock = clock;
176
131
this.logger = logger;
···
201
156
202
157
/**
203
158
* Generate next TID with monotonicity guarantees
204
-
*
159
+
*
205
160
* Per AT-Protocol spec and reference implementation:
206
161
* - Use max(currentTime, lastTimestamp) to handle clock drift
207
162
* - If same as last timestamp, increment the timestamp itself (acts as sequence)
···
210
165
async next(context?: string): Promise<string> {
211
166
return this.withMutex(async () => {
212
167
const currentTime = this.clock.now();
213
-
168
+
214
169
// Take max of current time and last timestamp (handles backwards clock drift)
215
170
let timestamp = Math.max(currentTime, this.state.lastTimestampUs);
216
-
171
+
217
172
// If we're at the same timestamp, increment by 1 microsecond
218
173
// This acts as our sequence counter while maintaining monotonicity
219
174
if (timestamp === this.state.lastTimestampUs) {
···
275
230
*/
276
231
async fromDate(date: Date, context?: string): Promise<string> {
277
232
const timestamp = date.getTime() * 1000; // Convert ms to µs
278
-
233
+
279
234
return this.withMutex(async () => {
280
235
// Ensure monotonicity: use max of input timestamp and last generated
281
236
let finalTimestamp = Math.max(timestamp, this.state.lastTimestampUs);
282
-
237
+
283
238
// If we're at the same timestamp, increment by 1 microsecond
284
239
if (finalTimestamp === this.state.lastTimestampUs) {
285
240
finalTimestamp = this.state.lastTimestampUs + 1;
+21
-16
src/utils/tid.ts
+21
-16
src/utils/tid.ts
···
1
1
/**
2
2
* TID (Timestamp Identifier) generation for ATProto
3
-
*
3
+
*
4
4
* This module provides a high-level API for TID generation.
5
5
* For the full implementation with monotonicity guarantees, see tid-clock.ts
6
-
*
6
+
*
7
7
* Based on: https://atproto.com/specs/tid
8
8
*/
9
9
···
16
16
* Initialize the global TID clock
17
17
* Should be called once at application startup
18
18
*/
19
-
export function initTidClock(options: {
20
-
mode?: 'production' | 'dry-run';
21
-
statePath?: string;
22
-
seed?: number;
23
-
clockId?: number;
24
-
} = {}): void {
19
+
export function initTidClock(
20
+
options: {
21
+
mode?: 'production' | 'dry-run';
22
+
statePath?: string;
23
+
seed?: number;
24
+
clockId?: number;
25
+
} = {},
26
+
): void {
25
27
const { mode = 'production', statePath, seed, clockId } = options;
26
28
27
29
if (mode === 'dry-run' && seed !== undefined) {
28
30
// Deterministic clock for dry-run with fixed clockId for reproducibility
29
-
globalClock = new TidClock(new FakeClock(seed), undefined, {
31
+
globalClock = new TidClock(new FakeClock(seed), undefined, {
30
32
statePath,
31
-
clockId: clockId ?? 0 // Use fixed clockId for deterministic TIDs
33
+
clockId: clockId ?? 0, // Use fixed clockId for deterministic TIDs
32
34
});
33
35
} else {
34
36
// Production real-time clock
35
-
globalClock = new TidClock(new RealClock(), undefined, { statePath, clockId });
37
+
globalClock = new TidClock(new RealClock(), undefined, {
38
+
statePath,
39
+
clockId,
40
+
});
36
41
}
37
42
}
38
43
···
49
54
50
55
/**
51
56
* Generate a TID from a Date object with monotonicity guarantees
52
-
*
57
+
*
53
58
* This is the recommended way to generate TIDs for historical records.
54
59
* TIDs are guaranteed to be:
55
60
* - Spec-compliant (13 chars, valid base32)
56
61
* - Strictly monotonic (even if dates are out of order)
57
62
* - Collision-free (duplicate detection)
58
-
*
63
+
*
59
64
* @param date - The date to generate a TID from
60
65
* @param context - Optional context for logging (e.g., "inject:playlist")
61
66
* @returns A valid 13-character TID string
···
67
72
68
73
/**
69
74
* Generate a TID from an ISO 8601 timestamp string
70
-
*
75
+
*
71
76
* @param isoString - ISO 8601 formatted datetime string
72
77
* @param context - Optional context for logging
73
78
* @returns A valid 13-character TID string
···
78
83
79
84
/**
80
85
* Generate next TID using current time
81
-
*
86
+
*
82
87
* @param context - Optional context for logging
83
88
* @returns A valid 13-character TID string
84
89
*/
···
89
94
90
95
/**
91
96
* Validate a TID string
92
-
*
97
+
*
93
98
* @param tid - The TID to validate
94
99
* @returns true if valid, false otherwise
95
100
*/
+32
-48
src/utils/ui.ts
+32
-48
src/utils/ui.ts
···
16
16
currentSpinner = ora({
17
17
text,
18
18
color: 'cyan',
19
-
spinner: 'dots'
19
+
spinner: 'dots',
20
20
}).start();
21
21
return currentSpinner;
22
22
}
···
74
74
* Create a progress bar for batch processing
75
75
*/
76
76
export function createProgressBar(total: number, title: string) {
77
-
const bar = new cliProgress.SingleBar({
78
-
format: `${chalk.cyan(title)} ${chalk.cyan('│')} {bar} ${chalk.cyan('│')} {percentage}% ${chalk.cyan('│')} {value}/{total} records ${chalk.cyan('│')} {speed} rec/s ${chalk.cyan('│')} ETA: {eta_formatted}`,
79
-
barCompleteChar: '\u2588',
80
-
barIncompleteChar: '\u2591',
81
-
hideCursor: true,
82
-
clearOnComplete: false,
83
-
stopOnComplete: true,
84
-
formatValue: (v: number, options: any, type: string) => {
85
-
switch (type) {
86
-
case 'speed':
87
-
return v.toFixed(1);
88
-
default:
89
-
return cliProgress.Format.ValueFormat(v, options, type);
90
-
}
91
-
}
92
-
}, cliProgress.Presets.shades_classic);
77
+
const bar = new cliProgress.SingleBar(
78
+
{
79
+
format: `${chalk.cyan(title)} ${chalk.cyan('│')} {bar} ${chalk.cyan('│')} {percentage}% ${chalk.cyan('│')} {value}/{total} records ${chalk.cyan('│')} {speed} rec/s ${chalk.cyan('│')} ETA: {eta_formatted}`,
80
+
barCompleteChar: '\u2588',
81
+
barIncompleteChar: '\u2591',
82
+
hideCursor: true,
83
+
clearOnComplete: false,
84
+
stopOnComplete: true,
85
+
formatValue: (v: number, options: any, type: string) => {
86
+
switch (type) {
87
+
case 'speed':
88
+
return v.toFixed(1);
89
+
default:
90
+
return cliProgress.Format.ValueFormat(v, options, type);
91
+
}
92
+
},
93
+
},
94
+
cliProgress.Presets.shades_classic,
95
+
);
93
96
94
97
bar.start(total, 0, { speed: 0 });
95
98
return bar;
···
149
152
* Format a box around text
150
153
*/
151
154
export function box(lines: string[]): void {
152
-
const maxLength = Math.max(...lines.map(l => l.length));
155
+
const maxLength = Math.max(...lines.map((l) => l.length));
153
156
const border = '─'.repeat(maxLength + 4);
154
-
157
+
155
158
console.log(chalk.cyan('┌' + border + '┐'));
156
-
lines.forEach(line => {
159
+
lines.forEach((line) => {
157
160
const padding = ' '.repeat(maxLength - line.length);
158
161
console.log(chalk.cyan('│ ') + line + padding + chalk.cyan(' │'));
159
162
});
···
170
173
}
171
174
172
175
/**
173
-
* Format duration in a nice way
174
-
*/
175
-
export function formatDuration(ms: number): string {
176
-
const seconds = Math.floor(ms / 1000);
177
-
const minutes = Math.floor(seconds / 60);
178
-
const hours = Math.floor(minutes / 60);
179
-
180
-
if (hours > 0) {
181
-
const mins = minutes % 60;
182
-
return `${hours}h ${mins}m`;
183
-
} else if (minutes > 0) {
184
-
const secs = seconds % 60;
185
-
return `${minutes}m ${secs}s`;
186
-
} else {
187
-
return `${seconds}s`;
188
-
}
189
-
}
190
-
191
-
/**
192
176
* Create a status line that updates in place
193
177
*/
194
178
export class StatusLine {
195
179
private lastLength = 0;
196
-
180
+
197
181
update(text: string): void {
198
182
// Clear previous line
199
183
process.stdout.write('\r' + ' '.repeat(this.lastLength) + '\r');
200
-
184
+
201
185
// Write new text
202
186
process.stdout.write(text);
203
187
this.lastLength = text.length;
204
188
}
205
-
189
+
206
190
clear(): void {
207
191
process.stdout.write('\r' + ' '.repeat(this.lastLength) + '\r');
208
192
this.lastLength = 0;
209
193
}
210
-
194
+
211
195
finish(text?: string): void {
212
196
if (text) {
213
197
this.clear();
···
223
207
* Create a table
224
208
*/
225
209
export function table(headers: string[], rows: string[][]): void {
226
-
const colWidths = headers.map((h, i) =>
227
-
Math.max(h.length, ...rows.map(r => r[i]?.length || 0))
210
+
const colWidths = headers.map((h, i) =>
211
+
Math.max(h.length, ...rows.map((r) => r[i]?.length || 0)),
228
212
);
229
-
213
+
230
214
// Header
231
215
const headerRow = headers.map((h, i) => h.padEnd(colWidths[i])).join(' │ ');
232
216
console.log(chalk.cyan(headerRow));
233
217
console.log(chalk.dim('─'.repeat(headerRow.length)));
234
-
218
+
235
219
// Rows
236
-
rows.forEach(row => {
220
+
rows.forEach((row) => {
237
221
const formattedRow = row.map((cell, i) => cell.padEnd(colWidths[i])).join(' │ ');
238
222
console.log(formattedRow);
239
223
});