this repo has no description
docs: add multi-source implementation plan
+1090
+1090
docs/plans/2026-02-13-multi-source-implementation.md
+1090
docs/plans/2026-02-13-multi-source-implementation.md
···
1
+
# Multi-Source Support Implementation Plan
2
+
3
+
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
4
+
5
+
**Goal:** Add source metadata to links, quotes, and images so Tumble can track where posts originate and scope duplicate detection per source.
6
+
7
+
**Architecture:** Add five nullable source columns to all three content tables via GORM struct changes. Modify the Store interface to accept a `SourceFilter` for queries and duplicate detection. Update API handlers to parse, pass through, and return source fields.
8
+
9
+
**Tech Stack:** Go, GORM, SQLite/MySQL, `net/http`, `httptest` for testing
10
+
11
+
---
12
+
13
+
### Task 1: Add source fields to data models
14
+
15
+
**Files:**
16
+
- Modify: `internal/data/store.go`
17
+
18
+
**Step 1: Add source fields to IRCLink struct**
19
+
20
+
In `internal/data/store.go`, add five fields to the `IRCLink` struct (after the `ContentType` field, line 15):
21
+
22
+
```go
23
+
type IRCLink struct {
24
+
ID int `json:"ircLinkID" gorm:"column:ircLinkID;primaryKey"`
25
+
Timestamp time.Time `json:"timestamp" gorm:"column:timestamp"`
26
+
User string `json:"user" gorm:"column:user;index"`
27
+
Title string `json:"title" gorm:"column:title"`
28
+
URL string `json:"url" gorm:"column:url"`
29
+
Clicks int `json:"clicks" gorm:"column:clicks;default:0"`
30
+
ContentType string `json:"content_type" gorm:"column:content_type"`
31
+
SourceType *string `json:"source_type,omitempty" gorm:"column:source_type;type:varchar(50);index:idx_source,priority:1"`
32
+
SourceNetwork *string `json:"source_network,omitempty" gorm:"column:source_network;type:varchar(255);index:idx_source,priority:2"`
33
+
SourceChannel *string `json:"source_channel,omitempty" gorm:"column:source_channel;type:varchar(255);index:idx_source,priority:3"`
34
+
SourceUserID *string `json:"source_user_id,omitempty" gorm:"column:source_user_id;type:varchar(255)"`
35
+
SourceUserName *string `json:"source_user_name,omitempty" gorm:"column:source_user_name;type:varchar(255)"`
36
+
}
37
+
```
38
+
39
+
**Step 2: Add source fields to Image struct**
40
+
41
+
Same five fields added to `Image` (after `MD5Sum`, line 29):
42
+
43
+
```go
44
+
type Image struct {
45
+
ID int `json:"imageID" gorm:"column:imageID;primaryKey"`
46
+
Timestamp time.Time `json:"timestamp" gorm:"column:timestamp"`
47
+
Title string `json:"title" gorm:"column:title"`
48
+
Link string `json:"link" gorm:"column:link"`
49
+
URL string `json:"url" gorm:"column:url"`
50
+
MD5Sum string `json:"md5sum" gorm:"column:md5sum"`
51
+
SourceType *string `json:"source_type,omitempty" gorm:"column:source_type;type:varchar(50);index:idx_source,priority:1"`
52
+
SourceNetwork *string `json:"source_network,omitempty" gorm:"column:source_network;type:varchar(255);index:idx_source,priority:2"`
53
+
SourceChannel *string `json:"source_channel,omitempty" gorm:"column:source_channel;type:varchar(255);index:idx_source,priority:3"`
54
+
SourceUserID *string `json:"source_user_id,omitempty" gorm:"column:source_user_id;type:varchar(255)"`
55
+
SourceUserName *string `json:"source_user_name,omitempty" gorm:"column:source_user_name;type:varchar(255)"`
56
+
}
57
+
```
58
+
59
+
**Step 3: Add source fields to Quote struct**
60
+
61
+
Same five fields added to `Quote` (after `Poster`, line 42):
62
+
63
+
```go
64
+
type Quote struct {
65
+
ID int `json:"quoteID" gorm:"column:quoteID;primaryKey"`
66
+
Timestamp time.Time `json:"timestamp" gorm:"column:timestamp"`
67
+
Quote string `json:"quote" gorm:"column:quote"`
68
+
Author string `json:"author" gorm:"column:author;type:varchar(255);index"`
69
+
Poster string `json:"poster,omitempty" gorm:"column:poster;type:varchar(255);index"`
70
+
SourceType *string `json:"source_type,omitempty" gorm:"column:source_type;type:varchar(50);index:idx_source,priority:1"`
71
+
SourceNetwork *string `json:"source_network,omitempty" gorm:"column:source_network;type:varchar(255);index:idx_source,priority:2"`
72
+
SourceChannel *string `json:"source_channel,omitempty" gorm:"column:source_channel;type:varchar(255);index:idx_source,priority:3"`
73
+
SourceUserID *string `json:"source_user_id,omitempty" gorm:"column:source_user_id;type:varchar(255)"`
74
+
SourceUserName *string `json:"source_user_name,omitempty" gorm:"column:source_user_name;type:varchar(255)"`
75
+
}
76
+
```
77
+
78
+
**Step 4: Add source fields to TimelineItem struct**
79
+
80
+
Add source fields to `TimelineItem` (after `ContentType`, line 65):
81
+
82
+
```go
83
+
type TimelineItem struct {
84
+
Type string `json:"type"`
85
+
ID int `json:"id"`
86
+
Timestamp time.Time `json:"timestamp"`
87
+
Title string `json:"title"`
88
+
URL string `json:"url"`
89
+
Content string `json:"content"`
90
+
Author string `json:"author"`
91
+
MD5Sum string `json:"md5sum"`
92
+
ContentType string `json:"contentType" gorm:"column:content_type"`
93
+
SourceType *string `json:"source_type,omitempty"`
94
+
SourceNetwork *string `json:"source_network,omitempty"`
95
+
SourceChannel *string `json:"source_channel,omitempty"`
96
+
SourceUserID *string `json:"source_user_id,omitempty"`
97
+
SourceUserName *string `json:"source_user_name,omitempty"`
98
+
}
99
+
```
100
+
101
+
**Step 5: Add SourceFilter type**
102
+
103
+
Add a new type after the `TimelineItem` struct:
104
+
105
+
```go
106
+
// SourceFilter is used to filter queries by source metadata.
107
+
// When all fields are nil, no source filtering is applied.
108
+
type SourceFilter struct {
109
+
SourceType *string
110
+
SourceNetwork *string
111
+
SourceChannel *string
112
+
}
113
+
```
114
+
115
+
**Step 6: Run tests to verify nothing broke**
116
+
117
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test`
118
+
Expected: All existing tests pass (struct additions are backward compatible)
119
+
120
+
**Step 7: Commit**
121
+
122
+
```bash
123
+
git add internal/data/store.go
124
+
git commit -m "feat: add source metadata fields to data models"
125
+
```
126
+
127
+
---
128
+
129
+
### Task 2: Update Store interface and GormStore for source-aware queries
130
+
131
+
**Files:**
132
+
- Modify: `internal/data/store.go` (interface)
133
+
- Modify: `internal/data/gorm_store.go` (implementation)
134
+
135
+
**Step 1: Write failing test for source-scoped InsertIRCLink**
136
+
137
+
Create test in a new file `internal/data/source_filter_test.go`:
138
+
139
+
```go
140
+
package data
141
+
142
+
import (
143
+
"testing"
144
+
)
145
+
146
+
func TestSourceFilter_IsEmpty(t *testing.T) {
147
+
tests := []struct {
148
+
name string
149
+
filter SourceFilter
150
+
expected bool
151
+
}{
152
+
{"all nil", SourceFilter{}, true},
153
+
{"type set", SourceFilter{SourceType: strPtr("irc")}, false},
154
+
{"network set", SourceFilter{SourceNetwork: strPtr("server")}, false},
155
+
{"channel set", SourceFilter{SourceChannel: strPtr("#chan")}, false},
156
+
{"all set", SourceFilter{
157
+
SourceType: strPtr("irc"),
158
+
SourceNetwork: strPtr("server"),
159
+
SourceChannel: strPtr("#chan"),
160
+
}, false},
161
+
}
162
+
163
+
for _, tt := range tests {
164
+
t.Run(tt.name, func(t *testing.T) {
165
+
result := tt.filter.IsEmpty()
166
+
if result != tt.expected {
167
+
t.Errorf("expected %v, got %v", tt.expected, result)
168
+
}
169
+
})
170
+
}
171
+
}
172
+
173
+
func strPtr(s string) *string {
174
+
return &s
175
+
}
176
+
```
177
+
178
+
**Step 2: Run test to verify it fails**
179
+
180
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/data/ -run TestSourceFilter`
181
+
Expected: FAIL — `SourceFilter` has no `IsEmpty` method
182
+
183
+
**Step 3: Implement IsEmpty on SourceFilter**
184
+
185
+
In `internal/data/store.go`, add method after the `SourceFilter` struct:
186
+
187
+
```go
188
+
// IsEmpty returns true if no source filter fields are set.
189
+
func (f SourceFilter) IsEmpty() bool {
190
+
return f.SourceType == nil && f.SourceNetwork == nil && f.SourceChannel == nil
191
+
}
192
+
```
193
+
194
+
**Step 4: Run test to verify it passes**
195
+
196
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/data/ -run TestSourceFilter`
197
+
Expected: PASS
198
+
199
+
**Step 5: Update Store interface signatures**
200
+
201
+
In `internal/data/store.go`, change these method signatures:
202
+
203
+
Old:
204
+
```go
205
+
InsertIRCLink(ctx context.Context, user, title, url, contentType string) (int, error)
206
+
GetIRCLinksByURL(ctx context.Context, url string) ([]IRCLink, error)
207
+
GetRecentIRCLinks(ctx context.Context, days int, offsetDays int) ([]IRCLink, error)
208
+
GetRecentQuotes(ctx context.Context, days int, offsetDays int) ([]Quote, error)
209
+
GetRecentImages(ctx context.Context, days int, offsetDays int) ([]Image, error)
210
+
InsertQuote(ctx context.Context, quote, author, poster string) (int, error)
211
+
SearchIRCLinks(ctx context.Context, query string) ([]IRCLink, error)
212
+
SearchQuotes(ctx context.Context, query string) ([]Quote, error)
213
+
```
214
+
215
+
New:
216
+
```go
217
+
InsertIRCLink(ctx context.Context, link *IRCLink) (int, error)
218
+
GetIRCLinksByURL(ctx context.Context, url string, filter SourceFilter) ([]IRCLink, error)
219
+
GetRecentIRCLinks(ctx context.Context, days int, offsetDays int, filter SourceFilter) ([]IRCLink, error)
220
+
GetRecentQuotes(ctx context.Context, days int, offsetDays int, filter SourceFilter) ([]Quote, error)
221
+
GetRecentImages(ctx context.Context, days int, offsetDays int, filter SourceFilter) ([]Image, error)
222
+
InsertQuote(ctx context.Context, quote *Quote) (int, error)
223
+
InsertImage(ctx context.Context, image *Image) (int, error)
224
+
SearchIRCLinks(ctx context.Context, query string, filter SourceFilter) ([]IRCLink, error)
225
+
SearchQuotes(ctx context.Context, query string, filter SourceFilter) ([]Quote, error)
226
+
```
227
+
228
+
**Step 6: Update GormStore.InsertIRCLink**
229
+
230
+
In `internal/data/gorm_store.go`, change `InsertIRCLink` (line 204):
231
+
232
+
Old:
233
+
```go
234
+
func (s *GormStore) InsertIRCLink(ctx context.Context, user, title, url, contentType string) (int, error) {
235
+
link := IRCLink{
236
+
User: user,
237
+
Title: title,
238
+
URL: url,
239
+
ContentType: contentType,
240
+
Timestamp: time.Now(),
241
+
Clicks: 0,
242
+
}
243
+
err := s.db.WithContext(ctx).Create(&link).Error
244
+
return link.ID, err
245
+
}
246
+
```
247
+
248
+
New:
249
+
```go
250
+
func (s *GormStore) InsertIRCLink(ctx context.Context, link *IRCLink) (int, error) {
251
+
link.Timestamp = time.Now()
252
+
link.Clicks = 0
253
+
err := s.db.WithContext(ctx).Create(link).Error
254
+
return link.ID, err
255
+
}
256
+
```
257
+
258
+
**Step 7: Update GormStore.InsertQuote**
259
+
260
+
Find the `InsertQuote` method in `gorm_store.go` and update similarly:
261
+
262
+
Old:
263
+
```go
264
+
func (s *GormStore) InsertQuote(ctx context.Context, quote, author, poster string) (int, error) {
265
+
q := Quote{
266
+
Quote: quote,
267
+
Author: author,
268
+
Poster: poster,
269
+
Timestamp: time.Now(),
270
+
}
271
+
err := s.db.WithContext(ctx).Create(&q).Error
272
+
return q.ID, err
273
+
}
274
+
```
275
+
276
+
New:
277
+
```go
278
+
func (s *GormStore) InsertQuote(ctx context.Context, quote *Quote) (int, error) {
279
+
quote.Timestamp = time.Now()
280
+
err := s.db.WithContext(ctx).Create(quote).Error
281
+
return quote.ID, err
282
+
}
283
+
```
284
+
285
+
**Step 8: Update GormStore.InsertImage**
286
+
287
+
Old:
288
+
```go
289
+
func (s *GormStore) InsertImage(ctx context.Context, title, link, url string) (int, error) {
290
+
img := Image{
291
+
Title: title,
292
+
Link: link,
293
+
URL: url,
294
+
Timestamp: time.Now(),
295
+
}
296
+
err := s.db.WithContext(ctx).Create(&img).Error
297
+
return img.ID, err
298
+
}
299
+
```
300
+
301
+
New:
302
+
```go
303
+
func (s *GormStore) InsertImage(ctx context.Context, image *Image) (int, error) {
304
+
image.Timestamp = time.Now()
305
+
err := s.db.WithContext(ctx).Create(image).Error
306
+
return image.ID, err
307
+
}
308
+
```
309
+
310
+
**Step 9: Update GormStore.GetIRCLinksByURL for source-scoped duplicate detection**
311
+
312
+
In `gorm_store.go` (line 191):
313
+
314
+
Old:
315
+
```go
316
+
func (s *GormStore) GetIRCLinksByURL(ctx context.Context, url string) ([]IRCLink, error) {
317
+
var links []IRCLink
318
+
err := s.db.WithContext(ctx).
319
+
Where("url = ?", url).
320
+
Order("timestamp DESC").
321
+
Find(&links).Error
322
+
return links, err
323
+
}
324
+
```
325
+
326
+
New:
327
+
```go
328
+
func (s *GormStore) GetIRCLinksByURL(ctx context.Context, url string, filter SourceFilter) ([]IRCLink, error) {
329
+
var links []IRCLink
330
+
query := s.db.WithContext(ctx).Where("url = ?", url)
331
+
query = applySourceFilter(query, filter)
332
+
err := query.Order("timestamp DESC").Find(&links).Error
333
+
return links, err
334
+
}
335
+
```
336
+
337
+
**Step 10: Update GormStore.GetRecentIRCLinks for source filtering**
338
+
339
+
In `gorm_store.go` (line 32):
340
+
341
+
Old:
342
+
```go
343
+
func (s *GormStore) GetRecentIRCLinks(ctx context.Context, startDays int, endDays int) ([]IRCLink, error) {
344
+
var links []IRCLink
345
+
now := time.Now()
346
+
startDate := now.AddDate(0, 0, -startDays)
347
+
endDate := now.AddDate(0, 0, -endDays)
348
+
349
+
err := s.db.WithContext(ctx).
350
+
Where("timestamp >= ? AND timestamp <= ?", startDate, endDate).
351
+
Order("timestamp DESC").
352
+
Find(&links).Error
353
+
return links, err
354
+
}
355
+
```
356
+
357
+
New:
358
+
```go
359
+
func (s *GormStore) GetRecentIRCLinks(ctx context.Context, startDays int, endDays int, filter SourceFilter) ([]IRCLink, error) {
360
+
var links []IRCLink
361
+
now := time.Now()
362
+
startDate := now.AddDate(0, 0, -startDays)
363
+
endDate := now.AddDate(0, 0, -endDays)
364
+
365
+
query := s.db.WithContext(ctx).
366
+
Where("timestamp >= ? AND timestamp <= ?", startDate, endDate)
367
+
query = applySourceFilter(query, filter)
368
+
err := query.Order("timestamp DESC").Find(&links).Error
369
+
return links, err
370
+
}
371
+
```
372
+
373
+
**Step 11: Apply same pattern to GetRecentQuotes, GetRecentImages, SearchIRCLinks, SearchQuotes**
374
+
375
+
Add `filter SourceFilter` parameter and `applySourceFilter(query, filter)` call to each. Follow exact same pattern as Step 10.
376
+
377
+
**Step 12: Add applySourceFilter helper**
378
+
379
+
Add this function to `gorm_store.go`:
380
+
381
+
```go
382
+
// applySourceFilter adds WHERE clauses for source metadata fields.
383
+
// When filter is empty, no clauses are added (backward compatible).
384
+
func applySourceFilter(query *gorm.DB, filter SourceFilter) *gorm.DB {
385
+
if filter.SourceType != nil {
386
+
query = query.Where("source_type = ?", *filter.SourceType)
387
+
}
388
+
if filter.SourceNetwork != nil {
389
+
query = query.Where("source_network = ?", *filter.SourceNetwork)
390
+
}
391
+
if filter.SourceChannel != nil {
392
+
query = query.Where("source_channel = ?", *filter.SourceChannel)
393
+
}
394
+
return query
395
+
}
396
+
```
397
+
398
+
**Step 13: Update all other callers of changed methods**
399
+
400
+
Search the codebase for all calls to `InsertIRCLink`, `InsertQuote`, `InsertImage`, `GetIRCLinksByURL`, `GetRecentIRCLinks`, `GetRecentQuotes`, `GetRecentImages`, `SearchIRCLinks`, `SearchQuotes`. Update each call site to pass the new parameters. For existing callers that don't have source context, pass `data.SourceFilter{}` (empty filter).
401
+
402
+
Key callers to update:
403
+
- `internal/handler/api_v1_links.go` — `apiV1CreateLink`, `apiV1ListLinks`
404
+
- `internal/handler/api_v1_quotes.go` — `apiV1CreateQuote`, `apiV1ListQuotes`
405
+
- `internal/handler/api_v1_search.go` — `APIv1SearchHandler`
406
+
- `internal/handler/irclink.go` — legacy handler (if it exists)
407
+
- `internal/handler/handlers.go` — any frontend handlers calling these
408
+
- `internal/scheduler/` — any background jobs
409
+
- All test mock implementations
410
+
411
+
**Step 14: Update mock stores in test files**
412
+
413
+
Update `mockAPIStore` in `internal/handler/api_v1_links_test.go` to match new signatures:
414
+
415
+
```go
416
+
func (m *mockAPIStore) GetRecentIRCLinks(ctx context.Context, days int, offsetDays int, filter data.SourceFilter) ([]data.IRCLink, error) {
417
+
if m.err != nil {
418
+
return nil, m.err
419
+
}
420
+
return m.links, nil
421
+
}
422
+
423
+
func (m *mockAPIStore) GetIRCLinksByURL(ctx context.Context, url string, filter data.SourceFilter) ([]data.IRCLink, error) {
424
+
if m.linksByURLFn != nil {
425
+
return m.linksByURLFn(url)
426
+
}
427
+
if m.err != nil {
428
+
return nil, m.err
429
+
}
430
+
return m.linksByURL, nil
431
+
}
432
+
433
+
func (m *mockAPIStore) InsertIRCLink(ctx context.Context, link *data.IRCLink) (int, error) {
434
+
if m.insertLinkFn != nil {
435
+
return m.insertLinkFn(link.User, link.Title, link.URL, link.ContentType)
436
+
}
437
+
if m.err != nil {
438
+
return 0, m.err
439
+
}
440
+
return m.insertedLinkID, nil
441
+
}
442
+
```
443
+
444
+
Do the same for `mockQuoteStore` in `api_v1_quotes_test.go` and any other mock stores.
445
+
446
+
**Step 15: Run tests**
447
+
448
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test`
449
+
Expected: All tests pass. Compilation succeeds with updated signatures.
450
+
451
+
**Step 16: Commit**
452
+
453
+
```bash
454
+
git add internal/data/store.go internal/data/gorm_store.go internal/data/source_filter_test.go
455
+
git add internal/handler/
456
+
git commit -m "feat: update Store interface for source-aware queries
457
+
458
+
Change Insert methods to accept struct pointers instead of
459
+
individual parameters. Add SourceFilter to Get and Search
460
+
methods. Add applySourceFilter helper for GORM queries."
461
+
```
462
+
463
+
---
464
+
465
+
### Task 3: Update API handlers to accept and return source fields
466
+
467
+
**Files:**
468
+
- Modify: `internal/handler/api_v1_types.go`
469
+
- Modify: `internal/handler/api_v1_links.go`
470
+
- Modify: `internal/handler/api_v1_quotes.go`
471
+
- Modify: `internal/handler/api_v1_search.go`
472
+
473
+
**Step 1: Write failing test for source fields in link creation**
474
+
475
+
Add to `internal/handler/api_v1_links_test.go`, in the `TestAPIv1_CreateLink` function's test table:
476
+
477
+
```go
478
+
{
479
+
name: "valid link with source fields",
480
+
body: `{"url":"https://example.com","user":"testuser","source_type":"slack","source_network":"T12345","source_channel":"C67890","source_user_id":"U99999","source_user_name":"testuser"}`,
481
+
store: &mockAPIStore{insertedLinkID: 42},
482
+
expectedStatus: http.StatusCreated,
483
+
checkBody: func(t *testing.T, body []byte) {
484
+
var resp APILinkCreateResponse
485
+
if err := json.Unmarshal(body, &resp); err != nil {
486
+
t.Fatalf("failed to unmarshal: %v", err)
487
+
}
488
+
if resp.SourceType == nil || *resp.SourceType != "slack" {
489
+
t.Errorf("expected source_type 'slack', got %v", resp.SourceType)
490
+
}
491
+
if resp.SourceNetwork == nil || *resp.SourceNetwork != "T12345" {
492
+
t.Errorf("expected source_network 'T12345', got %v", resp.SourceNetwork)
493
+
}
494
+
},
495
+
},
496
+
```
497
+
498
+
**Step 2: Run test to verify it fails**
499
+
500
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/handler/ -run TestAPIv1_CreateLink/valid_link_with_source_fields`
501
+
Expected: FAIL — `APILinkCreateResponse` has no `SourceType` field
502
+
503
+
**Step 3: Add source fields to API response types**
504
+
505
+
In `internal/handler/api_v1_types.go`, add source fields to `APILinkResponse`:
506
+
507
+
```go
508
+
type APILinkResponse struct {
509
+
ID int `json:"id"`
510
+
URL string `json:"url"`
511
+
Title string `json:"title"`
512
+
User string `json:"user"`
513
+
Clicks int `json:"clicks"`
514
+
CreatedAt time.Time `json:"created_at"`
515
+
Tags []string `json:"tags,omitempty"`
516
+
SourceType *string `json:"source_type,omitempty"`
517
+
SourceNetwork *string `json:"source_network,omitempty"`
518
+
SourceChannel *string `json:"source_channel,omitempty"`
519
+
SourceUserID *string `json:"source_user_id,omitempty"`
520
+
SourceUserName *string `json:"source_user_name,omitempty"`
521
+
}
522
+
```
523
+
524
+
Add source fields to `APIQuoteResponse`:
525
+
526
+
```go
527
+
type APIQuoteResponse struct {
528
+
ID int `json:"id"`
529
+
Quote string `json:"quote"`
530
+
Author string `json:"author"`
531
+
Poster string `json:"poster,omitempty"`
532
+
CreatedAt time.Time `json:"created_at"`
533
+
Tags []string `json:"tags,omitempty"`
534
+
SourceType *string `json:"source_type,omitempty"`
535
+
SourceNetwork *string `json:"source_network,omitempty"`
536
+
SourceChannel *string `json:"source_channel,omitempty"`
537
+
SourceUserID *string `json:"source_user_id,omitempty"`
538
+
SourceUserName *string `json:"source_user_name,omitempty"`
539
+
}
540
+
```
541
+
542
+
**Step 4: Add source fields to request types**
543
+
544
+
In `internal/handler/api_v1_links.go`, update `APILinkCreateRequest`:
545
+
546
+
```go
547
+
type APILinkCreateRequest struct {
548
+
URL string `json:"url"`
549
+
User string `json:"user"`
550
+
Tags []string `json:"tags,omitempty"`
551
+
SourceType *string `json:"source_type,omitempty"`
552
+
SourceNetwork *string `json:"source_network,omitempty"`
553
+
SourceChannel *string `json:"source_channel,omitempty"`
554
+
SourceUserID *string `json:"source_user_id,omitempty"`
555
+
SourceUserName *string `json:"source_user_name,omitempty"`
556
+
}
557
+
```
558
+
559
+
In `internal/handler/api_v1_quotes.go`, update `APIQuoteCreateRequest`:
560
+
561
+
```go
562
+
type APIQuoteCreateRequest struct {
563
+
Quote string `json:"quote"`
564
+
Author string `json:"author"`
565
+
Poster string `json:"poster"`
566
+
Tags []string `json:"tags,omitempty"`
567
+
SourceType *string `json:"source_type,omitempty"`
568
+
SourceNetwork *string `json:"source_network,omitempty"`
569
+
SourceChannel *string `json:"source_channel,omitempty"`
570
+
SourceUserID *string `json:"source_user_id,omitempty"`
571
+
SourceUserName *string `json:"source_user_name,omitempty"`
572
+
}
573
+
```
574
+
575
+
**Step 5: Update apiV1CreateLink to pass source fields through**
576
+
577
+
In `internal/handler/api_v1_links.go`, update `apiV1CreateLink` to build an `IRCLink` struct and pass source fields:
578
+
579
+
```go
580
+
// Build source filter for duplicate detection
581
+
sourceFilter := data.SourceFilter{
582
+
SourceType: req.SourceType,
583
+
SourceNetwork: req.SourceNetwork,
584
+
SourceChannel: req.SourceChannel,
585
+
}
586
+
587
+
// Check for duplicates (scoped by source when provided)
588
+
existingLinks, err := h.Store.GetIRCLinksByURL(ctx, req.URL, sourceFilter)
589
+
590
+
// Build the link struct with source metadata
591
+
link := &data.IRCLink{
592
+
User: req.User,
593
+
Title: req.URL,
594
+
URL: req.URL,
595
+
ContentType: "",
596
+
SourceType: req.SourceType,
597
+
SourceNetwork: req.SourceNetwork,
598
+
SourceChannel: req.SourceChannel,
599
+
SourceUserID: req.SourceUserID,
600
+
SourceUserName: req.SourceUserName,
601
+
}
602
+
linkID, err := h.Store.InsertIRCLink(ctx, link)
603
+
```
604
+
605
+
Update the response builder to include source fields:
606
+
607
+
```go
608
+
resp := APILinkCreateResponse{
609
+
APILinkResponse: APILinkResponse{
610
+
ID: linkID,
611
+
URL: req.URL,
612
+
Title: req.URL,
613
+
User: req.User,
614
+
Clicks: 0,
615
+
CreatedAt: time.Now(),
616
+
Tags: tagStrings,
617
+
SourceType: req.SourceType,
618
+
SourceNetwork: req.SourceNetwork,
619
+
SourceChannel: req.SourceChannel,
620
+
SourceUserID: req.SourceUserID,
621
+
SourceUserName: req.SourceUserName,
622
+
},
623
+
IsDuplicate: isDuplicate,
624
+
PreviousSubmissions: previousSubmissions,
625
+
}
626
+
```
627
+
628
+
**Step 6: Update apiV1CreateQuote similarly**
629
+
630
+
Pass source fields through to `InsertQuote` and include in response.
631
+
632
+
**Step 7: Update apiV1ListLinks to parse source query params and pass filter**
633
+
634
+
In `internal/handler/api_v1_links.go`, in `apiV1ListLinks`:
635
+
636
+
```go
637
+
// Parse source filter parameters
638
+
var sourceFilter data.SourceFilter
639
+
if st := r.URL.Query().Get("source_type"); st != "" {
640
+
sourceFilter.SourceType = &st
641
+
}
642
+
if sn := r.URL.Query().Get("source_network"); sn != "" {
643
+
sourceFilter.SourceNetwork = &sn
644
+
}
645
+
if sc := r.URL.Query().Get("source_channel"); sc != "" {
646
+
sourceFilter.SourceChannel = &sc
647
+
}
648
+
649
+
links, err := h.Store.GetRecentIRCLinks(ctx, 365, 0, sourceFilter)
650
+
```
651
+
652
+
Update the response conversion loop to include source fields from each link:
653
+
654
+
```go
655
+
data = append(data, APILinkResponse{
656
+
ID: link.ID,
657
+
URL: link.URL,
658
+
Title: link.Title,
659
+
User: link.User,
660
+
Clicks: link.Clicks,
661
+
CreatedAt: link.Timestamp,
662
+
Tags: h.getTagStrings(ctx, "link", link.ID),
663
+
SourceType: link.SourceType,
664
+
SourceNetwork: link.SourceNetwork,
665
+
SourceChannel: link.SourceChannel,
666
+
SourceUserID: link.SourceUserID,
667
+
SourceUserName: link.SourceUserName,
668
+
})
669
+
```
670
+
671
+
**Step 8: Update apiV1ListQuotes the same way**
672
+
673
+
Parse source query params, pass filter to `GetRecentQuotes`, include source fields in response.
674
+
675
+
**Step 9: Update apiV1GetLink and apiV1GetQuote responses**
676
+
677
+
Include source fields from the fetched link/quote in the response structs.
678
+
679
+
**Step 10: Update APIv1SearchHandler**
680
+
681
+
In `internal/handler/api_v1_search.go`, parse source query params and pass filter to `SearchIRCLinks` and `SearchQuotes`. Include source fields in the response conversion loops.
682
+
683
+
**Step 11: Run tests**
684
+
685
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test`
686
+
Expected: All tests pass including the new source fields test
687
+
688
+
**Step 12: Commit**
689
+
690
+
```bash
691
+
git add internal/handler/
692
+
git commit -m "feat: accept and return source fields in API endpoints
693
+
694
+
POST links/quotes accepts optional source_type, source_network,
695
+
source_channel, source_user_id, source_user_name fields.
696
+
GET links/quotes/search supports source_type, source_network,
697
+
source_channel query parameters for filtering.
698
+
All responses include source fields with omitempty."
699
+
```
700
+
701
+
---
702
+
703
+
### Task 4: Write comprehensive tests for source-scoped duplicate detection
704
+
705
+
**Files:**
706
+
- Modify: `internal/handler/api_v1_links_test.go`
707
+
708
+
**Step 1: Write test cases for scoped duplicate detection**
709
+
710
+
Add a new test function `TestAPIv1_CreateLink_SourceDuplicates`:
711
+
712
+
```go
713
+
func TestAPIv1_CreateLink_SourceDuplicates(t *testing.T) {
714
+
now := time.Now()
715
+
716
+
tests := []struct {
717
+
name string
718
+
body string
719
+
existingLinks []data.IRCLink
720
+
expectedStatus int
721
+
expectedDuplicate bool
722
+
}{
723
+
{
724
+
name: "same URL different source is not duplicate",
725
+
body: `{"url":"https://example.com","user":"alice","source_type":"slack","source_network":"T111","source_channel":"C222"}`,
726
+
existingLinks: nil, // source-scoped query returns nothing
727
+
expectedStatus: http.StatusCreated,
728
+
expectedDuplicate: false,
729
+
},
730
+
{
731
+
name: "same URL same source is duplicate",
732
+
body: `{"url":"https://example.com","user":"bob","source_type":"slack","source_network":"T111","source_channel":"C222"}`,
733
+
existingLinks: []data.IRCLink{
734
+
{ID: 10, User: "alice", URL: "https://example.com", Timestamp: now},
735
+
},
736
+
expectedStatus: http.StatusCreated,
737
+
expectedDuplicate: true,
738
+
},
739
+
{
740
+
name: "no source fields uses global duplicate check",
741
+
body: `{"url":"https://example.com","user":"charlie"}`,
742
+
existingLinks: []data.IRCLink{
743
+
{ID: 10, User: "alice", URL: "https://example.com", Timestamp: now},
744
+
},
745
+
expectedStatus: http.StatusCreated,
746
+
expectedDuplicate: true,
747
+
},
748
+
}
749
+
750
+
for _, tt := range tests {
751
+
t.Run(tt.name, func(t *testing.T) {
752
+
store := &mockAPIStore{
753
+
linksByURL: tt.existingLinks,
754
+
insertedLinkID: 42,
755
+
}
756
+
handler := NewHandler(store, &config.Config{})
757
+
req := httptest.NewRequest(http.MethodPost, "/api/v1/links", strings.NewReader(tt.body))
758
+
req.RemoteAddr = "127.0.0.1:12345"
759
+
w := httptest.NewRecorder()
760
+
761
+
handler.APIv1LinksHandler(w, req)
762
+
763
+
if w.Code != tt.expectedStatus {
764
+
t.Errorf("expected status %d, got %d", tt.expectedStatus, w.Code)
765
+
}
766
+
767
+
var resp APILinkCreateResponse
768
+
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
769
+
t.Fatalf("failed to unmarshal: %v", err)
770
+
}
771
+
if resp.IsDuplicate != tt.expectedDuplicate {
772
+
t.Errorf("expected is_duplicate=%v, got %v", tt.expectedDuplicate, resp.IsDuplicate)
773
+
}
774
+
})
775
+
}
776
+
}
777
+
```
778
+
779
+
**Step 2: Run tests**
780
+
781
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/handler/ -run TestAPIv1_CreateLink_SourceDuplicates`
782
+
Expected: PASS
783
+
784
+
**Step 3: Write test for source filtering on GET**
785
+
786
+
Add `TestAPIv1_ListLinks_SourceFiltering`:
787
+
788
+
```go
789
+
func TestAPIv1_ListLinks_SourceFiltering(t *testing.T) {
790
+
slackType := "slack"
791
+
slackNetwork := "T12345"
792
+
slackChannel := "C67890"
793
+
794
+
tests := []struct {
795
+
name string
796
+
queryParams string
797
+
links []data.IRCLink
798
+
expectedTotal int
799
+
}{
800
+
{
801
+
name: "no filter returns all",
802
+
queryParams: "",
803
+
links: []data.IRCLink{
804
+
{ID: 1, User: "alice", URL: "https://a.com", Timestamp: time.Now()},
805
+
{ID: 2, User: "bob", URL: "https://b.com", Timestamp: time.Now(), SourceType: &slackType},
806
+
},
807
+
expectedTotal: 2,
808
+
},
809
+
{
810
+
name: "filter by source_type",
811
+
queryParams: "?source_type=slack",
812
+
links: []data.IRCLink{
813
+
{ID: 2, User: "bob", URL: "https://b.com", Timestamp: time.Now(), SourceType: &slackType},
814
+
},
815
+
expectedTotal: 1,
816
+
},
817
+
}
818
+
819
+
for _, tt := range tests {
820
+
t.Run(tt.name, func(t *testing.T) {
821
+
store := &mockAPIStore{links: tt.links}
822
+
handler := NewHandler(store, &config.Config{})
823
+
req := httptest.NewRequest(http.MethodGet, "/api/v1/links"+tt.queryParams, nil)
824
+
w := httptest.NewRecorder()
825
+
826
+
handler.APIv1LinksHandler(w, req)
827
+
828
+
var resp APILinksResponse
829
+
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
830
+
t.Fatalf("failed to unmarshal: %v", err)
831
+
}
832
+
if resp.Meta.Total != tt.expectedTotal {
833
+
t.Errorf("expected total %d, got %d", tt.expectedTotal, resp.Meta.Total)
834
+
}
835
+
})
836
+
}
837
+
}
838
+
```
839
+
840
+
**Step 4: Write test for omitempty serialization**
841
+
842
+
Add `TestAPIv1_LinkResponse_SourceOmitEmpty`:
843
+
844
+
```go
845
+
func TestAPIv1_LinkResponse_SourceOmitEmpty(t *testing.T) {
846
+
t.Run("null source fields omitted from JSON", func(t *testing.T) {
847
+
store := &mockAPIStore{
848
+
links: []data.IRCLink{
849
+
{ID: 1, User: "alice", URL: "https://a.com", Timestamp: time.Now()},
850
+
},
851
+
}
852
+
handler := NewHandler(store, &config.Config{})
853
+
req := httptest.NewRequest(http.MethodGet, "/api/v1/links", nil)
854
+
w := httptest.NewRecorder()
855
+
856
+
handler.APIv1LinksHandler(w, req)
857
+
858
+
body := w.Body.String()
859
+
if strings.Contains(body, "source_type") {
860
+
t.Error("expected source_type to be omitted when null")
861
+
}
862
+
})
863
+
864
+
t.Run("source fields present when set", func(t *testing.T) {
865
+
slackType := "slack"
866
+
store := &mockAPIStore{
867
+
links: []data.IRCLink{
868
+
{ID: 1, User: "alice", URL: "https://a.com", Timestamp: time.Now(), SourceType: &slackType},
869
+
},
870
+
}
871
+
handler := NewHandler(store, &config.Config{})
872
+
req := httptest.NewRequest(http.MethodGet, "/api/v1/links", nil)
873
+
w := httptest.NewRecorder()
874
+
875
+
handler.APIv1LinksHandler(w, req)
876
+
877
+
body := w.Body.String()
878
+
if !strings.Contains(body, `"source_type":"slack"`) {
879
+
t.Errorf("expected source_type in response, got: %s", body)
880
+
}
881
+
})
882
+
}
883
+
```
884
+
885
+
**Step 5: Run all tests**
886
+
887
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test`
888
+
Expected: All tests pass
889
+
890
+
**Step 6: Commit**
891
+
892
+
```bash
893
+
git add internal/handler/api_v1_links_test.go
894
+
git commit -m "test: add tests for source-scoped duplicate detection and filtering"
895
+
```
896
+
897
+
---
898
+
899
+
### Task 5: Write backfill SQL script
900
+
901
+
**Files:**
902
+
- Create: `sql/backfill_sources.sql`
903
+
904
+
**Step 1: Create the backfill script**
905
+
906
+
```sql
907
+
-- One-time backfill: set source metadata on all existing rows.
908
+
-- All existing data originates from IRC, #soggies channel on jameswhite.org.
909
+
-- Run this manually after deploying the source fields migration.
910
+
--
911
+
-- Usage (SQLite): sqlite3 tumble.db < sql/backfill_sources.sql
912
+
-- Usage (MySQL): mysql -u user -p tumble < sql/backfill_sources.sql
913
+
914
+
UPDATE ircLink
915
+
SET source_type = 'irc',
916
+
source_network = 'jameswhite.org',
917
+
source_channel = '#soggies'
918
+
WHERE source_type IS NULL;
919
+
920
+
UPDATE quote
921
+
SET source_type = 'irc',
922
+
source_network = 'jameswhite.org',
923
+
source_channel = '#soggies'
924
+
WHERE source_type IS NULL;
925
+
926
+
UPDATE image
927
+
SET source_type = 'irc',
928
+
source_network = 'jameswhite.org',
929
+
source_channel = '#soggies'
930
+
WHERE source_type IS NULL;
931
+
```
932
+
933
+
**Step 2: Commit**
934
+
935
+
```bash
936
+
git add sql/backfill_sources.sql
937
+
git commit -m "feat: add one-time backfill script for source metadata"
938
+
```
939
+
940
+
---
941
+
942
+
### Task 6: Update OpenAPI specification
943
+
944
+
**Files:**
945
+
- Modify: `internal/assets/openapi.json`
946
+
947
+
**Step 1: Add source fields to LinkCreateRequest schema**
948
+
949
+
Find the `LinkCreateRequest` schema in `openapi.json` and add:
950
+
951
+
```json
952
+
"source_type": {
953
+
"type": "string",
954
+
"description": "Source platform (e.g., irc, slack, discord, api, web)",
955
+
"example": "slack"
956
+
},
957
+
"source_network": {
958
+
"type": "string",
959
+
"description": "Source network identifier (e.g., IRC server, Slack team ID)",
960
+
"example": "T12345"
961
+
},
962
+
"source_channel": {
963
+
"type": "string",
964
+
"description": "Source channel identifier (e.g., IRC channel, Slack channel ID)",
965
+
"example": "C67890"
966
+
},
967
+
"source_user_id": {
968
+
"type": "string",
969
+
"description": "Platform-specific user ID",
970
+
"example": "U99999"
971
+
},
972
+
"source_user_name": {
973
+
"type": "string",
974
+
"description": "Display/mention name at time of post",
975
+
"example": "stahnma"
976
+
}
977
+
```
978
+
979
+
**Step 2: Add same fields to QuoteCreateRequest schema**
980
+
981
+
Same five fields.
982
+
983
+
**Step 3: Add source fields to APILinkResponse and APIQuoteResponse schemas**
984
+
985
+
Same five fields added to response schemas.
986
+
987
+
**Step 4: Add source query parameters to GET /api/v1/links**
988
+
989
+
Add three optional query parameters:
990
+
991
+
```json
992
+
{
993
+
"name": "source_type",
994
+
"in": "query",
995
+
"required": false,
996
+
"schema": { "type": "string" },
997
+
"description": "Filter by source platform (e.g., irc, slack, discord)"
998
+
},
999
+
{
1000
+
"name": "source_network",
1001
+
"in": "query",
1002
+
"required": false,
1003
+
"schema": { "type": "string" },
1004
+
"description": "Filter by source network (requires source_type)"
1005
+
},
1006
+
{
1007
+
"name": "source_channel",
1008
+
"in": "query",
1009
+
"required": false,
1010
+
"schema": { "type": "string" },
1011
+
"description": "Filter by source channel (requires source_type and source_network)"
1012
+
}
1013
+
```
1014
+
1015
+
**Step 5: Add same query parameters to GET /api/v1/quotes and GET /api/v1/search**
1016
+
1017
+
**Step 6: Update 208 response description**
1018
+
1019
+
Update the duplicate detection documentation for POST /api/v1/links to note that duplicate detection is scoped per source when source fields are provided.
1020
+
1021
+
**Step 7: Run the app to verify docs render**
1022
+
1023
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make restart`
1024
+
Visit: `http://localhost:8080/api/docs` and verify the new fields appear.
1025
+
Then: `make kill`
1026
+
1027
+
**Step 8: Commit**
1028
+
1029
+
```bash
1030
+
git add internal/assets/openapi.json
1031
+
git commit -m "docs: update OpenAPI spec with source metadata fields
1032
+
1033
+
Add source_type, source_network, source_channel, source_user_id,
1034
+
and source_user_name to request/response schemas. Add source
1035
+
filter query parameters to GET endpoints."
1036
+
```
1037
+
1038
+
---
1039
+
1040
+
### Task 7: Final integration verification
1041
+
1042
+
**Step 1: Run full test suite**
1043
+
1044
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test`
1045
+
Expected: All tests pass
1046
+
1047
+
**Step 2: Run API integration tests**
1048
+
1049
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test-api`
1050
+
Expected: All integration tests pass
1051
+
1052
+
**Step 3: Manual smoke test**
1053
+
1054
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && make restart`
1055
+
1056
+
Test creating a link with source fields:
1057
+
```bash
1058
+
curl -s -X POST http://localhost:8080/api/v1/links \
1059
+
-H "Content-Type: application/json" \
1060
+
-d '{"url":"https://example.com/test","user":"testuser","source_type":"slack","source_network":"T12345","source_channel":"C67890","source_user_id":"U99999","source_user_name":"testuser"}' | jq .
1061
+
```
1062
+
1063
+
Verify source fields in response. Then test filtering:
1064
+
```bash
1065
+
curl -s "http://localhost:8080/api/v1/links?source_type=slack" | jq .
1066
+
```
1067
+
1068
+
Test that creating same URL without source fields still works:
1069
+
```bash
1070
+
curl -s -X POST http://localhost:8080/api/v1/links \
1071
+
-H "Content-Type: application/json" \
1072
+
-d '{"url":"https://example.com/test","user":"otheruser"}' | jq .
1073
+
```
1074
+
1075
+
Then: `make kill`
1076
+
1077
+
**Step 4: Run go fmt**
1078
+
1079
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && go fmt ./...`
1080
+
1081
+
**Step 5: Check for trailing whitespace**
1082
+
1083
+
Run: `cd /Users/stahnma/development/personal/tumble/tumble && git diff --check`
1084
+
1085
+
**Step 6: Final commit if any formatting fixes needed**
1086
+
1087
+
```bash
1088
+
git add -A
1089
+
git commit -m "chore: format and clean up"
1090
+
```