this repo has no description
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

docs: add multi-source implementation plan

+1090
+1090
docs/plans/2026-02-13-multi-source-implementation.md
··· 1 + # Multi-Source Support Implementation Plan 2 + 3 + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. 4 + 5 + **Goal:** Add source metadata to links, quotes, and images so Tumble can track where posts originate and scope duplicate detection per source. 6 + 7 + **Architecture:** Add five nullable source columns to all three content tables via GORM struct changes. Modify the Store interface to accept a `SourceFilter` for queries and duplicate detection. Update API handlers to parse, pass through, and return source fields. 8 + 9 + **Tech Stack:** Go, GORM, SQLite/MySQL, `net/http`, `httptest` for testing 10 + 11 + --- 12 + 13 + ### Task 1: Add source fields to data models 14 + 15 + **Files:** 16 + - Modify: `internal/data/store.go` 17 + 18 + **Step 1: Add source fields to IRCLink struct** 19 + 20 + In `internal/data/store.go`, add five fields to the `IRCLink` struct (after the `ContentType` field, line 15): 21 + 22 + ```go 23 + type IRCLink struct { 24 + ID int `json:"ircLinkID" gorm:"column:ircLinkID;primaryKey"` 25 + Timestamp time.Time `json:"timestamp" gorm:"column:timestamp"` 26 + User string `json:"user" gorm:"column:user;index"` 27 + Title string `json:"title" gorm:"column:title"` 28 + URL string `json:"url" gorm:"column:url"` 29 + Clicks int `json:"clicks" gorm:"column:clicks;default:0"` 30 + ContentType string `json:"content_type" gorm:"column:content_type"` 31 + SourceType *string `json:"source_type,omitempty" gorm:"column:source_type;type:varchar(50);index:idx_source,priority:1"` 32 + SourceNetwork *string `json:"source_network,omitempty" gorm:"column:source_network;type:varchar(255);index:idx_source,priority:2"` 33 + SourceChannel *string `json:"source_channel,omitempty" gorm:"column:source_channel;type:varchar(255);index:idx_source,priority:3"` 34 + SourceUserID *string `json:"source_user_id,omitempty" gorm:"column:source_user_id;type:varchar(255)"` 35 + SourceUserName *string `json:"source_user_name,omitempty" gorm:"column:source_user_name;type:varchar(255)"` 36 + } 37 + ``` 38 + 39 + **Step 2: Add source fields to Image struct** 40 + 41 + Same five fields added to `Image` (after `MD5Sum`, line 29): 42 + 43 + ```go 44 + type Image struct { 45 + ID int `json:"imageID" gorm:"column:imageID;primaryKey"` 46 + Timestamp time.Time `json:"timestamp" gorm:"column:timestamp"` 47 + Title string `json:"title" gorm:"column:title"` 48 + Link string `json:"link" gorm:"column:link"` 49 + URL string `json:"url" gorm:"column:url"` 50 + MD5Sum string `json:"md5sum" gorm:"column:md5sum"` 51 + SourceType *string `json:"source_type,omitempty" gorm:"column:source_type;type:varchar(50);index:idx_source,priority:1"` 52 + SourceNetwork *string `json:"source_network,omitempty" gorm:"column:source_network;type:varchar(255);index:idx_source,priority:2"` 53 + SourceChannel *string `json:"source_channel,omitempty" gorm:"column:source_channel;type:varchar(255);index:idx_source,priority:3"` 54 + SourceUserID *string `json:"source_user_id,omitempty" gorm:"column:source_user_id;type:varchar(255)"` 55 + SourceUserName *string `json:"source_user_name,omitempty" gorm:"column:source_user_name;type:varchar(255)"` 56 + } 57 + ``` 58 + 59 + **Step 3: Add source fields to Quote struct** 60 + 61 + Same five fields added to `Quote` (after `Poster`, line 42): 62 + 63 + ```go 64 + type Quote struct { 65 + ID int `json:"quoteID" gorm:"column:quoteID;primaryKey"` 66 + Timestamp time.Time `json:"timestamp" gorm:"column:timestamp"` 67 + Quote string `json:"quote" gorm:"column:quote"` 68 + Author string `json:"author" gorm:"column:author;type:varchar(255);index"` 69 + Poster string `json:"poster,omitempty" gorm:"column:poster;type:varchar(255);index"` 70 + SourceType *string `json:"source_type,omitempty" gorm:"column:source_type;type:varchar(50);index:idx_source,priority:1"` 71 + SourceNetwork *string `json:"source_network,omitempty" gorm:"column:source_network;type:varchar(255);index:idx_source,priority:2"` 72 + SourceChannel *string `json:"source_channel,omitempty" gorm:"column:source_channel;type:varchar(255);index:idx_source,priority:3"` 73 + SourceUserID *string `json:"source_user_id,omitempty" gorm:"column:source_user_id;type:varchar(255)"` 74 + SourceUserName *string `json:"source_user_name,omitempty" gorm:"column:source_user_name;type:varchar(255)"` 75 + } 76 + ``` 77 + 78 + **Step 4: Add source fields to TimelineItem struct** 79 + 80 + Add source fields to `TimelineItem` (after `ContentType`, line 65): 81 + 82 + ```go 83 + type TimelineItem struct { 84 + Type string `json:"type"` 85 + ID int `json:"id"` 86 + Timestamp time.Time `json:"timestamp"` 87 + Title string `json:"title"` 88 + URL string `json:"url"` 89 + Content string `json:"content"` 90 + Author string `json:"author"` 91 + MD5Sum string `json:"md5sum"` 92 + ContentType string `json:"contentType" gorm:"column:content_type"` 93 + SourceType *string `json:"source_type,omitempty"` 94 + SourceNetwork *string `json:"source_network,omitempty"` 95 + SourceChannel *string `json:"source_channel,omitempty"` 96 + SourceUserID *string `json:"source_user_id,omitempty"` 97 + SourceUserName *string `json:"source_user_name,omitempty"` 98 + } 99 + ``` 100 + 101 + **Step 5: Add SourceFilter type** 102 + 103 + Add a new type after the `TimelineItem` struct: 104 + 105 + ```go 106 + // SourceFilter is used to filter queries by source metadata. 107 + // When all fields are nil, no source filtering is applied. 108 + type SourceFilter struct { 109 + SourceType *string 110 + SourceNetwork *string 111 + SourceChannel *string 112 + } 113 + ``` 114 + 115 + **Step 6: Run tests to verify nothing broke** 116 + 117 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test` 118 + Expected: All existing tests pass (struct additions are backward compatible) 119 + 120 + **Step 7: Commit** 121 + 122 + ```bash 123 + git add internal/data/store.go 124 + git commit -m "feat: add source metadata fields to data models" 125 + ``` 126 + 127 + --- 128 + 129 + ### Task 2: Update Store interface and GormStore for source-aware queries 130 + 131 + **Files:** 132 + - Modify: `internal/data/store.go` (interface) 133 + - Modify: `internal/data/gorm_store.go` (implementation) 134 + 135 + **Step 1: Write failing test for source-scoped InsertIRCLink** 136 + 137 + Create test in a new file `internal/data/source_filter_test.go`: 138 + 139 + ```go 140 + package data 141 + 142 + import ( 143 + "testing" 144 + ) 145 + 146 + func TestSourceFilter_IsEmpty(t *testing.T) { 147 + tests := []struct { 148 + name string 149 + filter SourceFilter 150 + expected bool 151 + }{ 152 + {"all nil", SourceFilter{}, true}, 153 + {"type set", SourceFilter{SourceType: strPtr("irc")}, false}, 154 + {"network set", SourceFilter{SourceNetwork: strPtr("server")}, false}, 155 + {"channel set", SourceFilter{SourceChannel: strPtr("#chan")}, false}, 156 + {"all set", SourceFilter{ 157 + SourceType: strPtr("irc"), 158 + SourceNetwork: strPtr("server"), 159 + SourceChannel: strPtr("#chan"), 160 + }, false}, 161 + } 162 + 163 + for _, tt := range tests { 164 + t.Run(tt.name, func(t *testing.T) { 165 + result := tt.filter.IsEmpty() 166 + if result != tt.expected { 167 + t.Errorf("expected %v, got %v", tt.expected, result) 168 + } 169 + }) 170 + } 171 + } 172 + 173 + func strPtr(s string) *string { 174 + return &s 175 + } 176 + ``` 177 + 178 + **Step 2: Run test to verify it fails** 179 + 180 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/data/ -run TestSourceFilter` 181 + Expected: FAIL — `SourceFilter` has no `IsEmpty` method 182 + 183 + **Step 3: Implement IsEmpty on SourceFilter** 184 + 185 + In `internal/data/store.go`, add method after the `SourceFilter` struct: 186 + 187 + ```go 188 + // IsEmpty returns true if no source filter fields are set. 189 + func (f SourceFilter) IsEmpty() bool { 190 + return f.SourceType == nil && f.SourceNetwork == nil && f.SourceChannel == nil 191 + } 192 + ``` 193 + 194 + **Step 4: Run test to verify it passes** 195 + 196 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/data/ -run TestSourceFilter` 197 + Expected: PASS 198 + 199 + **Step 5: Update Store interface signatures** 200 + 201 + In `internal/data/store.go`, change these method signatures: 202 + 203 + Old: 204 + ```go 205 + InsertIRCLink(ctx context.Context, user, title, url, contentType string) (int, error) 206 + GetIRCLinksByURL(ctx context.Context, url string) ([]IRCLink, error) 207 + GetRecentIRCLinks(ctx context.Context, days int, offsetDays int) ([]IRCLink, error) 208 + GetRecentQuotes(ctx context.Context, days int, offsetDays int) ([]Quote, error) 209 + GetRecentImages(ctx context.Context, days int, offsetDays int) ([]Image, error) 210 + InsertQuote(ctx context.Context, quote, author, poster string) (int, error) 211 + SearchIRCLinks(ctx context.Context, query string) ([]IRCLink, error) 212 + SearchQuotes(ctx context.Context, query string) ([]Quote, error) 213 + ``` 214 + 215 + New: 216 + ```go 217 + InsertIRCLink(ctx context.Context, link *IRCLink) (int, error) 218 + GetIRCLinksByURL(ctx context.Context, url string, filter SourceFilter) ([]IRCLink, error) 219 + GetRecentIRCLinks(ctx context.Context, days int, offsetDays int, filter SourceFilter) ([]IRCLink, error) 220 + GetRecentQuotes(ctx context.Context, days int, offsetDays int, filter SourceFilter) ([]Quote, error) 221 + GetRecentImages(ctx context.Context, days int, offsetDays int, filter SourceFilter) ([]Image, error) 222 + InsertQuote(ctx context.Context, quote *Quote) (int, error) 223 + InsertImage(ctx context.Context, image *Image) (int, error) 224 + SearchIRCLinks(ctx context.Context, query string, filter SourceFilter) ([]IRCLink, error) 225 + SearchQuotes(ctx context.Context, query string, filter SourceFilter) ([]Quote, error) 226 + ``` 227 + 228 + **Step 6: Update GormStore.InsertIRCLink** 229 + 230 + In `internal/data/gorm_store.go`, change `InsertIRCLink` (line 204): 231 + 232 + Old: 233 + ```go 234 + func (s *GormStore) InsertIRCLink(ctx context.Context, user, title, url, contentType string) (int, error) { 235 + link := IRCLink{ 236 + User: user, 237 + Title: title, 238 + URL: url, 239 + ContentType: contentType, 240 + Timestamp: time.Now(), 241 + Clicks: 0, 242 + } 243 + err := s.db.WithContext(ctx).Create(&link).Error 244 + return link.ID, err 245 + } 246 + ``` 247 + 248 + New: 249 + ```go 250 + func (s *GormStore) InsertIRCLink(ctx context.Context, link *IRCLink) (int, error) { 251 + link.Timestamp = time.Now() 252 + link.Clicks = 0 253 + err := s.db.WithContext(ctx).Create(link).Error 254 + return link.ID, err 255 + } 256 + ``` 257 + 258 + **Step 7: Update GormStore.InsertQuote** 259 + 260 + Find the `InsertQuote` method in `gorm_store.go` and update similarly: 261 + 262 + Old: 263 + ```go 264 + func (s *GormStore) InsertQuote(ctx context.Context, quote, author, poster string) (int, error) { 265 + q := Quote{ 266 + Quote: quote, 267 + Author: author, 268 + Poster: poster, 269 + Timestamp: time.Now(), 270 + } 271 + err := s.db.WithContext(ctx).Create(&q).Error 272 + return q.ID, err 273 + } 274 + ``` 275 + 276 + New: 277 + ```go 278 + func (s *GormStore) InsertQuote(ctx context.Context, quote *Quote) (int, error) { 279 + quote.Timestamp = time.Now() 280 + err := s.db.WithContext(ctx).Create(quote).Error 281 + return quote.ID, err 282 + } 283 + ``` 284 + 285 + **Step 8: Update GormStore.InsertImage** 286 + 287 + Old: 288 + ```go 289 + func (s *GormStore) InsertImage(ctx context.Context, title, link, url string) (int, error) { 290 + img := Image{ 291 + Title: title, 292 + Link: link, 293 + URL: url, 294 + Timestamp: time.Now(), 295 + } 296 + err := s.db.WithContext(ctx).Create(&img).Error 297 + return img.ID, err 298 + } 299 + ``` 300 + 301 + New: 302 + ```go 303 + func (s *GormStore) InsertImage(ctx context.Context, image *Image) (int, error) { 304 + image.Timestamp = time.Now() 305 + err := s.db.WithContext(ctx).Create(image).Error 306 + return image.ID, err 307 + } 308 + ``` 309 + 310 + **Step 9: Update GormStore.GetIRCLinksByURL for source-scoped duplicate detection** 311 + 312 + In `gorm_store.go` (line 191): 313 + 314 + Old: 315 + ```go 316 + func (s *GormStore) GetIRCLinksByURL(ctx context.Context, url string) ([]IRCLink, error) { 317 + var links []IRCLink 318 + err := s.db.WithContext(ctx). 319 + Where("url = ?", url). 320 + Order("timestamp DESC"). 321 + Find(&links).Error 322 + return links, err 323 + } 324 + ``` 325 + 326 + New: 327 + ```go 328 + func (s *GormStore) GetIRCLinksByURL(ctx context.Context, url string, filter SourceFilter) ([]IRCLink, error) { 329 + var links []IRCLink 330 + query := s.db.WithContext(ctx).Where("url = ?", url) 331 + query = applySourceFilter(query, filter) 332 + err := query.Order("timestamp DESC").Find(&links).Error 333 + return links, err 334 + } 335 + ``` 336 + 337 + **Step 10: Update GormStore.GetRecentIRCLinks for source filtering** 338 + 339 + In `gorm_store.go` (line 32): 340 + 341 + Old: 342 + ```go 343 + func (s *GormStore) GetRecentIRCLinks(ctx context.Context, startDays int, endDays int) ([]IRCLink, error) { 344 + var links []IRCLink 345 + now := time.Now() 346 + startDate := now.AddDate(0, 0, -startDays) 347 + endDate := now.AddDate(0, 0, -endDays) 348 + 349 + err := s.db.WithContext(ctx). 350 + Where("timestamp >= ? AND timestamp <= ?", startDate, endDate). 351 + Order("timestamp DESC"). 352 + Find(&links).Error 353 + return links, err 354 + } 355 + ``` 356 + 357 + New: 358 + ```go 359 + func (s *GormStore) GetRecentIRCLinks(ctx context.Context, startDays int, endDays int, filter SourceFilter) ([]IRCLink, error) { 360 + var links []IRCLink 361 + now := time.Now() 362 + startDate := now.AddDate(0, 0, -startDays) 363 + endDate := now.AddDate(0, 0, -endDays) 364 + 365 + query := s.db.WithContext(ctx). 366 + Where("timestamp >= ? AND timestamp <= ?", startDate, endDate) 367 + query = applySourceFilter(query, filter) 368 + err := query.Order("timestamp DESC").Find(&links).Error 369 + return links, err 370 + } 371 + ``` 372 + 373 + **Step 11: Apply same pattern to GetRecentQuotes, GetRecentImages, SearchIRCLinks, SearchQuotes** 374 + 375 + Add `filter SourceFilter` parameter and `applySourceFilter(query, filter)` call to each. Follow exact same pattern as Step 10. 376 + 377 + **Step 12: Add applySourceFilter helper** 378 + 379 + Add this function to `gorm_store.go`: 380 + 381 + ```go 382 + // applySourceFilter adds WHERE clauses for source metadata fields. 383 + // When filter is empty, no clauses are added (backward compatible). 384 + func applySourceFilter(query *gorm.DB, filter SourceFilter) *gorm.DB { 385 + if filter.SourceType != nil { 386 + query = query.Where("source_type = ?", *filter.SourceType) 387 + } 388 + if filter.SourceNetwork != nil { 389 + query = query.Where("source_network = ?", *filter.SourceNetwork) 390 + } 391 + if filter.SourceChannel != nil { 392 + query = query.Where("source_channel = ?", *filter.SourceChannel) 393 + } 394 + return query 395 + } 396 + ``` 397 + 398 + **Step 13: Update all other callers of changed methods** 399 + 400 + Search the codebase for all calls to `InsertIRCLink`, `InsertQuote`, `InsertImage`, `GetIRCLinksByURL`, `GetRecentIRCLinks`, `GetRecentQuotes`, `GetRecentImages`, `SearchIRCLinks`, `SearchQuotes`. Update each call site to pass the new parameters. For existing callers that don't have source context, pass `data.SourceFilter{}` (empty filter). 401 + 402 + Key callers to update: 403 + - `internal/handler/api_v1_links.go` — `apiV1CreateLink`, `apiV1ListLinks` 404 + - `internal/handler/api_v1_quotes.go` — `apiV1CreateQuote`, `apiV1ListQuotes` 405 + - `internal/handler/api_v1_search.go` — `APIv1SearchHandler` 406 + - `internal/handler/irclink.go` — legacy handler (if it exists) 407 + - `internal/handler/handlers.go` — any frontend handlers calling these 408 + - `internal/scheduler/` — any background jobs 409 + - All test mock implementations 410 + 411 + **Step 14: Update mock stores in test files** 412 + 413 + Update `mockAPIStore` in `internal/handler/api_v1_links_test.go` to match new signatures: 414 + 415 + ```go 416 + func (m *mockAPIStore) GetRecentIRCLinks(ctx context.Context, days int, offsetDays int, filter data.SourceFilter) ([]data.IRCLink, error) { 417 + if m.err != nil { 418 + return nil, m.err 419 + } 420 + return m.links, nil 421 + } 422 + 423 + func (m *mockAPIStore) GetIRCLinksByURL(ctx context.Context, url string, filter data.SourceFilter) ([]data.IRCLink, error) { 424 + if m.linksByURLFn != nil { 425 + return m.linksByURLFn(url) 426 + } 427 + if m.err != nil { 428 + return nil, m.err 429 + } 430 + return m.linksByURL, nil 431 + } 432 + 433 + func (m *mockAPIStore) InsertIRCLink(ctx context.Context, link *data.IRCLink) (int, error) { 434 + if m.insertLinkFn != nil { 435 + return m.insertLinkFn(link.User, link.Title, link.URL, link.ContentType) 436 + } 437 + if m.err != nil { 438 + return 0, m.err 439 + } 440 + return m.insertedLinkID, nil 441 + } 442 + ``` 443 + 444 + Do the same for `mockQuoteStore` in `api_v1_quotes_test.go` and any other mock stores. 445 + 446 + **Step 15: Run tests** 447 + 448 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test` 449 + Expected: All tests pass. Compilation succeeds with updated signatures. 450 + 451 + **Step 16: Commit** 452 + 453 + ```bash 454 + git add internal/data/store.go internal/data/gorm_store.go internal/data/source_filter_test.go 455 + git add internal/handler/ 456 + git commit -m "feat: update Store interface for source-aware queries 457 + 458 + Change Insert methods to accept struct pointers instead of 459 + individual parameters. Add SourceFilter to Get and Search 460 + methods. Add applySourceFilter helper for GORM queries." 461 + ``` 462 + 463 + --- 464 + 465 + ### Task 3: Update API handlers to accept and return source fields 466 + 467 + **Files:** 468 + - Modify: `internal/handler/api_v1_types.go` 469 + - Modify: `internal/handler/api_v1_links.go` 470 + - Modify: `internal/handler/api_v1_quotes.go` 471 + - Modify: `internal/handler/api_v1_search.go` 472 + 473 + **Step 1: Write failing test for source fields in link creation** 474 + 475 + Add to `internal/handler/api_v1_links_test.go`, in the `TestAPIv1_CreateLink` function's test table: 476 + 477 + ```go 478 + { 479 + name: "valid link with source fields", 480 + body: `{"url":"https://example.com","user":"testuser","source_type":"slack","source_network":"T12345","source_channel":"C67890","source_user_id":"U99999","source_user_name":"testuser"}`, 481 + store: &mockAPIStore{insertedLinkID: 42}, 482 + expectedStatus: http.StatusCreated, 483 + checkBody: func(t *testing.T, body []byte) { 484 + var resp APILinkCreateResponse 485 + if err := json.Unmarshal(body, &resp); err != nil { 486 + t.Fatalf("failed to unmarshal: %v", err) 487 + } 488 + if resp.SourceType == nil || *resp.SourceType != "slack" { 489 + t.Errorf("expected source_type 'slack', got %v", resp.SourceType) 490 + } 491 + if resp.SourceNetwork == nil || *resp.SourceNetwork != "T12345" { 492 + t.Errorf("expected source_network 'T12345', got %v", resp.SourceNetwork) 493 + } 494 + }, 495 + }, 496 + ``` 497 + 498 + **Step 2: Run test to verify it fails** 499 + 500 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/handler/ -run TestAPIv1_CreateLink/valid_link_with_source_fields` 501 + Expected: FAIL — `APILinkCreateResponse` has no `SourceType` field 502 + 503 + **Step 3: Add source fields to API response types** 504 + 505 + In `internal/handler/api_v1_types.go`, add source fields to `APILinkResponse`: 506 + 507 + ```go 508 + type APILinkResponse struct { 509 + ID int `json:"id"` 510 + URL string `json:"url"` 511 + Title string `json:"title"` 512 + User string `json:"user"` 513 + Clicks int `json:"clicks"` 514 + CreatedAt time.Time `json:"created_at"` 515 + Tags []string `json:"tags,omitempty"` 516 + SourceType *string `json:"source_type,omitempty"` 517 + SourceNetwork *string `json:"source_network,omitempty"` 518 + SourceChannel *string `json:"source_channel,omitempty"` 519 + SourceUserID *string `json:"source_user_id,omitempty"` 520 + SourceUserName *string `json:"source_user_name,omitempty"` 521 + } 522 + ``` 523 + 524 + Add source fields to `APIQuoteResponse`: 525 + 526 + ```go 527 + type APIQuoteResponse struct { 528 + ID int `json:"id"` 529 + Quote string `json:"quote"` 530 + Author string `json:"author"` 531 + Poster string `json:"poster,omitempty"` 532 + CreatedAt time.Time `json:"created_at"` 533 + Tags []string `json:"tags,omitempty"` 534 + SourceType *string `json:"source_type,omitempty"` 535 + SourceNetwork *string `json:"source_network,omitempty"` 536 + SourceChannel *string `json:"source_channel,omitempty"` 537 + SourceUserID *string `json:"source_user_id,omitempty"` 538 + SourceUserName *string `json:"source_user_name,omitempty"` 539 + } 540 + ``` 541 + 542 + **Step 4: Add source fields to request types** 543 + 544 + In `internal/handler/api_v1_links.go`, update `APILinkCreateRequest`: 545 + 546 + ```go 547 + type APILinkCreateRequest struct { 548 + URL string `json:"url"` 549 + User string `json:"user"` 550 + Tags []string `json:"tags,omitempty"` 551 + SourceType *string `json:"source_type,omitempty"` 552 + SourceNetwork *string `json:"source_network,omitempty"` 553 + SourceChannel *string `json:"source_channel,omitempty"` 554 + SourceUserID *string `json:"source_user_id,omitempty"` 555 + SourceUserName *string `json:"source_user_name,omitempty"` 556 + } 557 + ``` 558 + 559 + In `internal/handler/api_v1_quotes.go`, update `APIQuoteCreateRequest`: 560 + 561 + ```go 562 + type APIQuoteCreateRequest struct { 563 + Quote string `json:"quote"` 564 + Author string `json:"author"` 565 + Poster string `json:"poster"` 566 + Tags []string `json:"tags,omitempty"` 567 + SourceType *string `json:"source_type,omitempty"` 568 + SourceNetwork *string `json:"source_network,omitempty"` 569 + SourceChannel *string `json:"source_channel,omitempty"` 570 + SourceUserID *string `json:"source_user_id,omitempty"` 571 + SourceUserName *string `json:"source_user_name,omitempty"` 572 + } 573 + ``` 574 + 575 + **Step 5: Update apiV1CreateLink to pass source fields through** 576 + 577 + In `internal/handler/api_v1_links.go`, update `apiV1CreateLink` to build an `IRCLink` struct and pass source fields: 578 + 579 + ```go 580 + // Build source filter for duplicate detection 581 + sourceFilter := data.SourceFilter{ 582 + SourceType: req.SourceType, 583 + SourceNetwork: req.SourceNetwork, 584 + SourceChannel: req.SourceChannel, 585 + } 586 + 587 + // Check for duplicates (scoped by source when provided) 588 + existingLinks, err := h.Store.GetIRCLinksByURL(ctx, req.URL, sourceFilter) 589 + 590 + // Build the link struct with source metadata 591 + link := &data.IRCLink{ 592 + User: req.User, 593 + Title: req.URL, 594 + URL: req.URL, 595 + ContentType: "", 596 + SourceType: req.SourceType, 597 + SourceNetwork: req.SourceNetwork, 598 + SourceChannel: req.SourceChannel, 599 + SourceUserID: req.SourceUserID, 600 + SourceUserName: req.SourceUserName, 601 + } 602 + linkID, err := h.Store.InsertIRCLink(ctx, link) 603 + ``` 604 + 605 + Update the response builder to include source fields: 606 + 607 + ```go 608 + resp := APILinkCreateResponse{ 609 + APILinkResponse: APILinkResponse{ 610 + ID: linkID, 611 + URL: req.URL, 612 + Title: req.URL, 613 + User: req.User, 614 + Clicks: 0, 615 + CreatedAt: time.Now(), 616 + Tags: tagStrings, 617 + SourceType: req.SourceType, 618 + SourceNetwork: req.SourceNetwork, 619 + SourceChannel: req.SourceChannel, 620 + SourceUserID: req.SourceUserID, 621 + SourceUserName: req.SourceUserName, 622 + }, 623 + IsDuplicate: isDuplicate, 624 + PreviousSubmissions: previousSubmissions, 625 + } 626 + ``` 627 + 628 + **Step 6: Update apiV1CreateQuote similarly** 629 + 630 + Pass source fields through to `InsertQuote` and include in response. 631 + 632 + **Step 7: Update apiV1ListLinks to parse source query params and pass filter** 633 + 634 + In `internal/handler/api_v1_links.go`, in `apiV1ListLinks`: 635 + 636 + ```go 637 + // Parse source filter parameters 638 + var sourceFilter data.SourceFilter 639 + if st := r.URL.Query().Get("source_type"); st != "" { 640 + sourceFilter.SourceType = &st 641 + } 642 + if sn := r.URL.Query().Get("source_network"); sn != "" { 643 + sourceFilter.SourceNetwork = &sn 644 + } 645 + if sc := r.URL.Query().Get("source_channel"); sc != "" { 646 + sourceFilter.SourceChannel = &sc 647 + } 648 + 649 + links, err := h.Store.GetRecentIRCLinks(ctx, 365, 0, sourceFilter) 650 + ``` 651 + 652 + Update the response conversion loop to include source fields from each link: 653 + 654 + ```go 655 + data = append(data, APILinkResponse{ 656 + ID: link.ID, 657 + URL: link.URL, 658 + Title: link.Title, 659 + User: link.User, 660 + Clicks: link.Clicks, 661 + CreatedAt: link.Timestamp, 662 + Tags: h.getTagStrings(ctx, "link", link.ID), 663 + SourceType: link.SourceType, 664 + SourceNetwork: link.SourceNetwork, 665 + SourceChannel: link.SourceChannel, 666 + SourceUserID: link.SourceUserID, 667 + SourceUserName: link.SourceUserName, 668 + }) 669 + ``` 670 + 671 + **Step 8: Update apiV1ListQuotes the same way** 672 + 673 + Parse source query params, pass filter to `GetRecentQuotes`, include source fields in response. 674 + 675 + **Step 9: Update apiV1GetLink and apiV1GetQuote responses** 676 + 677 + Include source fields from the fetched link/quote in the response structs. 678 + 679 + **Step 10: Update APIv1SearchHandler** 680 + 681 + In `internal/handler/api_v1_search.go`, parse source query params and pass filter to `SearchIRCLinks` and `SearchQuotes`. Include source fields in the response conversion loops. 682 + 683 + **Step 11: Run tests** 684 + 685 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test` 686 + Expected: All tests pass including the new source fields test 687 + 688 + **Step 12: Commit** 689 + 690 + ```bash 691 + git add internal/handler/ 692 + git commit -m "feat: accept and return source fields in API endpoints 693 + 694 + POST links/quotes accepts optional source_type, source_network, 695 + source_channel, source_user_id, source_user_name fields. 696 + GET links/quotes/search supports source_type, source_network, 697 + source_channel query parameters for filtering. 698 + All responses include source fields with omitempty." 699 + ``` 700 + 701 + --- 702 + 703 + ### Task 4: Write comprehensive tests for source-scoped duplicate detection 704 + 705 + **Files:** 706 + - Modify: `internal/handler/api_v1_links_test.go` 707 + 708 + **Step 1: Write test cases for scoped duplicate detection** 709 + 710 + Add a new test function `TestAPIv1_CreateLink_SourceDuplicates`: 711 + 712 + ```go 713 + func TestAPIv1_CreateLink_SourceDuplicates(t *testing.T) { 714 + now := time.Now() 715 + 716 + tests := []struct { 717 + name string 718 + body string 719 + existingLinks []data.IRCLink 720 + expectedStatus int 721 + expectedDuplicate bool 722 + }{ 723 + { 724 + name: "same URL different source is not duplicate", 725 + body: `{"url":"https://example.com","user":"alice","source_type":"slack","source_network":"T111","source_channel":"C222"}`, 726 + existingLinks: nil, // source-scoped query returns nothing 727 + expectedStatus: http.StatusCreated, 728 + expectedDuplicate: false, 729 + }, 730 + { 731 + name: "same URL same source is duplicate", 732 + body: `{"url":"https://example.com","user":"bob","source_type":"slack","source_network":"T111","source_channel":"C222"}`, 733 + existingLinks: []data.IRCLink{ 734 + {ID: 10, User: "alice", URL: "https://example.com", Timestamp: now}, 735 + }, 736 + expectedStatus: http.StatusCreated, 737 + expectedDuplicate: true, 738 + }, 739 + { 740 + name: "no source fields uses global duplicate check", 741 + body: `{"url":"https://example.com","user":"charlie"}`, 742 + existingLinks: []data.IRCLink{ 743 + {ID: 10, User: "alice", URL: "https://example.com", Timestamp: now}, 744 + }, 745 + expectedStatus: http.StatusCreated, 746 + expectedDuplicate: true, 747 + }, 748 + } 749 + 750 + for _, tt := range tests { 751 + t.Run(tt.name, func(t *testing.T) { 752 + store := &mockAPIStore{ 753 + linksByURL: tt.existingLinks, 754 + insertedLinkID: 42, 755 + } 756 + handler := NewHandler(store, &config.Config{}) 757 + req := httptest.NewRequest(http.MethodPost, "/api/v1/links", strings.NewReader(tt.body)) 758 + req.RemoteAddr = "127.0.0.1:12345" 759 + w := httptest.NewRecorder() 760 + 761 + handler.APIv1LinksHandler(w, req) 762 + 763 + if w.Code != tt.expectedStatus { 764 + t.Errorf("expected status %d, got %d", tt.expectedStatus, w.Code) 765 + } 766 + 767 + var resp APILinkCreateResponse 768 + if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { 769 + t.Fatalf("failed to unmarshal: %v", err) 770 + } 771 + if resp.IsDuplicate != tt.expectedDuplicate { 772 + t.Errorf("expected is_duplicate=%v, got %v", tt.expectedDuplicate, resp.IsDuplicate) 773 + } 774 + }) 775 + } 776 + } 777 + ``` 778 + 779 + **Step 2: Run tests** 780 + 781 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && go test -v ./internal/handler/ -run TestAPIv1_CreateLink_SourceDuplicates` 782 + Expected: PASS 783 + 784 + **Step 3: Write test for source filtering on GET** 785 + 786 + Add `TestAPIv1_ListLinks_SourceFiltering`: 787 + 788 + ```go 789 + func TestAPIv1_ListLinks_SourceFiltering(t *testing.T) { 790 + slackType := "slack" 791 + slackNetwork := "T12345" 792 + slackChannel := "C67890" 793 + 794 + tests := []struct { 795 + name string 796 + queryParams string 797 + links []data.IRCLink 798 + expectedTotal int 799 + }{ 800 + { 801 + name: "no filter returns all", 802 + queryParams: "", 803 + links: []data.IRCLink{ 804 + {ID: 1, User: "alice", URL: "https://a.com", Timestamp: time.Now()}, 805 + {ID: 2, User: "bob", URL: "https://b.com", Timestamp: time.Now(), SourceType: &slackType}, 806 + }, 807 + expectedTotal: 2, 808 + }, 809 + { 810 + name: "filter by source_type", 811 + queryParams: "?source_type=slack", 812 + links: []data.IRCLink{ 813 + {ID: 2, User: "bob", URL: "https://b.com", Timestamp: time.Now(), SourceType: &slackType}, 814 + }, 815 + expectedTotal: 1, 816 + }, 817 + } 818 + 819 + for _, tt := range tests { 820 + t.Run(tt.name, func(t *testing.T) { 821 + store := &mockAPIStore{links: tt.links} 822 + handler := NewHandler(store, &config.Config{}) 823 + req := httptest.NewRequest(http.MethodGet, "/api/v1/links"+tt.queryParams, nil) 824 + w := httptest.NewRecorder() 825 + 826 + handler.APIv1LinksHandler(w, req) 827 + 828 + var resp APILinksResponse 829 + if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { 830 + t.Fatalf("failed to unmarshal: %v", err) 831 + } 832 + if resp.Meta.Total != tt.expectedTotal { 833 + t.Errorf("expected total %d, got %d", tt.expectedTotal, resp.Meta.Total) 834 + } 835 + }) 836 + } 837 + } 838 + ``` 839 + 840 + **Step 4: Write test for omitempty serialization** 841 + 842 + Add `TestAPIv1_LinkResponse_SourceOmitEmpty`: 843 + 844 + ```go 845 + func TestAPIv1_LinkResponse_SourceOmitEmpty(t *testing.T) { 846 + t.Run("null source fields omitted from JSON", func(t *testing.T) { 847 + store := &mockAPIStore{ 848 + links: []data.IRCLink{ 849 + {ID: 1, User: "alice", URL: "https://a.com", Timestamp: time.Now()}, 850 + }, 851 + } 852 + handler := NewHandler(store, &config.Config{}) 853 + req := httptest.NewRequest(http.MethodGet, "/api/v1/links", nil) 854 + w := httptest.NewRecorder() 855 + 856 + handler.APIv1LinksHandler(w, req) 857 + 858 + body := w.Body.String() 859 + if strings.Contains(body, "source_type") { 860 + t.Error("expected source_type to be omitted when null") 861 + } 862 + }) 863 + 864 + t.Run("source fields present when set", func(t *testing.T) { 865 + slackType := "slack" 866 + store := &mockAPIStore{ 867 + links: []data.IRCLink{ 868 + {ID: 1, User: "alice", URL: "https://a.com", Timestamp: time.Now(), SourceType: &slackType}, 869 + }, 870 + } 871 + handler := NewHandler(store, &config.Config{}) 872 + req := httptest.NewRequest(http.MethodGet, "/api/v1/links", nil) 873 + w := httptest.NewRecorder() 874 + 875 + handler.APIv1LinksHandler(w, req) 876 + 877 + body := w.Body.String() 878 + if !strings.Contains(body, `"source_type":"slack"`) { 879 + t.Errorf("expected source_type in response, got: %s", body) 880 + } 881 + }) 882 + } 883 + ``` 884 + 885 + **Step 5: Run all tests** 886 + 887 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test` 888 + Expected: All tests pass 889 + 890 + **Step 6: Commit** 891 + 892 + ```bash 893 + git add internal/handler/api_v1_links_test.go 894 + git commit -m "test: add tests for source-scoped duplicate detection and filtering" 895 + ``` 896 + 897 + --- 898 + 899 + ### Task 5: Write backfill SQL script 900 + 901 + **Files:** 902 + - Create: `sql/backfill_sources.sql` 903 + 904 + **Step 1: Create the backfill script** 905 + 906 + ```sql 907 + -- One-time backfill: set source metadata on all existing rows. 908 + -- All existing data originates from IRC, #soggies channel on jameswhite.org. 909 + -- Run this manually after deploying the source fields migration. 910 + -- 911 + -- Usage (SQLite): sqlite3 tumble.db < sql/backfill_sources.sql 912 + -- Usage (MySQL): mysql -u user -p tumble < sql/backfill_sources.sql 913 + 914 + UPDATE ircLink 915 + SET source_type = 'irc', 916 + source_network = 'jameswhite.org', 917 + source_channel = '#soggies' 918 + WHERE source_type IS NULL; 919 + 920 + UPDATE quote 921 + SET source_type = 'irc', 922 + source_network = 'jameswhite.org', 923 + source_channel = '#soggies' 924 + WHERE source_type IS NULL; 925 + 926 + UPDATE image 927 + SET source_type = 'irc', 928 + source_network = 'jameswhite.org', 929 + source_channel = '#soggies' 930 + WHERE source_type IS NULL; 931 + ``` 932 + 933 + **Step 2: Commit** 934 + 935 + ```bash 936 + git add sql/backfill_sources.sql 937 + git commit -m "feat: add one-time backfill script for source metadata" 938 + ``` 939 + 940 + --- 941 + 942 + ### Task 6: Update OpenAPI specification 943 + 944 + **Files:** 945 + - Modify: `internal/assets/openapi.json` 946 + 947 + **Step 1: Add source fields to LinkCreateRequest schema** 948 + 949 + Find the `LinkCreateRequest` schema in `openapi.json` and add: 950 + 951 + ```json 952 + "source_type": { 953 + "type": "string", 954 + "description": "Source platform (e.g., irc, slack, discord, api, web)", 955 + "example": "slack" 956 + }, 957 + "source_network": { 958 + "type": "string", 959 + "description": "Source network identifier (e.g., IRC server, Slack team ID)", 960 + "example": "T12345" 961 + }, 962 + "source_channel": { 963 + "type": "string", 964 + "description": "Source channel identifier (e.g., IRC channel, Slack channel ID)", 965 + "example": "C67890" 966 + }, 967 + "source_user_id": { 968 + "type": "string", 969 + "description": "Platform-specific user ID", 970 + "example": "U99999" 971 + }, 972 + "source_user_name": { 973 + "type": "string", 974 + "description": "Display/mention name at time of post", 975 + "example": "stahnma" 976 + } 977 + ``` 978 + 979 + **Step 2: Add same fields to QuoteCreateRequest schema** 980 + 981 + Same five fields. 982 + 983 + **Step 3: Add source fields to APILinkResponse and APIQuoteResponse schemas** 984 + 985 + Same five fields added to response schemas. 986 + 987 + **Step 4: Add source query parameters to GET /api/v1/links** 988 + 989 + Add three optional query parameters: 990 + 991 + ```json 992 + { 993 + "name": "source_type", 994 + "in": "query", 995 + "required": false, 996 + "schema": { "type": "string" }, 997 + "description": "Filter by source platform (e.g., irc, slack, discord)" 998 + }, 999 + { 1000 + "name": "source_network", 1001 + "in": "query", 1002 + "required": false, 1003 + "schema": { "type": "string" }, 1004 + "description": "Filter by source network (requires source_type)" 1005 + }, 1006 + { 1007 + "name": "source_channel", 1008 + "in": "query", 1009 + "required": false, 1010 + "schema": { "type": "string" }, 1011 + "description": "Filter by source channel (requires source_type and source_network)" 1012 + } 1013 + ``` 1014 + 1015 + **Step 5: Add same query parameters to GET /api/v1/quotes and GET /api/v1/search** 1016 + 1017 + **Step 6: Update 208 response description** 1018 + 1019 + Update the duplicate detection documentation for POST /api/v1/links to note that duplicate detection is scoped per source when source fields are provided. 1020 + 1021 + **Step 7: Run the app to verify docs render** 1022 + 1023 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make restart` 1024 + Visit: `http://localhost:8080/api/docs` and verify the new fields appear. 1025 + Then: `make kill` 1026 + 1027 + **Step 8: Commit** 1028 + 1029 + ```bash 1030 + git add internal/assets/openapi.json 1031 + git commit -m "docs: update OpenAPI spec with source metadata fields 1032 + 1033 + Add source_type, source_network, source_channel, source_user_id, 1034 + and source_user_name to request/response schemas. Add source 1035 + filter query parameters to GET endpoints." 1036 + ``` 1037 + 1038 + --- 1039 + 1040 + ### Task 7: Final integration verification 1041 + 1042 + **Step 1: Run full test suite** 1043 + 1044 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test` 1045 + Expected: All tests pass 1046 + 1047 + **Step 2: Run API integration tests** 1048 + 1049 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make test-api` 1050 + Expected: All integration tests pass 1051 + 1052 + **Step 3: Manual smoke test** 1053 + 1054 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && make restart` 1055 + 1056 + Test creating a link with source fields: 1057 + ```bash 1058 + curl -s -X POST http://localhost:8080/api/v1/links \ 1059 + -H "Content-Type: application/json" \ 1060 + -d '{"url":"https://example.com/test","user":"testuser","source_type":"slack","source_network":"T12345","source_channel":"C67890","source_user_id":"U99999","source_user_name":"testuser"}' | jq . 1061 + ``` 1062 + 1063 + Verify source fields in response. Then test filtering: 1064 + ```bash 1065 + curl -s "http://localhost:8080/api/v1/links?source_type=slack" | jq . 1066 + ``` 1067 + 1068 + Test that creating same URL without source fields still works: 1069 + ```bash 1070 + curl -s -X POST http://localhost:8080/api/v1/links \ 1071 + -H "Content-Type: application/json" \ 1072 + -d '{"url":"https://example.com/test","user":"otheruser"}' | jq . 1073 + ``` 1074 + 1075 + Then: `make kill` 1076 + 1077 + **Step 4: Run go fmt** 1078 + 1079 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && go fmt ./...` 1080 + 1081 + **Step 5: Check for trailing whitespace** 1082 + 1083 + Run: `cd /Users/stahnma/development/personal/tumble/tumble && git diff --check` 1084 + 1085 + **Step 6: Final commit if any formatting fixes needed** 1086 + 1087 + ```bash 1088 + git add -A 1089 + git commit -m "chore: format and clean up" 1090 + ```