bring back yahoo pipes!
1# Crush Memory - Pipes Project
2
3## User Preferences
4
5- Follow Herald's Go architecture patterns
6- Use direct SQL for all database operations (no ORM)
7- Follow neo-brutalist design aesthetic (blue/orange for app, pink only for auth)
8- Use Indiko for authentication (OAuth 2.0 with PKCE)
9- Run on port 3001 (Indiko runs on 3000)
10- Use charmbracelet/log for structured logging
11
12## Architecture Patterns
13
14### Authentication Flow
15- OAuth 2.0 client that uses Indiko for authentication
16- PKCE flow with code verifier/challenge (required by IndieAuth spec)
17- Auto-registration with Indiko (client ID = app URL)
18- Session-based auth with 30-day cookies
19- Automatic token refresh using refresh tokens
20
21### Database (SQLite)
22- Direct SQL queries (no ORM like Kysely or GORM)
23- SQLite with WAL mode for concurrency
24- Foreign key constraints enabled
25- Schema created automatically on startup
26
27### Project Structure
28```
29pipes/
30├── main.go # Entry point, CLI setup
31├── go.mod # Dependencies
32├── config/
33│ ├── app.go # Config struct & loading (like Herald)
34│ └── validate.go # Config validation
35├── store/
36│ ├── db.go # SQLite setup & schema
37│ ├── users.go # User operations
38│ ├── pipes.go # Pipe CRUD
39│ ├── executions.go # Execution history
40│ └── cache.go # Source cache operations
41├── auth/
42│ ├── oauth.go # OAuth 2.0 client (Indiko)
43│ ├── session.go # Session management
44│ └── middleware.go # Auth middleware
45├── engine/
46│ ├── executor.go # Pipeline execution engine
47│ ├── scheduler.go # Cron-based scheduler (Herald pattern)
48│ └── registry.go # Node type registry
49├── nodes/
50│ ├── node.go # Node interface
51│ ├── sources/
52│ │ ├── rss.go # RSS/Atom source
53│ │ └── http.go # HTTP API source
54│ └── transforms/
55│ ├── filter.go # Filter transform
56│ ├── sort.go # Sort transform
57│ ├── limit.go # Limit transform
58│ ├── merge.go # Merge sources
59│ ├── dedupe.go # Deduplicate items
60│ └── extract.go # Extract/transform fields
61└── web/
62 ├── server.go # HTTP server setup
63 ├── handlers.go # Route handlers
64 ├── api.go # JSON API endpoints
65 └── templates/
66 ├── layout.html # Base layout
67 ├── index.html # Landing page
68 ├── dashboard.html # User dashboard
69 ├── editor.html # Visual editor
70 └── style.css # Frutiger Aero styles
71```
72
73### Database Schema
74- **users**: id, indiko_sub, username, name, email, photo, url, role, created_at, updated_at
75 - `indiko_sub` is the "sub" field from Indiko's userinfo endpoint
76 - `role` is either "user" or "admin" (synced from Indiko)
77- **sessions**: id, user_id, access_token, refresh_token, expires_at, created_at
78 - 30-day sessions with automatic token refresh
79- **pipes**: id, user_id, name, description, config (JSON), is_public, created_at, updated_at
80 - `config` is JSON: {version, nodes[], connections[], settings}
81- **scheduled_jobs**: id, pipe_id, cron_expression, next_run_at, last_run_at, enabled, created_at, updated_at
82- **pipe_executions**: id, pipe_id, status, trigger_type, started_at, completed_at, duration_ms, items_processed, error_message, metadata
83- **execution_logs**: id, execution_id, node_id, level, message, timestamp, metadata
84- **source_cache**: id, pipe_id, node_id, cache_key, data, etag, last_modified, expires_at, created_at
85
86### OAuth Configuration
87- **Client ID**: App's URL (e.g., `http://localhost:3001`)
88- **Client Secret**: Optional, for pre-registered clients (stored in .env)
89- **Scopes**: `profile email`
90- **PKCE**: Required (S256 code challenge method)
91- **Callback URL**: `{ORIGIN}/auth/callback`
92
93## Configuration
94
95### YAML Config (config.yaml)
96Contains all non-sensitive configuration:
97```yaml
98# Server settings
99host: localhost
100port: 3001
101origin: http://localhost:3001
102env: development
103log_level: info # debug, info, warn, error, fatal
104
105# Database
106db_path: pipes.db
107
108# OAuth (Indiko)
109indiko_url: http://localhost:3000
110indiko_client_id: http://localhost:3001
111indiko_client_secret: ${INDIKO_CLIENT_SECRET} # Loaded from .env
112oauth_callback_url: http://localhost:3001/auth/callback
113
114# Session
115session_secret: ${SESSION_SECRET} # Loaded from .env
116session_cookie_name: pipes_session
117```
118
119### Environment Variables (.env)
120Contains **only secrets**:
121```env
122# OAuth (Indiko)
123INDIKO_CLIENT_SECRET=your_client_secret_here
124
125# Session (generate with: openssl rand -base64 32)
126SESSION_SECRET=your_random_secret_here
127```
128
129## Routes
130
131### Public
132- `GET /` - Landing page (redirects to /dashboard if authenticated)
133- `GET /auth/login` - Start OAuth flow
134- `GET /auth/callback` - OAuth callback handler
135
136### Authenticated
137- `GET /dashboard` - User dashboard (requires auth)
138- `GET /pipes/:id/edit` - Visual editor (requires auth)
139- `POST /auth/logout` - End session
140- `GET /api/me` - Get current user info
141- `GET /api/pipes` - List user's pipes
142- `GET /api/pipes/:id` - Get pipe config
143- `POST /api/pipes` - Create pipe
144- `PUT /api/pipes/:id` - Update pipe
145- `DELETE /api/pipes/:id` - Delete pipe
146- `POST /api/pipes/:id/execute` - Execute pipe manually
147- `GET /api/pipes/:id/executions` - Execution history
148- `GET /api/executions/:id/logs` - Execution logs
149- `GET /api/node-types` - Available node types
150
151## Code Style
152
153- Use `gofmt` for formatting
154- Structured logging with charmbracelet/log: `logger.Info("message", "key", value)`
155- Error wrapping: `fmt.Errorf("context: %w", err)`
156- Context propagation for cancellation
157- Session cookies named `pipes_session`
158- Authorization header: `Bearer {token}`
159
160## Design Aesthetic
161
162**Neo-Brutalism** - Bold, geometric design:
163- Space Grotesk font family
164- Hard borders (3-4px solid #26242b)
165- Hard box shadows (6-12px offset, no blur)
166- High contrast, sharp edges
167- Uppercase text, tight letter-spacing
168- Color palette:
169 - **App colors**:
170 - Primary: #2563eb (blue) - used for headings, secondary buttons
171 - Secondary: #ff6b35 (orange) - used for primary buttons, accents
172 - Background: #f5f5f0 (warm off-white)
173 - Dark: #26242b (near-black)
174 - White: #fff
175 - **Auth/Indiko colors** (login, error pages):
176 - Auth primary: #AB4967 (muted pink)
177 - Auth text: #fff (white - for text on pink buttons)
178
179## Commands
180
181```bash
182go build -o pipes . # Build
183./pipes serve # Run server
184./pipes init # Initialize config files
185./pipes help # Show help
186./pipes version # Show version
187```