Crush Memory - Pipes Project#

User Preferences#

• Follow Herald's Go architecture patterns
• Use direct SQL for all database operations (no ORM)
• Follow neo-brutalist design aesthetic (blue/orange for app, pink only for auth)
• Use Indiko for authentication (OAuth 2.0 with PKCE)
• Run on port 3001 (Indiko runs on 3000)
• Use charmbracelet/log for structured logging

Architecture Patterns#

Authentication Flow#

• OAuth 2.0 client that uses Indiko for authentication
• PKCE flow with code verifier/challenge (required by IndieAuth spec)
• Auto-registration with Indiko (client ID = app URL)
• Session-based auth with 30-day cookies
• Automatic token refresh using refresh tokens

Database (SQLite)#

• Direct SQL queries (no ORM like Kysely or GORM)
• SQLite with WAL mode for concurrency
• Foreign key constraints enabled
• Schema created automatically on startup

Project Structure#

pipes/
├── main.go                     # Entry point, CLI setup
├── go.mod                      # Dependencies
├── config/
│   ├── app.go                 # Config struct & loading (like Herald)
│   └── validate.go            # Config validation
├── store/
│   ├── db.go                  # SQLite setup & schema
│   ├── users.go               # User operations
│   ├── pipes.go               # Pipe CRUD
│   ├── executions.go          # Execution history
│   └── cache.go               # Source cache operations
├── auth/
│   ├── oauth.go               # OAuth 2.0 client (Indiko)
│   ├── session.go             # Session management
│   └── middleware.go          # Auth middleware
├── engine/
│   ├── executor.go            # Pipeline execution engine
│   ├── scheduler.go           # Cron-based scheduler (Herald pattern)
│   └── registry.go            # Node type registry
├── nodes/
│   ├── node.go                # Node interface
│   ├── sources/
│   │   ├── rss.go            # RSS/Atom source
│   │   └── http.go           # HTTP API source
│   └── transforms/
│       ├── filter.go         # Filter transform
│       ├── sort.go           # Sort transform
│       ├── limit.go          # Limit transform
│       ├── merge.go          # Merge sources
│       ├── dedupe.go         # Deduplicate items
│       └── extract.go        # Extract/transform fields
└── web/
    ├── server.go             # HTTP server setup
    ├── handlers.go           # Route handlers
    ├── api.go                # JSON API endpoints
    └── templates/
        ├── layout.html       # Base layout
        ├── index.html        # Landing page
        ├── dashboard.html    # User dashboard
        ├── editor.html       # Visual editor
        └── style.css         # Frutiger Aero styles

Database Schema#

• users: id, indiko_sub, username, name, email, photo, url, role, created_at, updated_at
  • indiko_sub is the "sub" field from Indiko's userinfo endpoint
  • role is either "user" or "admin" (synced from Indiko)
• sessions: id, user_id, access_token, refresh_token, expires_at, created_at
  • 30-day sessions with automatic token refresh
• pipes: id, user_id, name, description, config (JSON), is_public, created_at, updated_at
  • config is JSON: {version, nodes[], connections[], settings}
• scheduled_jobs: id, pipe_id, cron_expression, next_run_at, last_run_at, enabled, created_at, updated_at
• pipe_executions: id, pipe_id, status, trigger_type, started_at, completed_at, duration_ms, items_processed, error_message, metadata
• execution_logs: id, execution_id, node_id, level, message, timestamp, metadata
• source_cache: id, pipe_id, node_id, cache_key, data, etag, last_modified, expires_at, created_at

OAuth Configuration#

• Client ID: App's URL (e.g., http://localhost:3001)
• Client Secret: Optional, for pre-registered clients (stored in .env)
• Scopes: profile email
• PKCE: Required (S256 code challenge method)
• Callback URL: {ORIGIN}/auth/callback

Configuration#

YAML Config (config.yaml)#

Contains all non-sensitive configuration:

# Server settings
host: localhost
port: 3001
origin: http://localhost:3001
env: development
log_level: info  # debug, info, warn, error, fatal

# Database
db_path: pipes.db

# OAuth (Indiko)
indiko_url: http://localhost:3000
indiko_client_id: http://localhost:3001
indiko_client_secret: ${INDIKO_CLIENT_SECRET}  # Loaded from .env
oauth_callback_url: http://localhost:3001/auth/callback

# Session
session_secret: ${SESSION_SECRET}  # Loaded from .env
session_cookie_name: pipes_session

Environment Variables (.env)#

Contains only secrets:

# OAuth (Indiko)
INDIKO_CLIENT_SECRET=your_client_secret_here

# Session (generate with: openssl rand -base64 32)
SESSION_SECRET=your_random_secret_here

Routes#

Public#

• GET / - Landing page (redirects to /dashboard if authenticated)
• GET /auth/login - Start OAuth flow
• GET /auth/callback - OAuth callback handler

Authenticated#

• GET /dashboard - User dashboard (requires auth)
• GET /pipes/:id/edit - Visual editor (requires auth)
• POST /auth/logout - End session
• GET /api/me - Get current user info
• GET /api/pipes - List user's pipes
• GET /api/pipes/:id - Get pipe config
• POST /api/pipes - Create pipe
• PUT /api/pipes/:id - Update pipe
• DELETE /api/pipes/:id - Delete pipe
• POST /api/pipes/:id/execute - Execute pipe manually
• GET /api/pipes/:id/executions - Execution history
• GET /api/executions/:id/logs - Execution logs
• GET /api/node-types - Available node types

Code Style#

• Use gofmt for formatting
• Structured logging with charmbracelet/log: logger.Info("message", "key", value)
• Error wrapping: fmt.Errorf("context: %w", err)
• Context propagation for cancellation
• Session cookies named pipes_session
• Authorization header: Bearer {token}

Design Aesthetic#

Neo-Brutalism - Bold, geometric design:

• Space Grotesk font family
• Hard borders (3-4px solid #26242b)
• Hard box shadows (6-12px offset, no blur)
• High contrast, sharp edges
• Uppercase text, tight letter-spacing
• Color palette:
  • App colors:
    • Primary: #2563eb (blue) - used for headings, secondary buttons
    • Secondary: #ff6b35 (orange) - used for primary buttons, accents
    • Background: #f5f5f0 (warm off-white)
    • Dark: #26242b (near-black)
    • White: #fff
  • Auth/Indiko colors (login, error pages):
    • Auth primary: #AB4967 (muted pink)
    • Auth text: #fff (white - for text on pink buttons)

Commands#

go build -o pipes .     # Build
./pipes serve           # Run server
./pipes init            # Initialize config files
./pipes help            # Show help
./pipes version         # Show version