stahnma.us / tumble
this repo has no description
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

feat: databse migrations now in play

Previously the database had once schema file and that was edited. Now we have
migrations so changes can be made over time. The migrations support SQLite
(used for local development) and MySQL compatible databses.

Michael Stahnke 143a3d78 00a7b1ec

+321 -44
+1
.gitignore
··· 1 1 tumble.sqlite 2 + tumble-test.sqlite 2 3 bin/* 3 4 *.log
+16
Makefile
··· 48 48 reset-db: ## Remove sqlite database 49 49 rm -f tumble.sqlite 50 50 51 + backup: ## Backup the current database 52 + cp tumble.sqlite tumble.sqlite.bak 53 + @echo "Database backed up to tumble.sqlite.bak" 54 + 55 + restore: ## Restore the database from backup 56 + cp tumble.sqlite.bak tumble.sqlite 57 + @echo "Database restored from tumble.sqlite.bak" 58 + 51 59 load-fixtures: ## Load test fixtures 52 60 ./tests/load_fixtures.sh 61 + 62 + run-test: build ## Run the application with the test database 63 + $(BUILD_DIR)/$(BINARY_NAME) conf/config-test.yaml 64 + 65 + 66 + 67 + test-db: build ## Create a fresh test database with fixtures 68 + ./tests/setup_test_db.sh 53 69 54 70 help: ## Show this help message 55 71 @grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
+30 -1
README.md
··· 12 12 13 13 If you are not on EL, things should still work. Just `make install` or package it yourself. 14 14 15 - 16 15 ## Database Support 17 16 18 17 Tumble now supports both **MySQL** and **SQLite** databases. Choose the one that fits your needs: ··· 20 19 - **SQLite**: Recommended for development, testing, and small deployments. No separate database server required. 21 20 - **MySQL**: Recommended for production deployments with higher traffic. 22 21 22 + ## Development Workflows 23 + 24 + ### Database Migrations 25 + 26 + Tumble uses [golang-migrate](https://github.com/golang-migrate/migrate) to manage database schemas. Migrations are automatically applied on application startup. 27 + 28 + - **Location**: Migration files are stored in `sql/mysql/` and `sql/sqlite/`. 29 + - **Versioning**: Files are named sequentially (e.g., `000001_init.up.sql`). 30 + - **Adding Migrations**: To change the schema, create a new numbered `.up.sql` file in the appropriate directory. 31 + 32 + ### Testing Infrastructure 33 + 34 + A robust test infrastructure is available for rapid development: 35 + 36 + - **Run Tests**: `make test-api` runs the integration tests. 37 + - **Test Database**: `make test-db` creates a fresh, disposable SQLite database (`tumble-test.sqlite`), runs migrations, and loads sample fixtures. 38 + - **Run with Test DB**: `make run-test` starts the application using the test database. 39 + - **Backup/Restore**: `make backup` and `make restore` allow you to snapshot your current development database. 40 + 41 + To customize the test environment, you can edit `conf/config-test.yaml`. 42 + 23 43 ## Quick Setup 24 44 25 45 ### 1. Configure Database ··· 27 47 Create or edit `config.yaml` in the htdocs directory: 28 48 29 49 **For SQLite (easiest):** 50 + 30 51 ```yaml 31 52 driver: sqlite 32 53 database_file: tumble.db ··· 34 55 ``` 35 56 36 57 **For MySQL:** 58 + 37 59 ```yaml 38 60 driver: mysql 39 61 database: tumble ··· 84 106 ## Migration from MySQL to SQLite 85 107 86 108 1. Export your MySQL data: 109 + 87 110 ```bash 88 111 mysqldump -u tumble -p tumble > tumble_backup.sql 89 112 ``` ··· 91 114 2. Update `config.yaml` to use SQLite 92 115 93 116 3. Run setup script: 117 + 94 118 ```bash 95 119 perl scripts/setup_database.pl 96 120 ``` ··· 127 151 ``` 128 152 129 153 With debug mode enabled, you'll see: 154 + 130 155 - All SQL queries being executed 131 156 - Row counts for each query result 132 157 - Detailed query execution information 133 158 134 159 **Example:** 160 + 135 161 ```bash 136 162 # Enable debug mode 137 163 export TUMBLE_DEBUG=1 ··· 143 169 ### Log Output Examples 144 170 145 171 **Successful MySQL connection:** 172 + 146 173 ``` 147 174 [MySQL] Attempting to connect to database 'tumble' on host 'localhost' as user 'tumble' 148 175 [MySQL] Connection SUCCESSFUL ··· 151 178 ``` 152 179 153 180 **SQLite with missing database file:** 181 + 154 182 ``` 155 183 [SQLite] Attempting to connect to database file: tumble.db 156 184 [SQLite] Database file DOES NOT EXIST ··· 162 190 ``` 163 191 164 192 **Connection failure with diagnostics:** 193 + 165 194 ``` 166 195 [MySQL] Connection FAILED: Access denied for user 'tumble'@'localhost' 167 196 [MySQL] Diagnostics:
+4 -1
cmd/tumble/main.go
··· 177 177 mux.HandleFunc("/api/openapi.json", h.OpenAPISpecHandler) 178 178 179 179 // Start 180 - addr := ":8080" // Default or from config? Perl was CGI so port wasn't in config. 180 + addr := ":" + cfg.Port 181 + if cfg.Port == "" { 182 + addr = ":8080" 183 + } 181 184 slog.Info("Starting tumble server", "addr", addr) 182 185 if err := http.ListenAndServe(addr, loggingMiddleware(mux)); err != nil { 183 186 slog.Error("Server failed", "error", err)
+7
conf/config-test.yaml
··· 1 + driver: sqlite 2 + database: tumble-test.sqlite 3 + port: "8080" 4 + baseurl: localhost:8080 5 + logging: 6 + level: debug 7 + output: stdout
+1
go.mod
··· 12 12 require ( 13 13 filippo.io/edwards25519 v1.1.0 // indirect 14 14 github.com/dustin/go-humanize v1.0.1 // indirect 15 + github.com/golang-migrate/migrate/v4 v4.19.1 // indirect 15 16 github.com/google/uuid v1.6.0 // indirect 16 17 github.com/mattn/go-isatty v0.0.20 // indirect 17 18 github.com/ncruces/go-strftime v0.1.9 // indirect
+5
go.sum
··· 4 4 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= 5 5 github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo= 6 6 github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU= 7 + github.com/golang-migrate/migrate/v4 v4.19.1 h1:OCyb44lFuQfYXYLx1SCxPZQGU7mcaZ7gH9yH4jSFbBA= 8 + github.com/golang-migrate/migrate/v4 v4.19.1/go.mod h1:CTcgfjxhaUtsLipnLoQRWCrjYXycRz/g5+RWDuYgPrE= 7 9 github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs= 8 10 github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA= 9 11 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= ··· 18 20 golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b/go.mod h1:3//PLf8L/X+8b4vuAfHzxeRUl04Adcb341+IGKfnqS8= 19 21 golang.org/x/mod v0.27.0 h1:kb+q2PyFnEADO2IEF935ehFUXlWiNjJWtRNgBLSfbxQ= 20 22 golang.org/x/mod v0.27.0/go.mod h1:rWI627Fq0DEoudcK+MBkNkCe0EetEaDSwJJkCcjpazc= 23 + golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA= 21 24 golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU= 22 25 golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY= 23 26 golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw= 24 27 golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= 28 + golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= 25 29 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 26 30 golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk= 27 31 golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= 28 32 golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg= 29 33 golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s= 34 + golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ= 30 35 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 31 36 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 32 37 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+4
internal/config/config.go
··· 14 14 Password string `yaml:"password"` 15 15 BaseURL string `yaml:"baseurl"` 16 16 Driver string `yaml:"driver"` 17 + Port string `yaml:"port"` 17 18 Logging Logging `yaml:"logging"` 18 19 } 19 20 ··· 44 45 } 45 46 if cfg.Logging.Output == "" { 46 47 cfg.Logging.Output = "stdout" 48 + } 49 + if cfg.Port == "" { 50 + cfg.Port = "8080" 47 51 } 48 52 49 53 return &cfg, nil
+59
internal/data/migrations.go
··· 1 + package data 2 + 3 + import ( 4 + "database/sql" 5 + "fmt" 6 + "log/slog" 7 + 8 + "github.com/golang-migrate/migrate/v4" 9 + "github.com/golang-migrate/migrate/v4/database" 10 + "github.com/golang-migrate/migrate/v4/database/mysql" 11 + "github.com/golang-migrate/migrate/v4/source/iofs" 12 + 13 + tumblesql "tumble/sql" 14 + ) 15 + 16 + // RunMigrations executes pending migrations. 17 + // driverName should be "mysql" or "sqlite" 18 + func RunMigrations(db *sql.DB, driverName string, databaseName string) error { 19 + slog.Info("Running migrations", "driver", driverName) 20 + 21 + sourceDriver, err := iofs.New(tumblesql.MigrationFS, driverName) 22 + if err != nil { 23 + return fmt.Errorf("failed to create iofs source: %w", err) 24 + } 25 + 26 + var databaseDriver database.Driver 27 + 28 + switch driverName { 29 + case "mysql": 30 + databaseDriver, err = mysql.WithInstance(db, &mysql.Config{}) 31 + if err != nil { 32 + return fmt.Errorf("failed to create mysql driver: %w", err) 33 + } 34 + case "sqlite": 35 + // Custom driver for modernc/sqlite (no CGO) 36 + // We implement this in sqlite_driver.go 37 + databaseDriver, err = WithInstance(db, &Config{}) 38 + if err != nil { 39 + return fmt.Errorf("failed to create sqlite driver: %w", err) 40 + } 41 + default: 42 + return fmt.Errorf("unsupported driver: %s", driverName) 43 + } 44 + 45 + m, err := migrate.NewWithInstance( 46 + "iofs", sourceDriver, 47 + driverName, databaseDriver, 48 + ) 49 + if err != nil { 50 + return fmt.Errorf("failed to create migrate instance: %w", err) 51 + } 52 + 53 + if err := m.Up(); err != nil && err != migrate.ErrNoChange { 54 + return fmt.Errorf("failed to run up migrations: %w", err) 55 + } 56 + 57 + slog.Info("Migrations completed successfully") 58 + return nil 59 + }
+5 -29
internal/data/mysql.go
··· 394 394 } 395 395 396 396 func (s *MySQLStore) Bootstrap(ctx context.Context) error { 397 - schema, err := SchemaFS.ReadFile("schema.mysql") 398 - if err != nil { 399 - return err 400 - } 401 - // Simple split by ; might fail on complex SQL, but for this schema it's fine. 402 - // Actually, the schema has multi-line statements. 403 - // A robust solution executes the whole script if the driver supports it, or splits carefully. 404 - // MySQL driver often supports multiple statements if enabled, but better to execute one by one if split properly. 405 - // For this specific schema, splitting by `;` works because there are no semicolons inside strings/triggers. 406 - // HOWEVER, creating a new method to execute script is cleaner. 407 - 408 - // Actually, just executing the whole thing might work if multiStatements=true in DSN, but let's assume not. 409 - // We'll follow a simple split approach for now, or just execute the known CREATE statements. 410 - // Since we want to use the embedded file, we should parse it. 411 - 412 - // Simpler: Just execute the file content? 413 - // Drivers behave differently. 414 - // Let's rely on the file content being simple enough. 415 - 416 - queries := splitSQL(string(schema)) 417 - for _, q := range queries { 418 - if q == "" { 419 - continue 420 - } 421 - if _, err := s.db.ExecContext(ctx, q); err != nil { 422 - return fmt.Errorf("failed to execute query %q: %w", q, err) 423 - } 424 - } 425 - return nil 397 + // User databaseName "tumble" or extract from config? 398 + // DSN parsing is complex. For now we assume "mysql" driver name is sufficient. 399 + // Actually, RunMigrations needs databaseName to keep migrate logic happy implicitly? 400 + // The driverName check we wrote just uses "mysql". 401 + return RunMigrations(s.db, "mysql", "mysql") 426 402 }
+1 -11
internal/data/sqlite.go
··· 391 391 } 392 392 393 393 func (s *SQLiteStore) Bootstrap(ctx context.Context) error { 394 - schema, err := SchemaFS.ReadFile("schema.sqlite") 395 - if err != nil { 396 - return err 397 - } 398 - 399 - // modernc.org/sqlite usually handles multiple statements in one Exec. 400 - // Let's try executing the whole block. 401 - if _, err := s.db.ExecContext(ctx, string(schema)); err != nil { 402 - return err 403 - } 404 - return nil 394 + return RunMigrations(s.db, "sqlite", "sqlite") 405 395 }
+137
internal/data/sqlite_driver.go
··· 1 + package data 2 + 3 + import ( 4 + "database/sql" 5 + "fmt" 6 + "io" 7 + "io/ioutil" 8 + "strings" 9 + 10 + migrate "github.com/golang-migrate/migrate/v4/database" 11 + ) 12 + 13 + func init() { 14 + // We don't register automatically to avoid importing this package everywhere 15 + } 16 + 17 + type Config struct { 18 + MigrationsTable string 19 + NoTxWrap bool 20 + } 21 + 22 + type SQLite struct { 23 + db *sql.DB 24 + config *Config 25 + } 26 + 27 + func WithInstance(instance *sql.DB, config *Config) (migrate.Driver, error) { 28 + if config == nil { 29 + config = &Config{} 30 + } 31 + if config.MigrationsTable == "" { 32 + config.MigrationsTable = "schema_migrations" 33 + } 34 + return &SQLite{ 35 + db: instance, 36 + config: config, 37 + }, nil 38 + } 39 + 40 + func (s *SQLite) Open(url string) (migrate.Driver, error) { 41 + return nil, fmt.Errorf("not implemented, use WithInstance") 42 + } 43 + 44 + func (s *SQLite) Close() error { 45 + // We don't close the DB here as it's passed in instance 46 + return nil 47 + } 48 + 49 + func (s *SQLite) Lock() error { 50 + return nil // SQLite doesn't need explicit locking usually if single threaded migration or handled by db lock 51 + } 52 + 53 + func (s *SQLite) Unlock() error { 54 + return nil 55 + } 56 + 57 + func (s *SQLite) Run(migration io.Reader) error { 58 + migr, err := ioutil.ReadAll(migration) 59 + if err != nil { 60 + return err 61 + } 62 + query := string(migr) 63 + if strings.TrimSpace(query) == "" { 64 + return nil 65 + } 66 + 67 + if s.config.NoTxWrap { 68 + _, err := s.db.Exec(query) 69 + return err 70 + } 71 + 72 + tx, err := s.db.Begin() 73 + if err != nil { 74 + return err 75 + } 76 + if _, err := tx.Exec(query); err != nil { 77 + tx.Rollback() 78 + return err 79 + } 80 + return tx.Commit() 81 + } 82 + 83 + func (s *SQLite) SetVersion(version int, dirty bool) error { 84 + tx, err := s.db.Begin() 85 + if err != nil { 86 + return err 87 + } 88 + // Delete all 89 + if _, err := tx.Exec(fmt.Sprintf("DELETE FROM %s", s.config.MigrationsTable)); err != nil { 90 + tx.Rollback() 91 + return err 92 + } 93 + // Insert new 94 + if _, err := tx.Exec(fmt.Sprintf("INSERT INTO %s (version, dirty) VALUES (?, ?)", s.config.MigrationsTable), version, dirty); err != nil { 95 + tx.Rollback() 96 + return err 97 + } 98 + return tx.Commit() 99 + } 100 + 101 + func (s *SQLite) Version() (version int, dirty bool, err error) { 102 + query := fmt.Sprintf("SELECT version, dirty FROM %s LIMIT 1", s.config.MigrationsTable) 103 + err = s.db.QueryRow(query).Scan(&version, &dirty) 104 + if err != nil { 105 + if err == sql.ErrNoRows { 106 + return migrate.NilVersion, false, nil 107 + } 108 + // If table doesn't exist, create it and return nil version 109 + // simple check: try creating 110 + if err := s.ensureVersionTable(); err != nil { 111 + return 0, false, err 112 + } 113 + // Retry 114 + err = s.db.QueryRow(query).Scan(&version, &dirty) 115 + if err == sql.ErrNoRows { 116 + return migrate.NilVersion, false, nil 117 + } 118 + return version, dirty, err 119 + } 120 + return version, dirty, nil 121 + } 122 + 123 + func (s *SQLite) Drop() error { 124 + // Not implemented 125 + return nil 126 + } 127 + 128 + func (s *SQLite) ensureVersionTable() error { 129 + query := fmt.Sprintf(` 130 + CREATE TABLE IF NOT EXISTS %s ( 131 + version INTEGER PRIMARY KEY, 132 + dirty BOOLEAN NOT NULL 133 + ) 134 + `, s.config.MigrationsTable) 135 + _, err := s.db.Exec(query) 136 + return err 137 + }
+6
sql/embed.go
··· 1 + package sql 2 + 3 + import "embed" 4 + 5 + //go:embed mysql/*.sql sqlite/*.sql 6 + var MigrationFS embed.FS
sql/schema.mysql sql/mysql/000001_init.up.sql
sql/schema.sqlite sql/sqlite/000001_init.up.sql
+6 -2
tests/load_fixtures.sh
··· 2 2 # Load fixtures into the database 3 3 # Usage: ./tests/load_fixtures.sh 4 4 5 - BASE_URL="http://localhost:8080" 5 + BASE_URL="${API_BASE_URL:-http://localhost:8080}" 6 + DB_PATH="${DB_PATH:-tumble.sqlite}" 6 7 ADD_LINK_SCRIPT="./tests/add_link.sh" 8 + 9 + echo "Using BASE_URL: $BASE_URL" 10 + echo "Using DB_PATH: $DB_PATH" 7 11 8 12 echo "Loading fixtures..." 9 13 ··· 79 83 $ADD_LINK_SCRIPT "gif_master" "https://giphy.com/gifs/cant-hardly-wait-kW8mnYSNkUYKc" 80 84 81 85 echo "Loading backdated 'Hot Links' directly into DB..." 82 - sqlite3 tumble.sqlite < tests/fixtures_hot.sql 86 + sqlite3 "$DB_PATH" < tests/fixtures_hot.sql 83 87 84 88 echo "Fixtures loaded."
+39
tests/setup_test_db.sh
··· 1 + #!/bin/bash 2 + set -e 3 + 4 + # Configuration 5 + BINARY="./bin/tumble" 6 + CONFIG="conf/config-test.yaml" 7 + DB_PATH="tumble-test.sqlite" 8 + PORT="8080" 9 + BASE_URL="http://localhost:$PORT" 10 + 11 + echo "Setting up $DB_PATH..." 12 + rm -f "$DB_PATH" 13 + 14 + # Start server 15 + echo "Starting server..." 16 + $BINARY "$CONFIG" & 17 + PID=$! 18 + echo "Server PID: $PID" 19 + 20 + # Ensure cleanup 21 + trap "echo 'Stopping server...'; kill $PID || true" EXIT 22 + 23 + # Wait for server to be ready 24 + echo "Waiting for server to be ready on port $PORT..." 25 + for i in {1..30}; do 26 + if curl -s "http://localhost:$PORT" >/dev/null; then 27 + echo "Server is up!" 28 + break 29 + fi 30 + sleep 1 31 + done 32 + 33 + # Run fixtures 34 + echo "Running fixtures..." 35 + export DB_PATH="$DB_PATH" 36 + export API_BASE_URL="$BASE_URL" 37 + ./tests/load_fixtures.sh 38 + 39 + echo "Test database created successfully."