this repo has no description
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge pull request #24 from websages/readme_lic

chore: Update README, add license

authored by

Michael Stahnke and committed by
GitHub
cdf86ad9 1e5a27f6

+140 -316
+21
LICENSE
··· 1 + MIT License 2 + 3 + Copyright (c) 2026 Michael Stahnke, et al 4 + 5 + Permission is hereby granted, free of charge, to any person obtaining a copy 6 + of this software and associated documentation files (the "Software"), to deal 7 + in the Software without restriction, including without limitation the rights 8 + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 + copies of the Software, and to permit persons to whom the Software is 10 + furnished to do so, subject to the following conditions: 11 + 12 + The above copyright notice and this permission notice shall be included in all 13 + copies or substantial portions of the Software. 14 + 15 + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 + SOFTWARE.
+119 -316
README.md
··· 4 4 5 5 ## History 6 6 7 - Tumble was a "Wouldn't it be cool?" project handed to [Scott Schnedier](https://github.com/sschneid) back in 2004. The idea was to create a website similar to a tumbleblog. Obviously, eventually tumblr ccame along and the rest was history. 7 + Tumble was a "Wouldn't it be cool?" project handed to [Scott Schnedier](https://github.com/sschneid) back in 2004. The idea was to create a website similar to a tumbleblog. Eventually tumblr came along and the rest was history. 8 8 9 - ## Deployment 9 + The project was originally written in Perl and has since been rewritten in Go. 10 10 11 - Run 11 + # Developer Guide 12 12 13 - `make build` 13 + ## Quick Start 14 14 15 - Have a configuration file. 15 + This project uses [Flox](https://flox.dev) to manage its development environment. The Flox environment provides Go, make, jq, SQLite, ImageMagick, and libxml2. 16 16 17 - See the systemd unit files in the `contrib` directory for more information. 17 + ```bash 18 + # Activate the development environment 19 + flox activate 18 20 21 + # Build 22 + make build 19 23 20 - ## Database Support 24 + # Configure (edit to match your environment) 25 + cp conf/config.yaml.sqlite conf/config.yaml 26 + vi conf/config.yaml 21 27 22 - Tumble now supports both **MySQL** and **SQLite** databases. Choose the one that fits your needs: 28 + # Run 29 + make restart 30 + ``` 23 31 24 - - **SQLite**: Recommended for development, testing, and small deployments. No separate database server required. 25 - - **MySQL**: Recommended for production deployments with higher traffic. 32 + Tumble will automatically create database tables on first startup. 26 33 27 - ## Development Workflows 34 + ## Configuration 28 35 29 - ### Database Migrations 30 - 31 - Tumble uses **GORM AutoMigrate** to manage database schemas. Migrations are automatically applied on application startup. 32 - 33 - - **Mechanism**: The application checks the database schema on startup and creates/updates tables to match the Go structs in `internal/data/store.go`. 36 + Tumble uses [Viper](https://github.com/spf13/viper) for configuration via YAML files, environment variables, or defaults. 34 37 35 - ### Testing Infrastructure 38 + **Configuration search paths:** `conf/`, current directory (`.`) 36 39 37 - A robust test infrastructure is available for rapid development: 38 - 39 - - **Run Tests**: `make test-api` runs the integration tests. 40 - - **Test Database**: `make test-db` creates a fresh, disposable SQLite database (`tumble-test.sqlite`), runs migrations, and loads sample fixtures. 41 - - **Run with Test DB**: `make run-test` starts the application using the test database. 42 - - **Backup/Restore**: `make backup` and `make restore` allow you to snapshot your current development database. 43 - 44 - To customize the test environment, you can edit `conf/config-test.yaml`. 45 - 46 - ### Manual MySQL Verification 47 - 48 - To validate MySQL migration support without Docker, you can run against a local MySQL server: 49 - 50 - 1. **Create Local Database/User**: 51 - 52 - ```bash 53 - mysql -u root -p -e "CREATE DATABASE tumble_test;" 54 - mysql -u root -p -e "CREATE USER 'tumble'@'localhost' IDENTIFIED BY 'password';" 55 - mysql -u root -p -e "GRANT ALL PRIVILEGES ON tumble_test.* TO 'tumble'@'localhost';" 56 - ``` 57 - 58 - 2. **Configure**: Check `conf/config-test-mysql.yaml` matches your local credentials. 59 - 60 - 3. **Run Application**: 61 - 62 - ```bash 63 - bin/tumble conf/config-test-mysql.yaml 64 - ``` 65 - 66 - 4. **Load Fixtures**: 67 - ```bash 68 - export DRIVER=mysql 69 - export MYSQL_PASSWORD=password 70 - ./tests/load_fixtures.sh 71 - ``` 72 - 73 - ## Quick Setup 74 - 75 - ### 1. Configure Database and Application 76 - 77 - Tumble uses **Viper** for configuration, allowing you to configure the application using a file (`config.yaml`), environment variables, or default values. 78 - 79 - **Configuration Search Paths:** 80 - 81 - - `conf/` 82 - - `htdocs/` 83 - - Current directory (`.`) 84 - 85 - **Configuration File (config.yaml):** 86 - 87 - **For SQLite (default):** 40 + ### SQLite (recommended for development) 88 41 89 42 ```yaml 90 43 driver: sqlite ··· 94 47 request_timeout: 2s 95 48 ``` 96 49 97 - **For MySQL:** 50 + ### MySQL 98 51 99 52 ```yaml 100 53 driver: mysql ··· 106 59 baseurl: your.domain.com 107 60 ``` 108 61 109 - **Environment Variables:** 62 + ### Environment Variables 110 63 111 - You can override any configuration value using environment variables prefixed with `TUMBLE_`. Use underscores (`_`) to access nested keys. 64 + Override any config value with the `TUMBLE_` prefix. Use underscores for nested keys. 112 65 113 - - `TUMBLE_PORT=9090` 114 - - `TUMBLE_DRIVER=mysql` 115 - - `TUMBLE_DATABASE=production_db` 116 - - `TUMBLE_MODE=development` (Options: `development`, `production`. Default: `production`) 117 - - `TUMBLE_EMBED_ASSETS=true` (Options: `true`, `false`. Default: `true`) 118 - - `TUMBLE_LOGGING_LEVEL=debug` 119 - - `TUMBLE_REQUEST_TIMEOUT=2s` (Default: `2s`) 120 - - `TUMBLE_CLICK_SIGNING_KEY=your-secret` (Optional, enables signed click tracking) 66 + | Variable | Description | Default | 67 + |----------|-------------|---------| 68 + | `TUMBLE_PORT` | Listen port | `8080` | 69 + | `TUMBLE_DRIVER` | Database driver (`sqlite`, `mysql`) | `mysql` | 70 + | `TUMBLE_DATABASE` | Database name or file path | | 71 + | `TUMBLE_MODE` | `development` or `production` | `production` | 72 + | `TUMBLE_EMBED_ASSETS` | Load assets from binary or filesystem | `true` | 73 + | `TUMBLE_LOGGING_LEVEL` | Log level (`debug`, `info`, `warn`, `error`) | `info` | 74 + | `TUMBLE_REQUEST_TIMEOUT` | HTTP request timeout | `2s` | 75 + | `TUMBLE_ADMIN_SECRET` | Secret for admin API operations | | 76 + | `TUMBLE_CLICK_SIGNING_KEY` | Secret for signed click tracking | | 121 77 122 78 ### Environment Modes (`TUMBLE_MODE`) 123 79 124 - - **development**: 125 - - **Logging**: Text format, Debug level, Full SQL query logging. 126 - - **Errors**: Displays detailed error messages in the browser. 127 - - **production** (default): 128 - - **Logging**: JSON format, Info level, Error-only SQL logging. 129 - - **Errors**: Displays generic "Internal Server Error" message to users. 80 + - **development**: Text-format logging, debug level, full SQL query logging, detailed error messages in the browser. 81 + - **production** (default): JSON-format logging, info level, error-only SQL logging, generic error messages to users. 130 82 131 83 ### Asset Embedding (`TUMBLE_EMBED_ASSETS`) 132 84 133 - Controls whether templates and static assets are loaded from the embedded binary or from the filesystem. 85 + Controls whether templates and static assets are loaded from the compiled binary or the filesystem. 134 86 135 - - **true** (default): Assets are loaded from the compiled binary. This is the recommended setting for deployment - you only need the binary and a config file. 136 - - **false**: Assets are loaded from the filesystem (`internal/templates/views/` and `internal/assets/`). Templates are re-parsed on every request, enabling hot-reload during development. 87 + - **true** (default): Assets are loaded from the binary. Only the binary and a config file are needed for deployment. 88 + - **false**: Assets are loaded from `internal/templates/views/` and `internal/assets/`. Templates are re-parsed on every request, enabling hot-reload during development. 137 89 138 - This setting is independent of `TUMBLE_MODE`, allowing you to run in development mode (for verbose logging and detailed errors) while still using embedded assets for deployment. 90 + This setting is independent of `TUMBLE_MODE`. 139 91 140 - **Deployment Example:** 92 + ## Database Support 141 93 142 - ```yaml 143 - mode: development # Get detailed error messages and text logging 144 - embed_assets: true # But still use embedded assets (no source checkout needed) 145 - ``` 94 + Tumble supports both **MySQL** and **SQLite** databases. 146 95 147 - ### 2. Initialize Database 96 + - **SQLite**: Recommended for development, testing, and small deployments. No separate database server required. 97 + - **MySQL**: Recommended for production deployments with higher traffic. 148 98 149 - Simply start the application! Tumble will automatically detect your database type (configured in `config.yaml` or via env vars) and create the necessary tables if they don't exist. 99 + ### Migrations 150 100 151 - ### 3. Configure Web Server 101 + Tumble uses **GORM AutoMigrate** to manage database schemas. Migrations are automatically applied on application startup based on the Go structs in `internal/data/store.go`. 152 102 153 - 1. Update `/etc/httpd/conf.d/tumble.conf` with your server configuration 154 - 2. Disable SELinux or set proper context 155 - 3. Start httpd: 103 + ### MySQL Setup 156 104 157 - ```bash 158 - chkconfig httpd on 159 - service httpd start 160 - ``` 161 - 162 - ## Detailed Setup (MySQL) 163 - 164 - If you prefer manual MySQL setup: 105 + To set up a MySQL database manually: 165 106 166 107 ```bash 167 - # Install MySQL 168 - yum install mysql-server 169 - service mysqld start 170 - chkconfig mysqld on 171 - 172 - # Create database and user 173 - mysql -u root -p < sql/sql_setup 174 - 175 - # Run schema 176 - mysql -u tumble -p tumble < sql/schema.mysql 108 + mysql -u root -p < sql/user_and_database_setup.mysql 177 109 ``` 178 110 179 - ## Migration from MySQL to SQLite 111 + Then configure `conf/config.yaml` with your MySQL credentials. 180 112 181 - 1. Export your MySQL data: 113 + ## Development 182 114 183 - ```bash 184 - mysqldump -u tumble -p tumble > tumble_backup.sql 185 - ``` 115 + ### Makefile Targets 186 116 187 - 2. Update `config.yaml` to use SQLite 188 - 189 - 3. Run setup script: 190 - 191 - ```bash 192 - perl scripts/setup_database.pl 193 - ``` 194 - 195 - 4. Import data (requires conversion from MySQL to SQLite format) 196 - 197 - See [docs/database_setup.md](docs/database_setup.md) for detailed instructions. 198 - 199 - ## Debugging and Logging 117 + | Command | Description | 118 + |---------|-------------| 119 + | `make build` | Build the binary | 120 + | `make restart` | Build and restart the application | 121 + | `make kill` | Stop the running application | 122 + | `make test` | Run unit tests | 123 + | `make test-api` | Run API integration tests | 124 + | `make test-db` | Create a fresh test database with fixtures | 125 + | `make run-test` | Run the application with the test database | 126 + | `make backup` | Backup the current SQLite database | 127 + | `make restore` | Restore the SQLite database from backup | 128 + | `make fmt` | Run `go fmt` and clean up whitespace | 129 + | `make build-linux` | Cross-compile for Linux amd64 | 130 + | `make help` | Show all available targets | 200 131 201 - Tumble provides comprehensive database logging to help troubleshoot connection issues and diagnose problems. 132 + ### Testing 202 133 203 - ### Automatic Database Diagnostics 134 + - `make test-api` runs integration tests against a test instance. 135 + - `make test-db` creates a fresh SQLite test database with sample fixtures. 136 + - `make run-test` starts the application using `conf/config-test.yaml`. 204 137 205 - When the application starts, it automatically: 206 - 207 - - **Logs connection attempts** with database details (host, database name, file paths) 208 - - **Verifies database health** by checking for expected tables (`ircLink`, `image`, `quote`) 209 - - **Reports table statistics** including row counts for each table 210 - - **Warns about issues** such as: 211 - - Missing database files (SQLite) 212 - - Empty databases (no tables) 213 - - Missing expected tables 214 - - Empty tables (no data) 215 - - File permission problems (SQLite) 216 - - Connection failures with troubleshooting suggestions 217 - 218 - ### Debug Mode 219 - 220 - For verbose logging of all database operations, enable debug mode: 138 + To test against MySQL, edit `conf/config-test-mysql.yaml` and run: 221 139 222 140 ```bash 223 - export TUMBLE_DEBUG=1 141 + bin/tumble conf/config-test-mysql.yaml 224 142 ``` 225 143 226 - With debug mode enabled, you'll see: 144 + ### Logging 227 145 228 - - All SQL queries being executed 229 - - Row counts for each query result 230 - - Detailed query execution information 146 + Set `TUMBLE_MODE=development` or `TUMBLE_LOGGING_LEVEL=debug` for verbose output including SQL queries. Logs are written to `tumble.log` by default. 231 147 232 - **Example:** 148 + ## Deployment 233 149 234 - ```bash 235 - # Enable debug mode 236 - export TUMBLE_DEBUG=1 150 + See the systemd unit files in the `contrib/` directory for production deployment examples. 237 151 238 - # Start your web server or run the application 239 - perl -I htdocs/lib htdocs/index.cgi 240 - ``` 152 + The recommended approach: 241 153 242 - ### Log Output Examples 154 + 1. Build the binary with `make build` (or `make build-linux` for Linux targets). 155 + 2. Create a `config.yaml` with your production settings. 156 + 3. Run the binary: `bin/tumble config.yaml` 243 157 244 - **Successful MySQL connection:** 158 + With `embed_assets: true` (the default), only the binary and config file are needed on the server. 245 159 246 - ``` 247 - [MySQL] Attempting to connect to database 'tumble' on host 'localhost' as user 'tumble' 248 - [MySQL] Connection SUCCESSFUL 249 - [MySQL] Database contains 3 table(s) 250 - [MySQL] Tables: ircLink, image, quote 251 - ``` 160 + The production systemd units in `contrib/` use Flox to manage the MySQL service (`tumble-db.service` runs `flox activate --start-services`). See `contrib/README` for details. 252 161 253 - **SQLite with missing database file:** 162 + ## API 254 163 255 - ``` 256 - [SQLite] Attempting to connect to database file: tumble.db 257 - [SQLite] Database file DOES NOT EXIST 258 - [SQLite] - SQLite will create a new empty database file 259 - [SQLite] - You will need to run the database setup script to create tables 260 - [SQLite] Connection SUCCESSFUL 261 - [SQLite] WARNING: Database appears to be empty (no tables found) 262 - [SQLite] - You need to run the database setup script 263 - ``` 164 + Full API documentation is available at `/api/docs` on a running instance, generated from `internal/assets/openapi.json`. 264 165 265 - **Connection failure with diagnostics:** 166 + ### Link Submission 266 167 267 - ``` 268 - [MySQL] Connection FAILED: Access denied for user 'tumble'@'localhost' 269 - [MySQL] Diagnostics: 270 - [MySQL] - DSN: dbi:mysql:tumble;host=localhost 271 - [MySQL] - Username: tumble 272 - [MySQL] Troubleshooting suggestions: 273 - [MySQL] 1. Verify MySQL server is running: systemctl status mysql 274 - [MySQL] 2. Check credentials in config.yaml are correct 275 - [MySQL] 3. Verify user has permissions: GRANT ALL ON tumble.* TO 'tumble'@'localhost' 276 - ``` 168 + Submit links via `POST /link/`. Duplicate detection is automatic: 277 169 278 - ### API Features 279 - 280 - #### Link Submission with Duplicate Detection 281 - 282 - When submitting links via `/link/`, the API automatically detects if a URL has been previously posted and provides contextual information: 283 - 284 - - **Behavior**: Links are always added to the database, even if duplicates exist 285 170 - **JSON API** (`Accept: application/json` or `source=api`): 286 - - **201 Created**: New link (first time posted) 287 - - **208 Already Reported**: Duplicate detected, includes details of all previous submissions 171 + - **201 Created**: New link 172 + - **208 Already Reported**: Duplicate, includes previous submission details 288 173 - **IRC Source** (`source=irc`): Returns ID with duplicate marker if applicable 289 - - New: `"123"` 290 - - Duplicate: `"123 (duplicate, previously posted by alice)"` 291 - - **HTML Response**: Shows duplicate notification with original poster and timestamp 292 - 293 - **Example JSON Response (Duplicate):** 294 - 295 - ```json 296 - { 297 - "link_id": 456, 298 - "is_duplicate": true, 299 - "previous_submissions": [ 300 - { 301 - "link_id": 123, 302 - "user": "alice", 303 - "timestamp": "2026-01-15T10:30:00Z", 304 - "title": "Example Page" 305 - } 306 - ] 307 - } 308 - ``` 174 + - **HTML**: Shows duplicate notification with original poster and timestamp 309 175 310 - For complete API documentation, visit `/docs` on your running instance or see `internal/assets/openapi.json`. 176 + The legacy `/irclink/` endpoint is still supported but `/link/` is preferred. 311 177 312 - > **Note:** The legacy `/irclink/` endpoint is still supported for backwards compatibility but `/link/` is preferred. 178 + ### Link and Quote Deletion 313 179 314 - #### Link Deletion 315 - 316 - Links can be deleted via the API using the `DELETE` method on `/link/123` (where `123` is the link ID). This requires authentication using an admin secret. 317 - 318 - **Configuration:** 319 - 320 - Add an `admin_secret` to your `config.yaml`: 321 - 322 - ```yaml 323 - admin_secret: "your-random-secret-string" 324 - ``` 325 - 326 - You can also set it via environment variable: `TUMBLE_ADMIN_SECRET=your-secret` 327 - 328 - **Usage:** 180 + Delete links or quotes via `DELETE /link/{id}` or `DELETE /quote/{id}`. Requires authentication: 329 181 330 182 ```bash 331 - # Using X-Admin-Secret header (recommended) 183 + # Using header (recommended) 332 184 curl -X DELETE -H "X-Admin-Secret: your-secret" https://your-server/link/123 333 185 334 186 # Using query parameter 335 187 curl -X DELETE "https://your-server/link/123?secret=your-secret" 336 188 ``` 337 189 338 - **Responses:** 339 - 340 - - **200 OK**: Link deleted successfully 341 - - **400 Bad Request**: Missing or invalid ID 342 - - **403 Forbidden**: Missing or invalid admin secret 343 - - **404 Not Found**: Link does not exist 344 - 345 - If no `admin_secret` is configured, deletion falls back to localhost-only access for backwards compatibility. 346 - 347 - #### Quote Deletion 348 - 349 - Quotes can be deleted via the API using the `DELETE` method on `/quote/123` (where `123` is the quote ID). This uses the same authentication mechanism as link deletion. 350 - 351 - **Usage:** 352 - 353 - ```bash 354 - # Using X-Admin-Secret header (recommended) 355 - curl -X DELETE -H "X-Admin-Secret: your-secret" https://your-server/quote/123 190 + Configure `admin_secret` in your config file or via `TUMBLE_ADMIN_SECRET`. Without it, deletion falls back to localhost-only access. 356 191 357 - # Using query parameter 358 - curl -X DELETE "https://your-server/quote/123?secret=your-secret" 359 - ``` 192 + ### Click Signature Tracking 360 193 361 - **Responses:** 194 + Optional HMAC-signed URLs for verified click tracking. When `click_signing_key` is configured, links render as `/link/123?sig=abc123...` and the server validates signatures to distinguish real clicks from bots. 362 195 363 - - **200 OK**: Quote deleted successfully 364 - - **400 Bad Request**: Missing or invalid ID 365 - - **403 Forbidden**: Missing or invalid admin secret 366 - - **404 Not Found**: Quote does not exist 196 + ### Dead Link Detection 367 197 368 - #### Click Signature Tracking 198 + Tumble automatically detects dead links (4xx/5xx responses) and checks the [Wayback Machine](https://archive.org) for archived snapshots. A background job runs on startup and daily. Archive.org requests are rate-limited to 5 req/s. 369 199 370 - Tumble supports signed URLs for verified click tracking. When enabled, links include an HMAC signature that validates clicks came from the rendered page rather than bots or direct URL access. 200 + ### Caching 371 201 372 - **Configuration:** 202 + Link previews are cached in the database. Disable with `caching.enabled: false` in config. Invalidate manually via `GET /api/caching/invalidate?url=<encoded_url>`. 373 203 374 - Add a `click_signing_key` to your `config.yaml`: 204 + ## Project Structure 375 205 376 - ```yaml 377 - click_signing_key: "your-random-secret-string" 378 206 ``` 379 - 380 - Or set via environment variable: `TUMBLE_CLICK_SIGNING_KEY=your-secret` 381 - 382 - **How it works:** 383 - 384 - - When configured, links render as `/link/123?sig=abc123...` instead of `/link/123` 385 - - The signature is an HMAC-SHA256 hash of the link ID using your secret key 386 - - On redirect, the server validates the signature to distinguish verified clicks from unverified access 387 - - This helps track genuine user engagement vs. crawler/bot traffic 388 - 389 - **Note:** If no `click_signing_key` is configured, links work normally without signatures. This feature is optional and doesn't affect basic functionality. 390 - 391 - #### Dead Link Detection and Archive.org Integration 392 - 393 - With ~20 years of links, many URLs have gone dead over time. Tumble 394 - automatically detects dead links and checks the [Wayback Machine](https://archive.org) 395 - for archived snapshots. 396 - 397 - - **Detection**: Links returning 4xx/5xx errors are flagged with an HTTP 398 - status badge and excluded from search results. 399 - - **Archive.org lookups**: A background job checks dead links against the 400 - Wayback Machine Availability API on startup and daily. Newly detected 401 - dead links are also checked on-demand. 402 - - **UI**: When an archived snapshot exists, a "View on Archive.org" link 403 - appears next to the error badge with the snapshot date, giving users 404 - access to the original content. Works in both the default theme and 405 - Scott Mode. 406 - - **Rate limiting**: Archive.org requests are limited to 5 req/s to stay 407 - well within API limits. 408 - 409 - #### Caching 410 - 411 - Link previews are cached in the database to reduce external requests. 207 + cmd/tumble/ Entry point 208 + internal/ 209 + assets/ Static assets and OpenAPI spec 210 + config/ Configuration loading 211 + data/ Database models and queries 212 + handler/ HTTP handlers 213 + templates/ HTML templates 214 + service/ Business logic 215 + scheduler/ Background jobs 216 + archive/ Archive.org integration 217 + conf/ Configuration files 218 + contrib/ Systemd unit files 219 + sql/ SQL setup scripts 220 + tests/ Integration tests and fixtures 221 + ``` 412 222 413 - - `caching.enabled`: Set to `false` to disable server-side caching. 414 - - To invalidate a cache entry manually: 415 - `GET /api/caching/invalidate?url=<encoded_url>` 223 + ## License 416 224 417 - ## Bugs 418 - 419 - * fix user-agent being hardy for link verification 420 - * abstract quantity of items to be in 'hot shit' category 421 - * Fix odd encoding bugs for web site titles 422 - * Probably lots of others, but it has been in production for 10 years. 225 + This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.