···11+<component name="CopyrightManager">
22+ <copyright>
33+ <option name="notice" value=" Lumina/Peonies Copyright (C) 2018-2026 MLC 'Strawmelonjuice' Bloeiman and contributors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see <https://www.gnu.org/licenses/>." />
44+ <option name="myName" value="GNU AGPLv3" />
55+ </copyright>
66+</component>
···11+when:
22+ - event: ["push", "manual"]
33+ branch: ["develop"]
44+ - event: ["pull_request"]
55+ branch: ["main"] # We have no main, yet.
66+77+engine: "nixery"
88+99+# using the default values
1010+clone:
1111+ skip: false
1212+ depth: 1
1313+ submodules: false
1414+1515+dependencies:
1616+ nixpkgs/nixpkgs-unstable:
1717+ # I wish I could just use the flake, but alas aargh
1818+ - mise
1919+2020+#environment:
2121+# MY_ENV_VAR: "MY_ENV_VALUE"
2222+2323+steps:
2424+ - name: "Run tests through Mise"
2525+ # I don't think this'll work, we have no database to reflect from!
2626+ command: "mise i && mise run check"
2727+ environment:
2828+ GOOS: "darwin"
2929+ GOARCH: "arm64"
···11+# Lumina(/peonies) server
22+33+> Notice:
44+> This project lives on [Tangled](https://tangled.org/strawmelonjuice.com/Lumina), it is mirrored to Codeberg and a few
55+> other places, but the main development happens on Tangled.
66+> Please report issues and contribute on Tangled.
77+88+Lumina is a project in development, as the short description says "Just trying out an old concept.". It is not in any
99+way ready for you to try. However, you are encouraged to contribute in any way!
1010+1111+Currently, as no stable is produced yet, code lives mainly in the `development` branch.
+8
metadata.json
···11+{
22+ "version": "1.0",
33+ "attribution": "MLC Strawmelonjuice Bloeiman",
44+ "tdm_reservation": true,
55+ "ai_training_allowed": false,
66+ "license": "EUPL-1.2",
77+ "derivative_work_claim": "AI models trained on this source are considered derivative works."
88+}
···11+The server should a poll-inspired syncing system for Federating posts with other servers (instances). This is a [[High-level requirements#^4c6bf0|must]].
22+33+Having established this, a _how_ remains. In an earlier iteration of Lumina/Ephew (`Lumina:Peonies:itr1`), this how was conceptually answered by introducing HTTP requests fetching other instances' post IDs and then letting the client fetch the actual post content. This is a sound strategy in theory, however, you would possibly fetch posts from an instance impersonating the instance you talked to previously.
44+To keep this from happening, a choice was made to only accept domain names as instance ID's, this choice is still present in the current iteration, however, DNS is not infallible.
55+66+To solve that, some form of key checking has to be done. Either by sharing a secret token and having an instance store it, or... more sane, by `ed25519`?
77+88+Furthermore, these relatively big HTTP requests would be rate-limited on the requesting side, on the serving side, request spammers would be autoremoved from the allowlist.
99+1010+That initially created the concept of WebSocket connections, preferably ones that stay open forever (which is a long time). However, more recently, the [polyproto](https://polyproto.org) has been on my mind.
1111+1212+### Polyproto
1313+1414+I have to look into protocol-specific details later, however, each Lumina instance could also be a Polyproto 'homeserver', thereby allowing the instance to communicate to other instances using an instance user e.g., `iic@peonies.xyz`, AND as users on the instance, e.g. `user+comment@peonies.xyz`. Then federating timelines would of course go over the iic username, however things like comments or DM's, that'd require more direct federation, would be sent directly using JSON from `<username>+<reason>@<instance>` to the instance user of another instance. One of the pitfalls of the earlier conceptual implementation, was that due to the rate-limits of HTTP polling, there was at least 30 seconds of delay, this concept seems to resolve that fantastically.
···11+I notice on other social media platforms that when an edit or upload has unforseen consequences, this usually results in
22+broken push notifications.
33+Notifications that lead to a 404.
44+55+When creating a push notification for something we know nothing about, this is what it is. But in Lumina these
66+notifications lead to a postview using a JSON string after the # in the url (url hash).
77+88+That JSON string can just contain a post or notification id, but we generate a preview of a post, and that
99+preview should be pushed but not saved once again in the database. Instead, we add to the JSON object. Client holds an
1010+absolute reference to the post id, but also requests a post by both id and a part of its preview. Lumina client should
1111+be smart enough to figure out what post this is even when the ID leads to a 404.
+69
notes/Design choices/Philosophies/'Timeline carries most' and the database.md
···11+Rationale: [[Backend > Timeline carries most]]
22+# Lumina Data Storage Architecture
33+44+This document outlines the data storage architecture of the Lumina social platform, based on the existing implementation
55+and design principles.
66+77+88+99+## Core Philosophy: "Timeline-carries-most"
1010+1111+Lumina's storage is designed around a central principle referred to as "Timeline-carries-most". The core idea is that
1212+the `timelines` table, which is expected to be the most frequently accessed table, should be as minimal and efficient as
1313+possible.
1414+1515+- It acts as an index, containing only a timeline ID (`tlid`), an item ID (`item_id`), and a `timestamp`.
1616+- It does not store any content itself, only the relationship between a timeline and an item.
1717+1818+1919+## Database System
2020+2121+Lumina supports two SQL database backends:
2222+2323+- **PostgreSQL**: The recommended database for production environments.
2424+- **SQLite**: Supported for testing and development purposes.
2525+2626+The choice of a database is configured via the `LUMINA_DB_TYPE` environment variable.
2727+2828+## Item and Content Storage
2929+3030+Content in Lumina is stored in a flexible, multi-table system that allows for various types of items to be added to
3131+timelines.
3232+3333+### 1. The Item Lookup Table (`itemtypelookupdb`)
3434+3535+This table acts as a central directory or "forwarding table". Its purpose is to map a generic `item_id` to its specific
3636+content type.
3737+3838+- It contains an `item_id` and an `itemtype` string.
3939+- The `itemtype` string directly corresponds to the name of the database table where the item's specific data is
4040+ stored (e.g., `post_text`, `post_article`).
4141+4242+### 2. Specific Content Tables
4343+4444+Each type of content has its own dedicated table. This design allows for adding new content types without altering the
4545+core timeline logic. The initial three content tables are:
4646+4747+- `post_text`: For short, microblog-style text posts.
4848+- `post_media`: For media-focused posts (e.g., images, videos). The table stores a reference to a **MinIO object ID**,
4949+ not the media file itself.
5050+- `post_article`: For long-form content with a title and body.
5151+5252+For direct messages among others, there will be more variants of these.
5353+5454+### 3. Handling Foreign Content (Federation)
5555+5656+The content tables are designed to accommodate posts from other federated instances. Each content table includes the
5757+following nullable fields:
5858+5959+- `foreign_instance_id`: Stores the identifier of the instance where the post originated.
6060+- `foreign_post_id`: Stores the post's original ID from its home instance.
6161+6262+This allows a local copy of the content to exist while preserving a reference to its original source.
6363+6464+## Caching Layer
6565+6666+- **Redis** is used as a caching and performance-optimization layer.
6767+- It is used for timeline caching and for ephemeral data structures like Bloom filters to quickly check for the
6868+ existence of usernames and emails.
6969+- Redis does not store any persistent, canonical data.
···11+Lumina is not minimalistic. It follows web design trends from decades ago with that approach, but also prefers a clean and modern design. This is why coloured elements are very important, but it also allows us to implement features considered old-school nowadays. Like the ability
+57
notes/Design choices/Rationale/Backend > Connection pooling and caching.md
···11+## Overview
22+- PostgreSQL and Redis are pooled with `bb8` (see `server/src/database.rs`).
33+- Redis is used only for performance: bloom filters and timeline caches; PostgreSQL remains source of truth.
44+- Background maintainer (`database::maintain`) periodically deletes old sessions and prunes timeline caches.
55+66+## PostgreSQL pool (bb8-postgres)
77+- Built from `tokio_postgres::Config` with `NoTls`.
88+- Pools are cloned everywhere via `DatabaseConnections::get_postgres_pool` to keep acquisition cheap.
99+- Avoid `unwrap()` on `.get()`; surface bb8 run errors via `LuminaError::Bb8RunErrorPg`.
1010+1111+Example (simplified):
1212+```rust
1313+let pg_pool = PgConn { postgres_pool, redis_pool };
1414+let client = pg_pool.postgres_pool.get().await?; // ? maps to LuminaError::Bb8RunErrorPg
1515+```
1616+1717+## Redis pool (bb8-redis)
1818+- Configured pool builder (currently max_size 50, 5s timeout, 5m idle timeout).
1919+- Connection type is `MultiplexedConnection`; use `redis::cmd(...).query_async(&mut **conn)`. Pool errors surface as `LuminaError::Bb8RunErrorRedis`.
2020+2121+### Bloom filters
2222+- Keys: `bloom:email`, `bloom:username`.
2323+- Populated at startup from Postgres (`database::setup`).
2424+- Checked in `user::register_validitycheck` before DB uniqueness queries.
2525+2626+Example add/check:
2727+```rust
2828+let mut conn = redis_pool.get().await?;
2929+redis::cmd("BF.ADD").arg("bloom:email").arg(email).query_async(&mut *conn).await?;
3030+let exists: bool = redis::cmd("BF.EXISTS").arg("bloom:email").arg(email).query_async(&mut *conn).await?;
3131+```
3232+3333+### Timeline cache
3434+- Cache keys: `timeline_cache:{tlid}:page:{page}`; metadata key: `timeline_cache:{tlid}:meta`.
3535+- Cache TTL: 3600s; high-traffic threshold: 100 lookups (global always high-traffic).
3636+- Write path (`cache_timeline_page`) stores page JSON and total count; read path (`get_cached_timeline_page`) returns `CachedTimelinePage`.
3737+- Invalidation: `invalidate_timeline_cache` SCANs matching keys and DELs; called after timeline writes and from the maintainer loop when timelines change.
3838+- Background invalidation cursor uses `timeline_cache_last_check` stored in Redis.
3939+4040+Example invalidate:
4141+```rust
4242+let mut conn = redis_pool.get().await?;
4343+timeline::invalidate_timeline_cache(&mut conn, tlid).await?;
4444+```
4545+4646+## Background maintainer
4747+- `database::maintain` (spawned at setup) runs two intervals:
4848+ - Every 60s: delete sessions older than 20 days.
4949+ - Every 300s: prune expired timeline cache entries; check timeline invalidations based on latest timestamps.
5050+5151+## Tests
5252+- `src/tests.rs` covers: pool setup, bloom filter add/exists, timeline cache invalidation.
5353+5454+## Operational cautions
5555+- Pool exhaustion: handle `.get()` errors; avoid panics from `unwrap()`.
5656+- Redis is non-authoritative; always fall back to Postgres on cache miss or bloom filter hit.
5757+- Keep TTLs and thresholds in sync if tuning (`timeline.rs` constants).
+20
notes/Design choices/Rationale/Backend > Error handling and logging.md
···11+## LuminaError
22+- Defined in `server/src/errors.rs`.
33+- Key variants: `DbError(LuminaDbError)`, `Bb8RunErrorPg(bb8::RunError<postgres::Error>)`, `Bb8RunErrorRedis(bb8::RunError<redis::RedisError>)`, auth/registration errors, `SerializationError(String)`, `RocketFaillure(Box<rocket::Error>)`.
44+- Conversions implemented for Rocket, Postgres, Redis, and bb8 run errors.
55+- Guidance: propagate the source error (`?`) to keep context; avoid lossy `to_string()` unless necessary.
66+77+## Logging
88+- Event logging macros `info_elog!`, `warn_elog!`, `error_elog!`, `success_elog!` are used across DB/timeline flows.
99+- `EventLogger` can log to stdout and (optionally) Postgres `logs` table (see `helpers/events.rs`).
1010+- When logging DB failures, prefer structured context (timeline id, page, user) to aid diagnosis.
1111+1212+## Failure-handling patterns
1313+- Pool acquisition: use `?` so `.get()` maps to `LuminaError::Bb8RunErrorPg/Redis`; avoid panics.
1414+- Redis is non-authoritative: on Redis errors, proceed with Postgres path to avoid request failure where possible.
1515+- Bloom filters: treat `BF.EXISTS` positives as hints; always confirm with Postgres.
1616+- Timeline cache: cache misses/failures should fall back to DB fetch; invalidation is best-effort.
1717+1818+## Operational notes
1919+- If Rocket state is missing (e.g., limiter), guards fail-open; verify state wiring at startup.
2020+- Add tracing/metrics around pool usage and cache hit/miss for production readiness.
···11+## Overview
22+- Token-bucket limiter in `server/src/rate_limiter.rs` using in-memory `HashMap` protected by `tokio::sync::Mutex`.
33+- Rocket request guard `RateLimit` pulls `State<GeneralRateLimiter>`; missing state = allow (fail-open).
44+- Separate wrapper types: `GeneralRateLimiter` and `AuthRateLimiter` so Rocket can manage both independently.
55+66+## Defaults / Tuning
77+- Constructor requires `refill_per_second` and `capacity`; no hardcoded defaults. Decide per endpoint.
88+- In-memory only: resets on process restart; not distributed. For multi-node, replace with shared store (e.g., Redis token bucket) or IP hash partitioning.
99+- Keyed by client IP (`Request::client_ip()`); missing IP maps to key "unknown".
1010+1111+## Usage pattern
1212+```rust
1313+// Configure and mount in Rocket managed state
1414+let limiter = GeneralRateLimiter::new(refill_per_second, capacity);
1515+rocket::build().manage(limiter);
1616+1717+// Handler signature adds guard
1818+#[get("/protected")]
1919+async fn protected(_rate: RateLimit) -> &'static str {
2020+ "ok"
2121+}
2222+```
2323+2424+## Gotchas
2525+- Fail-open if the guard cannot fetch state; ensure the limiter is registered in Rocket.
2626+- No per-route tuning baked in; provide distinct limiters via type wrappers if needed.
2727+- Single-threaded bottleneck: Mutex over HashMap is fine for moderate QPS; consider sharding or lock-free structure if contention grows.
···11+# Timeline-carries-most
22+[['Timeline carries most' and the database]]
33+44+One of the busiest tables you'll see is the timeline, containing just some ID's and timestamps.
55+66+| Kind | timeline ID | item ID | Timestamp |
77+| ---------------------------------------- | ----------- | ------- | ------------------ |
88+| `'USER'`, `'DIRECT'`, `'TL'`, `'BUBBLE'` | uuidv4 | uuidv4 | Database timestamp |
99+1010+The `global` timeline, here being `00000000-0000-0000-0000-000000000000` as the only constant-assigned timeline ID. The
1111+user-profiles being the same as their user id counterpart.
1212+1313+This is too vague to actually be able to pull a post, which is why the item forward table exists, combining a UUID and a
1414+string to forward to the right item.
1515+1616+Now I say `item`, not `post` here. This because you might expect only timelines (global, userprofiles) and bubbles (
1717+timelines meant for a specific subject, forming a community within the larger site) in this table, but direct message
1818+threads are actually also saved here.
1919+2020+This means this table might become a little overcrowded, and optimizations such as caching, sharding and mirrorring to
2121+Redis will be needed to keep it somewhat performant, especially since this is essentially a constant hot path. I am
2222+aware.
2323+2424+Which is why we also would need to log every timeline request to be able to identify for example over-requested
2525+timelines.
···11+# The `gleam` language
22+33+I am a big fan of the language. I think it's a great language for
44+writing maintainable and scalable code. I think it's a great language
55+for writing maintainable and scalable code. It's simplicity and the
66+young yet strong community set a good foundation for Lumina.
+4
notes/High-level requirements.md
···11+- Must provide a timeline for 'global' and user-specific (dynamic timelines), viewable on the front end.
22+- Must be able to federate, and users be able to interact with "external" content ^4c6bf0
33+- Must have a responsive web client implementing all 'Must' end-user-features.
44+- Must have a web admin panel
+17
notes/README.md
···11+This is `Lumina/Peonies`'s Obsidian vault for design choices, philosophies and concepts or even psuedocode.
22+## Earlier iterations
33+44+`Lumina:Peonies:itr2` is the current and seemingly final iteration of this project, as of 2026. It uses a Rust server and Gleam/Lustre SPA as web frontend.
55+66+This project has been conceptualised and prototyped into many earlier iterations before, each with different approaches and final result. Some known older iterations had different names, listing a few:
77+88+| Codenamed | About | Introduced |
99+| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
1010+| _Peonies-Lumina_ | factually `Lumina:Peonies:itr1`, had a much bigger approach where multiple backends were explored, including ones based on the BEAM (Gleam-Erlang backend to be precise), is what itr2 draws most inspiration of.<br><br>Having multiple backends with non-matching features proved to be too complicated to maintain or draw straight. | Federation, conceptually |
1111+| _Lumina-Ephew_ | A concept-only iteration that never made it past the drawing board. | Lumina's principles and the global chronological timeline |
1212+| _Ephew_ | A near-complete PHP implementation with a plain HTML+CSS frontend (no scripts), fell apart due to the quickly aging PHP ecosystem at the time. | introducing the idea that 'multiple types of posts can feel native' |
1313+| FNew | A public text-only message pinboard | ~~Criticism, mostly~~ |
1414+1515+1616+1717+The current iteration is a more well-documented and slower approach, giving time to learn and chances to refactor. It also comes in a time where the tech for it is perfect and
+1
notes/Todo's.md
···11+Todo's from these notes are deprecated and replaced by Issues on the repository.
+21
robots.txt
···11+# AI training and TDM are expressly reserved.
22+# See LICENSE file for full legal terms.
33+44+User-agent: *
55+Disallow: /
66+77+# Specifically targeting AI crawlers
88+User-agent: GPTBot
99+Disallow: /
1010+1111+User-agent: ChatGPT-User
1212+Disallow: /
1313+1414+User-agent: Google-Extended
1515+Disallow: /
1616+1717+User-agent: CCBot
1818+Disallow: /
1919+2020+X-Robots-Tag: noai
2121+X-Robots-Tag: noimageai