ceres: a small planet in a giant solar system
readme and some filtering on the authorFeed
+200
-43
+9
.env.template
+9
.env.template
.tangled/images/ceres.png
.tangled/images/ceres.png
This is a binary file and will not be displayed.
.tangled/images/profile.png
.tangled/images/profile.png
This is a binary file and will not be displayed.
+40
-2
README.md
+40
-2
README.md
···
1
-
# ceres
1
+
# Ceres
2
+
3
+
> Ceres is a "What if" experiment for me. A true put the rubber to the road and see if we can. We all know Bluesky currently has the largest app on The Atmosphere and it's very expensive to run mostly due to the data stored and queries needed to be ran on it. Ceres is an exploration of can we run a large atproto application like Bluesky cheaper with minimal backfill and some crafty caching without missing features. With that said it may be one day something hosted anyone can use, or it may just stay this mess of trying things out locally.
4
+
5
+
6
+
7
+
2
8
3
-
Ceres is a Bluesky AppView that aims to be easy on resources, does most of what Bluesky's AppView does (hopefully). Aka low storage requirements, and can still make posts, reply, and comment. But search and notifications are not fully in scope yet. May be a full thing one day
9
+
Ceres is a Bluesky AppView that aims to be easy on resources, does most of what Bluesky's AppView does (hopefully) with lower storage requirements, and can still make posts, reply, and comment, etc. But may be missing things like full counts, some links, and search. Not that I won't try, just starting simple.
10
+
11
+
# Ideas/goals to explore
12
+
- 1:1 compatibility with Bluesky's AppView. What I mean by this is you can find and replace `did:web:api.bsky.app` in any social-app fork and get a working AppView. May not be complete, but it will work in some capacity to spec of the `app.bsky.*` lexicons
13
+
- How much can we get away with loading directly with microcosm's tooling. Slingshot and constellation mostly
14
+
- Cache heavily and use the firehose to invalidate cached entries and re hydrate
15
+
- What does the storage requirements look like if the backfill is limited to hard to get counts like followings, posts, relationships, etc.
16
+
- Can we keep proper counts with the firehose once an account has been backfilled.
17
+
- Does it make sense to backfill at a community level, then run on demand for the rest. Aka blacksky.app, northsky.social, eurosky.social, etc.
18
+
- Can you backfill dynamically and on demand. Example, First time the AppView seen a did it fills in the missing bits of it's information and watch for new updates
19
+
- Can this all be done in a single binary with embedded storage? May not be able to serve millions. But if 100s can be set up easily and have the same view do you need to serve millions, or just one for your community?
20
+
- Will this be what finally forces fig to move constellation off of the raspberry pi?
21
+
- A small thing. But I had what I thought was an awesome idea to have a catch all endpoint that prints the `url`, `queries`, and `body` of the request so I can track what endpoint to work on first. Is this a good approach at reverse engineering an already built application
22
+
4
23
5
24
# Work done currently
6
25
- `app.bsky.actor.getProfile` - Just started, and let me tell you. It got hands
7
26
- Should be pulling in everything from the profile lexicon
27
+
- Loads images in via `getBlob`, will also most likely have a CDN flag
8
28
- Pulls in followers since it's easy from constellation
29
+
- labels are not there yet, but plan to pull them in. atproto-proxy requests has a header with the did's for the libelers the user is subscribed to. Plan isgrab via the headers -> query the labelers -> cache heavily
9
30
- Other things like labels, etc are not done yet
31
+
- `app.bsky.actor.getAuthorFeed`
32
+
- It "works" it loads in the feed and maps most things properly like making sure embed views of pictures and links are right
33
+
- Does not have any counts yet like replies, likes, etc.
34
+
- Is not returning filtering as expected. Example on the posts page it does not show the expected feed of posts and reposts, but includes replies since it's walking the collection atm and no caching
35
+
36
+
> Preferences are, well. A pain. I started to implement them since because of the age gated stuff they're kind of a need from any vanilla fork of the Bsky social-app. Long story short. They're stored in the PDS, it's easier to not have your own "prefs" if you are building a Bluesky Appview and to just clear the atproto-proxy header and grab those from the PDS via the social app than have your own. Will most likely yonk these endpoints or return a message like "can't do that".
37
+
10
38
- `app.bsky.actor.getPreferences` - Works, but not as expected. It has it's own internal preferences. Not the ones from your PDS. best to just use a social app that clears the atproto-proxy to get from your own PDS
11
39
- `app.bsky.actor.putPreferences` - [lol](https://github.com/bluesky-social/atproto/issues/4193). Best to just use a bsky-social app fork like https://blacksky.community for running this AppView
40
+
41
+
42
+
# Does it even work right now?
43
+
Kind of? The most feature complete endpoints atm are the `app.bsky.actor.getProfile` and `app.bsky.actor.getAuthorFeed` which allows me to load the profile page in a social-app, like below in a local fork of [blacksky.communty](https://blacksky.communty).
44
+
45
+
You can see that it is loading in my profile pictures, as well as details from my profile record, as well as my posts below that. But some counts like `following` and `posts` are funny numbers, or empty. This is because those are not as easy to grab without a backfill of some sort.
46
+
47
+
The current plan is to get what I can from constellation and slingshot, then see about what else I can backfill for things like relationships between records, counts, and labels. Also a fun tidbit.
48
+
49
+
+150
-41
src/server/xrpc/app_bsky_feed.rs
+150
-41
src/server/xrpc/app_bsky_feed.rs
···
14
14
use jacquard_api::app_bsky::embed::{external, images, video};
15
15
use jacquard_api::app_bsky::feed::get_author_feed::{GetAuthorFeedOutput, GetAuthorFeedRequest};
16
16
use jacquard_api::app_bsky::feed::post::{Post, PostEmbed};
17
-
use jacquard_api::app_bsky::feed::{FeedViewPost, PostView, PostViewEmbed};
18
-
use jacquard_api::com_atproto::repo::list_records::{ListRecords, ListRecordsResponse};
17
+
use jacquard_api::app_bsky::feed::{
18
+
FeedViewPost, FeedViewPostReason, PostView, PostViewEmbed, ReasonPin,
19
+
};
20
+
use jacquard_api::com_atproto::repo::list_records::{self, ListRecords, ListRecordsResponse};
19
21
use jacquard_api::com_atproto::repo::strong_ref::StrongRef;
20
22
use jacquard_axum::{ExtractXrpc, IntoRouter, service_auth::ExtractOptionalServiceAuth};
21
23
use jacquard_common::xrpc;
···
202
204
}
203
205
}
204
206
207
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
208
+
#[allow(clippy::enum_variant_names)]
209
+
enum AuthorFeedFilter {
210
+
PostsWithReplies,
211
+
PostsNoReplies,
212
+
PostsWithMedia,
213
+
PostsAndAuthorThreads,
214
+
PostsWithVideo,
215
+
}
216
+
217
+
impl AuthorFeedFilter {
218
+
fn parse(value: Option<&str>) -> Self {
219
+
match value {
220
+
Some("posts_no_replies") => Self::PostsNoReplies,
221
+
Some("posts_with_media") => Self::PostsWithMedia,
222
+
Some("posts_and_author_threads") => Self::PostsAndAuthorThreads,
223
+
Some("posts_with_video") => Self::PostsWithVideo,
224
+
_ => Self::PostsWithReplies,
225
+
}
226
+
}
227
+
228
+
fn matches(&self, post: &Post<'static>, author_did: &Did<'_>) -> bool {
229
+
match self {
230
+
Self::PostsWithReplies => true,
231
+
Self::PostsNoReplies => post.reply.is_none(),
232
+
Self::PostsAndAuthorThreads => match &post.reply {
233
+
None => true,
234
+
Some(reply) => reply.root.uri.authority().as_ref() == author_did.as_ref(),
235
+
},
236
+
Self::PostsWithMedia => matches!(
237
+
post.embed,
238
+
Some(PostEmbed::Images(_))
239
+
| Some(PostEmbed::Video(_))
240
+
| Some(PostEmbed::RecordWithMedia(_))
241
+
),
242
+
Self::PostsWithVideo => match &post.embed {
243
+
Some(PostEmbed::Video(_)) => true,
244
+
Some(PostEmbed::RecordWithMedia(rwm)) => {
245
+
matches!(rwm.media, RecordWithMediaMedia::Video(_))
246
+
}
247
+
_ => false,
248
+
},
249
+
}
250
+
}
251
+
}
252
+
253
+
async fn build_feed_view_post(
254
+
state: &AppState,
255
+
record: list_records::Record<'static>,
256
+
post: Option<Post<'static>>,
257
+
author: &ProfileViewBasic<'static>,
258
+
pds_url: &Uri<String>,
259
+
did: &Did<'_>,
260
+
reason: Option<FeedViewPostReason<'static>>,
261
+
) -> FeedViewPost<'static> {
262
+
let embed = match post.and_then(|p| p.embed) {
263
+
Some(e) => post_embed_to_view(e, state, pds_url, did).await,
264
+
None => None,
265
+
};
266
+
let post_view = PostView {
267
+
author: author.clone(),
268
+
bookmark_count: None,
269
+
cid: record.cid,
270
+
debug: None,
271
+
embed,
272
+
indexed_at: Datetime::now(),
273
+
labels: None,
274
+
like_count: None,
275
+
quote_count: None,
276
+
record: record.value,
277
+
reply_count: None,
278
+
repost_count: None,
279
+
threadgate: None,
280
+
uri: record.uri,
281
+
viewer: None,
282
+
extra_data: Default::default(),
283
+
};
284
+
FeedViewPost {
285
+
feed_context: None,
286
+
post: post_view,
287
+
reason,
288
+
reply: None,
289
+
req_id: None,
290
+
extra_data: Default::default(),
291
+
}
292
+
}
293
+
294
+
async fn fetch_pinned_feed_view(
295
+
state: &AppState,
296
+
pinned: &StrongRef<'static>,
297
+
author: &ProfileViewBasic<'static>,
298
+
pds_url: &Uri<String>,
299
+
did: &Did<'_>,
300
+
) -> Option<FeedViewPost<'static>> {
301
+
let fetched = match state.agent.fetch_record_slingshot(&pinned.uri).await {
302
+
Ok(f) => f,
303
+
Err(err) => {
304
+
log::warn!("fetch pinned post {}: {err:#}", pinned.uri);
305
+
return None;
306
+
}
307
+
};
308
+
309
+
let post = jacquard::common::from_data_owned::<Post<'static>>(fetched.value.clone()).ok();
310
+
let record = list_records::Record {
311
+
cid: fetched.cid.unwrap_or_else(|| pinned.cid.clone()),
312
+
uri: fetched.uri,
313
+
value: fetched.value,
314
+
extra_data: Default::default(),
315
+
};
316
+
let reason = Some(FeedViewPostReason::ReasonPin(Box::new(ReasonPin {
317
+
extra_data: Default::default(),
318
+
})));
319
+
Some(build_feed_view_post(state, record, post, author, pds_url, did, reason).await)
320
+
}
321
+
205
322
pub async fn get_author_feed(
206
323
State(state): State<AppState>,
207
324
ExtractOptionalServiceAuth(_auth): ExtractOptionalServiceAuth,
208
325
ExtractXrpc(req): ExtractXrpc<GetAuthorFeedRequest>,
209
326
) -> Result<Json<GetAuthorFeedOutput<'static>>, XrpcErrorResponse> {
210
327
info!(
211
-
"get_author_feed actor={} limit={:?}",
328
+
"get_author_feed actor={} limit={:?} filter={:?} include_pins={:?}",
212
329
req.actor.as_ref(),
213
-
req.limit
214
-
);
215
-
debug!(
216
-
"get_author_feed filter={:?} include_pins={:?} cursor={:?}",
217
-
req.filter, req.include_pins, req.cursor
330
+
req.limit,
331
+
req.filter,
332
+
req.include_pins,
218
333
);
334
+
debug!("get_author_feed cursor={:?}", req.cursor);
335
+
336
+
let filter = AuthorFeedFilter::parse(req.filter.as_deref());
337
+
let include_pins = req.include_pins.unwrap_or(false);
338
+
let is_first_page = req.cursor.is_none();
219
339
220
340
let ctx = resolve_author_context(&state, req.actor).await?;
221
341
···
265
385
266
386
let mut feed = Vec::with_capacity(output.records.len());
267
387
for record in output.records {
268
-
let embed_source = jacquard::common::from_data_owned::<Post<'static>>(record.value.clone())
269
-
.ok()
270
-
.and_then(|post| post.embed);
271
-
let embed = match embed_source {
272
-
Some(e) => post_embed_to_view(e, &state, &ctx.pds_url, &ctx.did).await,
273
-
None => None,
274
-
};
275
-
let post = PostView {
276
-
author: author.clone(),
277
-
bookmark_count: None,
278
-
cid: record.cid,
279
-
debug: None,
280
-
embed,
281
-
indexed_at: Datetime::now(),
282
-
labels: None,
283
-
like_count: None,
284
-
quote_count: None,
285
-
record: record.value,
286
-
reply_count: None,
287
-
repost_count: None,
288
-
threadgate: None,
289
-
uri: record.uri,
290
-
viewer: None,
291
-
extra_data: Default::default(),
292
-
};
293
-
feed.push(FeedViewPost {
294
-
feed_context: None,
295
-
post,
296
-
reason: None,
297
-
reply: None,
298
-
req_id: None,
299
-
extra_data: Default::default(),
300
-
});
388
+
let post = jacquard::common::from_data_owned::<Post<'static>>(record.value.clone()).ok();
389
+
let matches_filter = post
390
+
.as_ref()
391
+
.map(|p| filter.matches(p, &ctx.did))
392
+
.unwrap_or(false);
393
+
if !matches_filter {
394
+
continue;
395
+
}
396
+
feed.push(
397
+
build_feed_view_post(&state, record, post, &author, &ctx.pds_url, &ctx.did, None).await,
398
+
);
399
+
}
400
+
401
+
if include_pins
402
+
&& is_first_page
403
+
&& let Some(pinned) = ctx.profile.pinned_post.clone()
404
+
&& let Some(pinned_view) =
405
+
fetch_pinned_feed_view(&state, &pinned, &author, &ctx.pds_url, &ctx.did).await
406
+
{
407
+
let pinned_uri = pinned_view.post.uri.as_str().to_string();
408
+
feed.retain(|item| item.post.uri.as_str() != pinned_uri);
409
+
feed.insert(0, pinned_view);
301
410
}
302
411
303
412
Ok(Json(GetAuthorFeedOutput {