hydrant

very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall

rust fjall at-protocol atproto indexer

Fork 4

Use mainline libraries / upstream fjall changes etc #16

Closed

ptr.pet 4mo ago

Closed opened by

ptr.pet 4mo ago

It's been a little more complex than expected consuming hydrant because I was expecting I could read the fjall database myself. But, it turns out, Hydrant vendors a variety of forks internally, such as an fjall fork, and more.

As a consumer, interoperation with external libraries would be really appreciated.

I see in #3 there is intent to go the other direction: to fork/vendor jacquard too. I'd really love if we could go the other direction & use upstream libraries wherever possible, and work on getting features we want upstreamed. Ideally there'd be feature flags that might help us consume/use upstream versions instead, too, if the features are non-essential (for example the fjall fork added zstd support, which is good to have, but not essential).

ptr.pet 4mo ago

i don't want to maintain the codebase for different versions of upstream/forks, i don't really see the benefit of that, consumers can just use whatever hydrant uses. in case of jacquard i will probably completely replace it eventually, probably aside from jacquard-repo since we don't really use most of the things it provides and it just becomes an annoying "fight the dep" scenario since i already had to write my own versions of a few things / fork it in the past to fix things because i'd really rather not be locked to upstream...

i could consider a zstd build feature but zstd is pretty integral since its just miles better compared to any other compression and with no downsides so idk if i want to have it be "optional" really.

consumers also arent really expected to consume the fjall db either, its not something i want to "support". i can expose some abstracted methods to help with this, like dumping the db as an impl Iterator or such, if that helps? there is also the user-keyspace feature that adds an api that lets the user create a keyspace in the hydrant db... if you tell me what you really need that'd be more helpful than me having to maintain support for forks / upstream and so on...

jauntywunderkind.bsky.social 4mo ago

I did notice #3 ( https://tangled.org/ptr.pet/hydrant/issues/3 ), which is a strong signal of intent that rather than make use of existing libraries, hydrant is becoming it's own cinematic universe of forked off libraries.

You've clearly done a ton of great work & I understand the pain of working with the constraints of other libraries that aren't quite doing the right thing, especially in the LLM age. I support you going off & doing this great stuff. But this is really so the opposite the direction that I as a user would want from Hydrant. I want something that is a good citizen, where my knowledge & experience using it translates to other popular libraries.

I'm going to close this ticket, & find some other tools for sync/indexing. Thanks for the wonderful library. It's excellent stuff.

ptr.pet 4mo ago

I want something that is a good citizen, where my knowledge & experience using it translates to other popular libraries.

im not sure i understand this part, this is just API concerns no? hydrant already doesnt really expose the underlying libraries too much anyway, aside from the API we know wont change anyway. how does using forks vs vendoring things vs upstream help that? those are simply internal implementation details, not anything to do with the public API interface.

jauntywunderkind.bsky.social 4mo ago

I'm perhaps a bit atypical, in that, I'm not interested in treating a library like a black box. I've already been doing some work playing with fjall, and accessing the data through that. I know Hydrant doesn't really expose that much, but that's an angle I want, that's the level of investment I want in libraries and systems I'm going to invest myself into. That's just a philosophy of involvement that has paid off for me well. I care what's inside libraries, and how adaptable & common the core is.

Using Fjall was a huge draw for me for Hydrant, for example. That's an ecosystem I wanted expose to. And I wanted to be able to read Hydrant data myself in some cases, to build and extend my own tooling beyond what Hydrant maybe offered, doing other direct access systems. It was surprising to see that Fjall is vendored, and that there aren't tickets for upstreaming the features that got added (range deletes, zstd compression, etc).

I don't think that represents most people. I think Hydrant can do just fine with this philosophy. It's incredibly useful. And you're doing great work. I'd closed this ticket because I think this design philosophy is mostly a me issue. For a lot of people, whatever API is offered is enough. But it does make it harder for people to get involved, for interested folk to feel at home when they show up in Hydrant, and they find that it re-defines it's own universe of libraries: that is a cost. But I would not let it slow you down. Hydrant is great, you're doing really interesting work, keep at.