···11-test/
22-.DS_STORE
33-jacquard/
44-binaries/
55-# Generated by Cargo
66-# will have compiled files and executables
77-debug
88-target
99-1010-# These are backup files generated by rustfmt
1111-**/*.rs.bk
1212-1313-# MSVC Windows builds of rustc generate these, which store debugging information
1414-*.pdb
1515-1616-# Generated by cargo mutants
1717-# Contains mutation testing data
1818-**/mutants.out*/
1919-2020-# RustRover
2121-# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
2222-# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
2323-# and can be added to the global gitignore or merged into this file. For a more nuclear
2424-# option (not recommended) you can uncomment the following to ignore the entire idea folder.
2525-#.idea/
···11-# Wisp CLI
22-33-A command-line tool for deploying static sites to your AT Protocol repo to be served on [wisp.place](https://wisp.place), an AT indexer to serve such sites.
44-55-## Why?
66-77-The PDS serves as a way to verfiably, cryptographically prove that you own your site. That it was you (or at least someone who controls your account) who uploaded it. It is also a manifest of each file in the site to ensure file integrity. Keeping hosting seperate ensures that you could move your site across other servers or even serverless solutions to ensure speedy delievery while keeping it backed by an absolute source of truth being the manifest record and the blobs of each file in your repo.
88-99-## Features
1010-1111-- Deploy static sites directly to your AT Protocol repo
1212-- Supports both OAuth and app password authentication
1313-- Preserves directory structure and file integrity
1414-1515-## Soon
1616-1717--- Host sites
1818--- Manage and delete sites
1919--- Metrics and logs for self hosting.
2020-2121-## Installation
2222-2323-### From Source
2424-2525-```bash
2626-cargo build --release
2727-```
2828-2929-Check out the build scripts for cross complation using nix-shell.
3030-3131-The binary will be available at `target/release/wisp-cli`.
3232-3333-## Usage
3434-3535-### Commands
3636-3737-The CLI supports three main commands:
3838-- **deploy**: Upload a site to your PDS (default command)
3939-- **pull**: Download a site from a PDS to a local directory
4040-- **serve**: Serve a site locally with real-time firehose updates
4141-4242-### Basic Deployment
4343-4444-Deploy the current directory:
4545-4646-```bash
4747-wisp-cli nekomimi.pet --path . --site my-site
4848-```
4949-5050-Deploy a specific directory:
5151-5252-```bash
5353-wisp-cli alice.bsky.social --path ./dist/ --site my-site
5454-```
5555-5656-Or use the explicit `deploy` subcommand:
5757-5858-```bash
5959-wisp-cli deploy alice.bsky.social --path ./dist/ --site my-site
6060-```
6161-6262-### Pull a Site
6363-6464-Download a site from a PDS to a local directory:
6565-6666-```bash
6767-wisp-cli pull alice.bsky.social --site my-site --path ./downloaded-site
6868-```
6969-7070-This will download all files from the site to the specified directory.
7171-7272-### Serve a Site Locally
7373-7474-Serve a site locally with real-time updates from the firehose:
7575-7676-```bash
7777-wisp-cli serve alice.bsky.social --site my-site --path ./site --port 8080
7878-```
7979-8080-This will:
8181-1. Download the site to the specified path
8282-2. Start a local server on the specified port (default: 8080)
8383-3. Watch the firehose for updates and automatically reload files when changed
8484-8585-### Authentication Methods
8686-8787-#### OAuth (Recommended)
8888-8989-By default, the CLI uses OAuth authentication with a local loopback server:
9090-9191-```bash
9292-wisp-cli alice.bsky.social --path ./my-site --site my-site
9393-```
9494-9595-This will:
9696-1. Open your browser for authentication
9797-2. Save the session to a file (default: `/tmp/wisp-oauth-session.json`)
9898-3. Reuse the session for future deployments
9999-100100-Specify a custom session file location:
101101-102102-```bash
103103-wisp-cli alice.bsky.social --path ./my-site --site my-site --store ~/.wisp-session.json
104104-```
105105-106106-#### App Password
107107-108108-For headless environments or CI/CD, use an app password:
109109-110110-```bash
111111-wisp-cli alice.bsky.social --path ./my-site --site my-site --password YOUR_APP_PASSWORD
112112-```
113113-114114-**Note:** When using `--password`, the `--store` option is ignored.
115115-116116-## Command-Line Options
117117-118118-### Deploy Command
119119-120120-```
121121-wisp-cli [deploy] [OPTIONS] <INPUT>
122122-123123-Arguments:
124124- <INPUT> Handle (e.g., alice.bsky.social), DID, or PDS URL
125125-126126-Options:
127127- -p, --path <PATH> Path to the directory containing your static site [default: .]
128128- -s, --site <SITE> Site name (defaults to directory name)
129129- --store <STORE> Path to auth store file (only used with OAuth) [default: /tmp/wisp-oauth-session.json]
130130- --password <PASSWORD> App Password for authentication (alternative to OAuth)
131131- --directory Enable directory listing mode for paths without index files
132132- --spa Enable SPA mode (serve index.html for all routes)
133133- -y, --yes Skip confirmation prompts (automatically accept warnings)
134134- -h, --help Print help
135135- -V, --version Print version
136136-```
137137-138138-### Pull Command
139139-140140-```
141141-wisp-cli pull [OPTIONS] --site <SITE> <INPUT>
142142-143143-Arguments:
144144- <INPUT> Handle (e.g., alice.bsky.social) or DID
145145-146146-Options:
147147- -s, --site <SITE> Site name (record key)
148148- -p, --path <PATH> Output directory for the downloaded site [default: .]
149149- -h, --help Print help
150150-```
151151-152152-### Serve Command
153153-154154-```
155155-wisp-cli serve [OPTIONS] --site <SITE> <INPUT>
156156-157157-Arguments:
158158- <INPUT> Handle (e.g., alice.bsky.social) or DID
159159-160160-Options:
161161- -s, --site <SITE> Site name (record key)
162162- -p, --path <PATH> Output directory for the site files [default: .]
163163- -P, --port <PORT> Port to serve on [default: 8080]
164164- -h, --help Print help
165165-```
166166-167167-## How It Works
168168-169169-1. **Authentication**: Authenticates using OAuth or app password
170170-2. **File Processing**:
171171- - Recursively walks the directory tree
172172- - Skips hidden files (starting with `.`)
173173- - Detects MIME types automatically
174174- - Compresses files with gzip
175175- - Base64 encodes compressed content
176176-3. **Upload**:
177177- - Uploads files as blobs to your PDS
178178- - Processes up to 5 files concurrently
179179- - Creates a `place.wisp.fs` record with the site manifest
180180-4. **Deployment**: Site is immediately available at `https://sites.wisp.place/{did}/{site-name}`
181181-182182-## File Processing
183183-184184-All files are automatically:
185185-186186-- **Compressed** with gzip (level 9)
187187-- **Base64 encoded** to bypass PDS content sniffing
188188-- **Uploaded** as `application/octet-stream` blobs
189189-- **Stored** with original MIME type metadata
190190-191191-The hosting service automatically decompresses non HTML/CSS/JS files when serving them.
192192-193193-## Limitations
194194-195195-- **Max file size**: 100MB per file (after compression) (this is a PDS limit, but not enforced by the CLI in case yours is higher)
196196-- **Max file count**: 2000 files
197197-- **Site name** must follow AT Protocol rkey format rules (alphanumeric, hyphens, underscores)
198198-199199-## Deploy with CI/CD
200200-201201-### GitHub Actions
202202-203203-```yaml
204204-name: Deploy to Wisp
205205-on:
206206- push:
207207- branches: [main]
208208-209209-jobs:
210210- deploy:
211211- runs-on: ubuntu-latest
212212- steps:
213213- - uses: actions/checkout@v3
214214-215215- - name: Setup Node
216216- uses: actions/setup-node@v3
217217- with:
218218- node-version: '25'
219219-220220- - name: Install dependencies
221221- run: npm install
222222-223223- - name: Build site
224224- run: npm run build
225225-226226- - name: Download Wisp CLI
227227- run: |
228228- curl -L https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-linux -o wisp-cli
229229- chmod +x wisp-cli
230230-231231- - name: Deploy to Wisp
232232- env:
233233- WISP_APP_PASSWORD: ${{ secrets.WISP_APP_PASSWORD }}
234234- run: |
235235- ./wisp-cli alice.bsky.social \
236236- --path ./dist \
237237- --site my-site \
238238- --password "$WISP_APP_PASSWORD"
239239-```
240240-241241-### Tangled.org
242242-243243-```yaml
244244-when:
245245- - event: ['push']
246246- branch: ['main']
247247- - event: ['manual']
248248-249249-engine: 'nixery'
250250-251251-clone:
252252- skip: false
253253- depth: 1
254254- submodules: false
255255-256256-dependencies:
257257- nixpkgs:
258258- - nodejs
259259- - coreutils
260260- - curl
261261- github:NixOS/nixpkgs/nixpkgs-unstable:
262262- - bun
263263-264264-environment:
265265- SITE_PATH: 'dist'
266266- SITE_NAME: 'my-site'
267267- WISP_HANDLE: 'your-handle.bsky.social'
268268-269269-steps:
270270- - name: build site
271271- command: |
272272- export PATH="$HOME/.nix-profile/bin:$PATH"
273273-274274- # regenerate lockfile
275275- rm package-lock.json bun.lock
276276- bun install @rolldown/binding-linux-arm64-gnu --save-optional
277277- bun install
278278-279279- # build with vite
280280- bun node_modules/.bin/vite build
281281-282282- - name: deploy to wisp
283283- command: |
284284- # Download Wisp CLI
285285- curl https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-linux -o wisp-cli
286286- chmod +x wisp-cli
287287-288288- # Deploy to Wisp
289289- ./wisp-cli \
290290- "$WISP_HANDLE" \
291291- --path "$SITE_PATH" \
292292- --site "$SITE_NAME" \
293293- --password "$WISP_APP_PASSWORD"
294294-```
295295-296296-### Generic Shell Script
297297-298298-```bash
299299-# Use app password from environment variable
300300-wisp-cli alice.bsky.social --path ./dist --site my-site --password "$WISP_APP_PASSWORD"
301301-```
302302-303303-## Output
304304-305305-Upon successful deployment, you'll see:
306306-307307-```
308308-Deployed site 'my-site': at://did:plc:abc123xyz/place.wisp.fs/my-site
309309-Available at: https://sites.wisp.place/did:plc:abc123xyz/my-site
310310-```
311311-312312-### Dependencies
313313-314314-- **jacquard**: AT Protocol client library
315315-- **clap**: Command-line argument parsing
316316-- **tokio**: Async runtime
317317-- **flate2**: Gzip compression
318318-- **base64**: Base64 encoding
319319-- **walkdir**: Directory traversal
320320-- **mime_guess**: MIME type detection
321321-322322-## License
323323-324324-MIT License
325325-326326-## Contributing
327327-328328-Just don't give me entirely claude slop especailly not in the PR description itself. You should be responsible for code you submit and aware of what it even is you're submitting.
329329-330330-## Links
331331-332332-- **Website**: https://wisp.place
333333-- **Main Repository**: https://tangled.org/@nekomimi.pet/wisp.place-monorepo
334334-- **AT Protocol**: https://atproto.com
335335-- **Jacquard Library**: https://tangled.org/@nonbinary.computer/jacquard
336336-337337-## Support
338338-339339-For issues and questions:
340340-- Check the main wisp.place documentation
341341-- Open an issue in the main repository
···11-// @generated by jacquard-lexicon. DO NOT EDIT.
22-//
33-// This file was automatically generated from Lexicon schemas.
44-// Any manual changes will be overwritten on the next regeneration.
55-66-/// Marker type indicating a builder field has been set
77-pub struct Set<T>(pub T);
88-impl<T> Set<T> {
99- /// Extract the inner value
1010- #[inline]
1111- pub fn into_inner(self) -> T {
1212- self.0
1313- }
1414-}
1515-1616-/// Marker type indicating a builder field has not been set
1717-pub struct Unset;
1818-/// Trait indicating a builder field is set (has a value)
1919-#[rustversion::attr(
2020- since(1.78.0),
2121- diagnostic::on_unimplemented(
2222- message = "the field `{Self}` was not set, but this method requires it to be set",
2323- label = "the field `{Self}` was not set"
2424- )
2525-)]
2626-pub trait IsSet: private::Sealed {}
2727-/// Trait indicating a builder field is unset (no value yet)
2828-#[rustversion::attr(
2929- since(1.78.0),
3030- diagnostic::on_unimplemented(
3131- message = "the field `{Self}` was already set, but this method requires it to be unset",
3232- label = "the field `{Self}` was already set"
3333- )
3434-)]
3535-pub trait IsUnset: private::Sealed {}
3636-impl<T> IsSet for Set<T> {}
3737-impl IsUnset for Unset {}
3838-mod private {
3939- /// Sealed trait to prevent external implementations
4040- pub trait Sealed {}
4141- impl<T> Sealed for super::Set<T> {}
4242- impl Sealed for super::Unset {}
4343-}
-11
rust-cli/crates/lexicons/src/lib.rs
···11-extern crate alloc;
22-33-// @generated by jacquard-lexicon. DO NOT EDIT.
44-//
55-// This file was automatically generated from Lexicon schemas.
66-// Any manual changes will be overwritten on the next regeneration.
77-88-pub mod builder_types;
99-1010-#[cfg(feature = "place_wisp")]
1111-pub mod place_wisp;
-8
rust-cli/crates/lexicons/src/place_wisp.rs
···11-// @generated by jacquard-lexicon. DO NOT EDIT.
22-//
33-// This file was automatically generated from Lexicon schemas.
44-// Any manual changes will be overwritten on the next regeneration.
55-66-pub mod fs;
77-pub mod settings;
88-pub mod subfs;
-1490
rust-cli/crates/lexicons/src/place_wisp/fs.rs
···11-// @generated by jacquard-lexicon. DO NOT EDIT.
22-//
33-// Lexicon: place.wisp.fs
44-//
55-// This file was automatically generated from Lexicon schemas.
66-// Any manual changes will be overwritten on the next regeneration.
77-88-#[jacquard_derive::lexicon]
99-#[derive(
1010- serde::Serialize,
1111- serde::Deserialize,
1212- Debug,
1313- Clone,
1414- PartialEq,
1515- Eq,
1616- jacquard_derive::IntoStatic
1717-)]
1818-#[serde(rename_all = "camelCase")]
1919-pub struct Directory<'a> {
2020- #[serde(borrow)]
2121- pub entries: Vec<crate::place_wisp::fs::Entry<'a>>,
2222- #[serde(borrow)]
2323- pub r#type: jacquard_common::CowStr<'a>,
2424-}
2525-2626-pub mod directory_state {
2727-2828- pub use crate::builder_types::{Set, Unset, IsSet, IsUnset};
2929- #[allow(unused)]
3030- use ::core::marker::PhantomData;
3131- mod sealed {
3232- pub trait Sealed {}
3333- }
3434- /// State trait tracking which required fields have been set
3535- pub trait State: sealed::Sealed {
3636- type Type;
3737- type Entries;
3838- }
3939- /// Empty state - all required fields are unset
4040- pub struct Empty(());
4141- impl sealed::Sealed for Empty {}
4242- impl State for Empty {
4343- type Type = Unset;
4444- type Entries = Unset;
4545- }
4646- ///State transition - sets the `type` field to Set
4747- pub struct SetType<S: State = Empty>(PhantomData<fn() -> S>);
4848- impl<S: State> sealed::Sealed for SetType<S> {}
4949- impl<S: State> State for SetType<S> {
5050- type Type = Set<members::r#type>;
5151- type Entries = S::Entries;
5252- }
5353- ///State transition - sets the `entries` field to Set
5454- pub struct SetEntries<S: State = Empty>(PhantomData<fn() -> S>);
5555- impl<S: State> sealed::Sealed for SetEntries<S> {}
5656- impl<S: State> State for SetEntries<S> {
5757- type Type = S::Type;
5858- type Entries = Set<members::entries>;
5959- }
6060- /// Marker types for field names
6161- #[allow(non_camel_case_types)]
6262- pub mod members {
6363- ///Marker type for the `type` field
6464- pub struct r#type(());
6565- ///Marker type for the `entries` field
6666- pub struct entries(());
6767- }
6868-}
6969-7070-/// Builder for constructing an instance of this type
7171-pub struct DirectoryBuilder<'a, S: directory_state::State> {
7272- _phantom_state: ::core::marker::PhantomData<fn() -> S>,
7373- __unsafe_private_named: (
7474- ::core::option::Option<Vec<crate::place_wisp::fs::Entry<'a>>>,
7575- ::core::option::Option<jacquard_common::CowStr<'a>>,
7676- ),
7777- _phantom: ::core::marker::PhantomData<&'a ()>,
7878-}
7979-8080-impl<'a> Directory<'a> {
8181- /// Create a new builder for this type
8282- pub fn new() -> DirectoryBuilder<'a, directory_state::Empty> {
8383- DirectoryBuilder::new()
8484- }
8585-}
8686-8787-impl<'a> DirectoryBuilder<'a, directory_state::Empty> {
8888- /// Create a new builder with all fields unset
8989- pub fn new() -> Self {
9090- DirectoryBuilder {
9191- _phantom_state: ::core::marker::PhantomData,
9292- __unsafe_private_named: (None, None),
9393- _phantom: ::core::marker::PhantomData,
9494- }
9595- }
9696-}
9797-9898-impl<'a, S> DirectoryBuilder<'a, S>
9999-where
100100- S: directory_state::State,
101101- S::Entries: directory_state::IsUnset,
102102-{
103103- /// Set the `entries` field (required)
104104- pub fn entries(
105105- mut self,
106106- value: impl Into<Vec<crate::place_wisp::fs::Entry<'a>>>,
107107- ) -> DirectoryBuilder<'a, directory_state::SetEntries<S>> {
108108- self.__unsafe_private_named.0 = ::core::option::Option::Some(value.into());
109109- DirectoryBuilder {
110110- _phantom_state: ::core::marker::PhantomData,
111111- __unsafe_private_named: self.__unsafe_private_named,
112112- _phantom: ::core::marker::PhantomData,
113113- }
114114- }
115115-}
116116-117117-impl<'a, S> DirectoryBuilder<'a, S>
118118-where
119119- S: directory_state::State,
120120- S::Type: directory_state::IsUnset,
121121-{
122122- /// Set the `type` field (required)
123123- pub fn r#type(
124124- mut self,
125125- value: impl Into<jacquard_common::CowStr<'a>>,
126126- ) -> DirectoryBuilder<'a, directory_state::SetType<S>> {
127127- self.__unsafe_private_named.1 = ::core::option::Option::Some(value.into());
128128- DirectoryBuilder {
129129- _phantom_state: ::core::marker::PhantomData,
130130- __unsafe_private_named: self.__unsafe_private_named,
131131- _phantom: ::core::marker::PhantomData,
132132- }
133133- }
134134-}
135135-136136-impl<'a, S> DirectoryBuilder<'a, S>
137137-where
138138- S: directory_state::State,
139139- S::Type: directory_state::IsSet,
140140- S::Entries: directory_state::IsSet,
141141-{
142142- /// Build the final struct
143143- pub fn build(self) -> Directory<'a> {
144144- Directory {
145145- entries: self.__unsafe_private_named.0.unwrap(),
146146- r#type: self.__unsafe_private_named.1.unwrap(),
147147- extra_data: Default::default(),
148148- }
149149- }
150150- /// Build the final struct with custom extra_data
151151- pub fn build_with_data(
152152- self,
153153- extra_data: std::collections::BTreeMap<
154154- jacquard_common::smol_str::SmolStr,
155155- jacquard_common::types::value::Data<'a>,
156156- >,
157157- ) -> Directory<'a> {
158158- Directory {
159159- entries: self.__unsafe_private_named.0.unwrap(),
160160- r#type: self.__unsafe_private_named.1.unwrap(),
161161- extra_data: Some(extra_data),
162162- }
163163- }
164164-}
165165-166166-fn lexicon_doc_place_wisp_fs() -> ::jacquard_lexicon::lexicon::LexiconDoc<'static> {
167167- ::jacquard_lexicon::lexicon::LexiconDoc {
168168- lexicon: ::jacquard_lexicon::lexicon::Lexicon::Lexicon1,
169169- id: ::jacquard_common::CowStr::new_static("place.wisp.fs"),
170170- revision: None,
171171- description: None,
172172- defs: {
173173- let mut map = ::std::collections::BTreeMap::new();
174174- map.insert(
175175- ::jacquard_common::smol_str::SmolStr::new_static("directory"),
176176- ::jacquard_lexicon::lexicon::LexUserType::Object(::jacquard_lexicon::lexicon::LexObject {
177177- description: None,
178178- required: Some(
179179- vec![
180180- ::jacquard_common::smol_str::SmolStr::new_static("type"),
181181- ::jacquard_common::smol_str::SmolStr::new_static("entries")
182182- ],
183183- ),
184184- nullable: None,
185185- properties: {
186186- #[allow(unused_mut)]
187187- let mut map = ::std::collections::BTreeMap::new();
188188- map.insert(
189189- ::jacquard_common::smol_str::SmolStr::new_static("entries"),
190190- ::jacquard_lexicon::lexicon::LexObjectProperty::Array(::jacquard_lexicon::lexicon::LexArray {
191191- description: None,
192192- items: ::jacquard_lexicon::lexicon::LexArrayItem::Ref(::jacquard_lexicon::lexicon::LexRef {
193193- description: None,
194194- r#ref: ::jacquard_common::CowStr::new_static("#entry"),
195195- }),
196196- min_length: None,
197197- max_length: Some(500usize),
198198- }),
199199- );
200200- map.insert(
201201- ::jacquard_common::smol_str::SmolStr::new_static("type"),
202202- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
203203- description: None,
204204- format: None,
205205- default: None,
206206- min_length: None,
207207- max_length: None,
208208- min_graphemes: None,
209209- max_graphemes: None,
210210- r#enum: None,
211211- r#const: None,
212212- known_values: None,
213213- }),
214214- );
215215- map
216216- },
217217- }),
218218- );
219219- map.insert(
220220- ::jacquard_common::smol_str::SmolStr::new_static("entry"),
221221- ::jacquard_lexicon::lexicon::LexUserType::Object(::jacquard_lexicon::lexicon::LexObject {
222222- description: None,
223223- required: Some(
224224- vec![
225225- ::jacquard_common::smol_str::SmolStr::new_static("name"),
226226- ::jacquard_common::smol_str::SmolStr::new_static("node")
227227- ],
228228- ),
229229- nullable: None,
230230- properties: {
231231- #[allow(unused_mut)]
232232- let mut map = ::std::collections::BTreeMap::new();
233233- map.insert(
234234- ::jacquard_common::smol_str::SmolStr::new_static("name"),
235235- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
236236- description: None,
237237- format: None,
238238- default: None,
239239- min_length: None,
240240- max_length: Some(255usize),
241241- min_graphemes: None,
242242- max_graphemes: None,
243243- r#enum: None,
244244- r#const: None,
245245- known_values: None,
246246- }),
247247- );
248248- map.insert(
249249- ::jacquard_common::smol_str::SmolStr::new_static("node"),
250250- ::jacquard_lexicon::lexicon::LexObjectProperty::Union(::jacquard_lexicon::lexicon::LexRefUnion {
251251- description: None,
252252- refs: vec![
253253- ::jacquard_common::CowStr::new_static("#file"),
254254- ::jacquard_common::CowStr::new_static("#directory"),
255255- ::jacquard_common::CowStr::new_static("#subfs")
256256- ],
257257- closed: None,
258258- }),
259259- );
260260- map
261261- },
262262- }),
263263- );
264264- map.insert(
265265- ::jacquard_common::smol_str::SmolStr::new_static("file"),
266266- ::jacquard_lexicon::lexicon::LexUserType::Object(::jacquard_lexicon::lexicon::LexObject {
267267- description: None,
268268- required: Some(
269269- vec![
270270- ::jacquard_common::smol_str::SmolStr::new_static("type"),
271271- ::jacquard_common::smol_str::SmolStr::new_static("blob")
272272- ],
273273- ),
274274- nullable: None,
275275- properties: {
276276- #[allow(unused_mut)]
277277- let mut map = ::std::collections::BTreeMap::new();
278278- map.insert(
279279- ::jacquard_common::smol_str::SmolStr::new_static("base64"),
280280- ::jacquard_lexicon::lexicon::LexObjectProperty::Boolean(::jacquard_lexicon::lexicon::LexBoolean {
281281- description: None,
282282- default: None,
283283- r#const: None,
284284- }),
285285- );
286286- map.insert(
287287- ::jacquard_common::smol_str::SmolStr::new_static("blob"),
288288- ::jacquard_lexicon::lexicon::LexObjectProperty::Blob(::jacquard_lexicon::lexicon::LexBlob {
289289- description: None,
290290- accept: None,
291291- max_size: None,
292292- }),
293293- );
294294- map.insert(
295295- ::jacquard_common::smol_str::SmolStr::new_static("encoding"),
296296- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
297297- description: Some(
298298- ::jacquard_common::CowStr::new_static(
299299- "Content encoding (e.g., gzip for compressed files)",
300300- ),
301301- ),
302302- format: None,
303303- default: None,
304304- min_length: None,
305305- max_length: None,
306306- min_graphemes: None,
307307- max_graphemes: None,
308308- r#enum: None,
309309- r#const: None,
310310- known_values: None,
311311- }),
312312- );
313313- map.insert(
314314- ::jacquard_common::smol_str::SmolStr::new_static("mimeType"),
315315- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
316316- description: Some(
317317- ::jacquard_common::CowStr::new_static(
318318- "Original MIME type before compression",
319319- ),
320320- ),
321321- format: None,
322322- default: None,
323323- min_length: None,
324324- max_length: None,
325325- min_graphemes: None,
326326- max_graphemes: None,
327327- r#enum: None,
328328- r#const: None,
329329- known_values: None,
330330- }),
331331- );
332332- map.insert(
333333- ::jacquard_common::smol_str::SmolStr::new_static("type"),
334334- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
335335- description: None,
336336- format: None,
337337- default: None,
338338- min_length: None,
339339- max_length: None,
340340- min_graphemes: None,
341341- max_graphemes: None,
342342- r#enum: None,
343343- r#const: None,
344344- known_values: None,
345345- }),
346346- );
347347- map
348348- },
349349- }),
350350- );
351351- map.insert(
352352- ::jacquard_common::smol_str::SmolStr::new_static("main"),
353353- ::jacquard_lexicon::lexicon::LexUserType::Record(::jacquard_lexicon::lexicon::LexRecord {
354354- description: Some(
355355- ::jacquard_common::CowStr::new_static(
356356- "Virtual filesystem manifest for a Wisp site",
357357- ),
358358- ),
359359- key: None,
360360- record: ::jacquard_lexicon::lexicon::LexRecordRecord::Object(::jacquard_lexicon::lexicon::LexObject {
361361- description: None,
362362- required: Some(
363363- vec![
364364- ::jacquard_common::smol_str::SmolStr::new_static("site"),
365365- ::jacquard_common::smol_str::SmolStr::new_static("root"),
366366- ::jacquard_common::smol_str::SmolStr::new_static("createdAt")
367367- ],
368368- ),
369369- nullable: None,
370370- properties: {
371371- #[allow(unused_mut)]
372372- let mut map = ::std::collections::BTreeMap::new();
373373- map.insert(
374374- ::jacquard_common::smol_str::SmolStr::new_static(
375375- "createdAt",
376376- ),
377377- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
378378- description: None,
379379- format: Some(
380380- ::jacquard_lexicon::lexicon::LexStringFormat::Datetime,
381381- ),
382382- default: None,
383383- min_length: None,
384384- max_length: None,
385385- min_graphemes: None,
386386- max_graphemes: None,
387387- r#enum: None,
388388- r#const: None,
389389- known_values: None,
390390- }),
391391- );
392392- map.insert(
393393- ::jacquard_common::smol_str::SmolStr::new_static(
394394- "fileCount",
395395- ),
396396- ::jacquard_lexicon::lexicon::LexObjectProperty::Integer(::jacquard_lexicon::lexicon::LexInteger {
397397- description: None,
398398- default: None,
399399- minimum: Some(0i64),
400400- maximum: Some(1000i64),
401401- r#enum: None,
402402- r#const: None,
403403- }),
404404- );
405405- map.insert(
406406- ::jacquard_common::smol_str::SmolStr::new_static("root"),
407407- ::jacquard_lexicon::lexicon::LexObjectProperty::Ref(::jacquard_lexicon::lexicon::LexRef {
408408- description: None,
409409- r#ref: ::jacquard_common::CowStr::new_static("#directory"),
410410- }),
411411- );
412412- map.insert(
413413- ::jacquard_common::smol_str::SmolStr::new_static("site"),
414414- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
415415- description: None,
416416- format: None,
417417- default: None,
418418- min_length: None,
419419- max_length: None,
420420- min_graphemes: None,
421421- max_graphemes: None,
422422- r#enum: None,
423423- r#const: None,
424424- known_values: None,
425425- }),
426426- );
427427- map
428428- },
429429- }),
430430- }),
431431- );
432432- map.insert(
433433- ::jacquard_common::smol_str::SmolStr::new_static("subfs"),
434434- ::jacquard_lexicon::lexicon::LexUserType::Object(::jacquard_lexicon::lexicon::LexObject {
435435- description: None,
436436- required: Some(
437437- vec![
438438- ::jacquard_common::smol_str::SmolStr::new_static("type"),
439439- ::jacquard_common::smol_str::SmolStr::new_static("subject")
440440- ],
441441- ),
442442- nullable: None,
443443- properties: {
444444- #[allow(unused_mut)]
445445- let mut map = ::std::collections::BTreeMap::new();
446446- map.insert(
447447- ::jacquard_common::smol_str::SmolStr::new_static("flat"),
448448- ::jacquard_lexicon::lexicon::LexObjectProperty::Boolean(::jacquard_lexicon::lexicon::LexBoolean {
449449- description: None,
450450- default: None,
451451- r#const: None,
452452- }),
453453- );
454454- map.insert(
455455- ::jacquard_common::smol_str::SmolStr::new_static("subject"),
456456- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
457457- description: Some(
458458- ::jacquard_common::CowStr::new_static(
459459- "AT-URI pointing to a place.wisp.subfs record containing this subtree.",
460460- ),
461461- ),
462462- format: Some(
463463- ::jacquard_lexicon::lexicon::LexStringFormat::AtUri,
464464- ),
465465- default: None,
466466- min_length: None,
467467- max_length: None,
468468- min_graphemes: None,
469469- max_graphemes: None,
470470- r#enum: None,
471471- r#const: None,
472472- known_values: None,
473473- }),
474474- );
475475- map.insert(
476476- ::jacquard_common::smol_str::SmolStr::new_static("type"),
477477- ::jacquard_lexicon::lexicon::LexObjectProperty::String(::jacquard_lexicon::lexicon::LexString {
478478- description: None,
479479- format: None,
480480- default: None,
481481- min_length: None,
482482- max_length: None,
483483- min_graphemes: None,
484484- max_graphemes: None,
485485- r#enum: None,
486486- r#const: None,
487487- known_values: None,
488488- }),
489489- );
490490- map
491491- },
492492- }),
493493- );
494494- map
495495- },
496496- }
497497-}
498498-499499-impl<'a> ::jacquard_lexicon::schema::LexiconSchema for Directory<'a> {
500500- fn nsid() -> &'static str {
501501- "place.wisp.fs"
502502- }
503503- fn def_name() -> &'static str {
504504- "directory"
505505- }
506506- fn lexicon_doc() -> ::jacquard_lexicon::lexicon::LexiconDoc<'static> {
507507- lexicon_doc_place_wisp_fs()
508508- }
509509- fn validate(
510510- &self,
511511- ) -> ::std::result::Result<(), ::jacquard_lexicon::validation::ConstraintError> {
512512- {
513513- let value = &self.entries;
514514- #[allow(unused_comparisons)]
515515- if value.len() > 500usize {
516516- return Err(::jacquard_lexicon::validation::ConstraintError::MaxLength {
517517- path: ::jacquard_lexicon::validation::ValidationPath::from_field(
518518- "entries",
519519- ),
520520- max: 500usize,
521521- actual: value.len(),
522522- });
523523- }
524524- }
525525- Ok(())
526526- }
527527-}
528528-529529-#[jacquard_derive::lexicon]
530530-#[derive(
531531- serde::Serialize,
532532- serde::Deserialize,
533533- Debug,
534534- Clone,
535535- PartialEq,
536536- Eq,
537537- jacquard_derive::IntoStatic
538538-)]
539539-#[serde(rename_all = "camelCase")]
540540-pub struct Entry<'a> {
541541- #[serde(borrow)]
542542- pub name: jacquard_common::CowStr<'a>,
543543- #[serde(borrow)]
544544- pub node: EntryNode<'a>,
545545-}
546546-547547-pub mod entry_state {
548548-549549- pub use crate::builder_types::{Set, Unset, IsSet, IsUnset};
550550- #[allow(unused)]
551551- use ::core::marker::PhantomData;
552552- mod sealed {
553553- pub trait Sealed {}
554554- }
555555- /// State trait tracking which required fields have been set
556556- pub trait State: sealed::Sealed {
557557- type Node;
558558- type Name;
559559- }
560560- /// Empty state - all required fields are unset
561561- pub struct Empty(());
562562- impl sealed::Sealed for Empty {}
563563- impl State for Empty {
564564- type Node = Unset;
565565- type Name = Unset;
566566- }
567567- ///State transition - sets the `node` field to Set
568568- pub struct SetNode<S: State = Empty>(PhantomData<fn() -> S>);
569569- impl<S: State> sealed::Sealed for SetNode<S> {}
570570- impl<S: State> State for SetNode<S> {
571571- type Node = Set<members::node>;
572572- type Name = S::Name;
573573- }
574574- ///State transition - sets the `name` field to Set
575575- pub struct SetName<S: State = Empty>(PhantomData<fn() -> S>);
576576- impl<S: State> sealed::Sealed for SetName<S> {}
577577- impl<S: State> State for SetName<S> {
578578- type Node = S::Node;
579579- type Name = Set<members::name>;
580580- }
581581- /// Marker types for field names
582582- #[allow(non_camel_case_types)]
583583- pub mod members {
584584- ///Marker type for the `node` field
585585- pub struct node(());
586586- ///Marker type for the `name` field
587587- pub struct name(());
588588- }
589589-}
590590-591591-/// Builder for constructing an instance of this type
592592-pub struct EntryBuilder<'a, S: entry_state::State> {
593593- _phantom_state: ::core::marker::PhantomData<fn() -> S>,
594594- __unsafe_private_named: (
595595- ::core::option::Option<jacquard_common::CowStr<'a>>,
596596- ::core::option::Option<EntryNode<'a>>,
597597- ),
598598- _phantom: ::core::marker::PhantomData<&'a ()>,
599599-}
600600-601601-impl<'a> Entry<'a> {
602602- /// Create a new builder for this type
603603- pub fn new() -> EntryBuilder<'a, entry_state::Empty> {
604604- EntryBuilder::new()
605605- }
606606-}
607607-608608-impl<'a> EntryBuilder<'a, entry_state::Empty> {
609609- /// Create a new builder with all fields unset
610610- pub fn new() -> Self {
611611- EntryBuilder {
612612- _phantom_state: ::core::marker::PhantomData,
613613- __unsafe_private_named: (None, None),
614614- _phantom: ::core::marker::PhantomData,
615615- }
616616- }
617617-}
618618-619619-impl<'a, S> EntryBuilder<'a, S>
620620-where
621621- S: entry_state::State,
622622- S::Name: entry_state::IsUnset,
623623-{
624624- /// Set the `name` field (required)
625625- pub fn name(
626626- mut self,
627627- value: impl Into<jacquard_common::CowStr<'a>>,
628628- ) -> EntryBuilder<'a, entry_state::SetName<S>> {
629629- self.__unsafe_private_named.0 = ::core::option::Option::Some(value.into());
630630- EntryBuilder {
631631- _phantom_state: ::core::marker::PhantomData,
632632- __unsafe_private_named: self.__unsafe_private_named,
633633- _phantom: ::core::marker::PhantomData,
634634- }
635635- }
636636-}
637637-638638-impl<'a, S> EntryBuilder<'a, S>
639639-where
640640- S: entry_state::State,
641641- S::Node: entry_state::IsUnset,
642642-{
643643- /// Set the `node` field (required)
644644- pub fn node(
645645- mut self,
646646- value: impl Into<EntryNode<'a>>,
647647- ) -> EntryBuilder<'a, entry_state::SetNode<S>> {
648648- self.__unsafe_private_named.1 = ::core::option::Option::Some(value.into());
649649- EntryBuilder {
650650- _phantom_state: ::core::marker::PhantomData,
651651- __unsafe_private_named: self.__unsafe_private_named,
652652- _phantom: ::core::marker::PhantomData,
653653- }
654654- }
655655-}
656656-657657-impl<'a, S> EntryBuilder<'a, S>
658658-where
659659- S: entry_state::State,
660660- S::Node: entry_state::IsSet,
661661- S::Name: entry_state::IsSet,
662662-{
663663- /// Build the final struct
664664- pub fn build(self) -> Entry<'a> {
665665- Entry {
666666- name: self.__unsafe_private_named.0.unwrap(),
667667- node: self.__unsafe_private_named.1.unwrap(),
668668- extra_data: Default::default(),
669669- }
670670- }
671671- /// Build the final struct with custom extra_data
672672- pub fn build_with_data(
673673- self,
674674- extra_data: std::collections::BTreeMap<
675675- jacquard_common::smol_str::SmolStr,
676676- jacquard_common::types::value::Data<'a>,
677677- >,
678678- ) -> Entry<'a> {
679679- Entry {
680680- name: self.__unsafe_private_named.0.unwrap(),
681681- node: self.__unsafe_private_named.1.unwrap(),
682682- extra_data: Some(extra_data),
683683- }
684684- }
685685-}
686686-687687-#[jacquard_derive::open_union]
688688-#[derive(
689689- serde::Serialize,
690690- serde::Deserialize,
691691- Debug,
692692- Clone,
693693- PartialEq,
694694- Eq,
695695- jacquard_derive::IntoStatic
696696-)]
697697-#[serde(tag = "$type")]
698698-#[serde(bound(deserialize = "'de: 'a"))]
699699-pub enum EntryNode<'a> {
700700- #[serde(rename = "place.wisp.fs#file")]
701701- File(Box<crate::place_wisp::fs::File<'a>>),
702702- #[serde(rename = "place.wisp.fs#directory")]
703703- Directory(Box<crate::place_wisp::fs::Directory<'a>>),
704704- #[serde(rename = "place.wisp.fs#subfs")]
705705- Subfs(Box<crate::place_wisp::fs::Subfs<'a>>),
706706-}
707707-708708-impl<'a> ::jacquard_lexicon::schema::LexiconSchema for Entry<'a> {
709709- fn nsid() -> &'static str {
710710- "place.wisp.fs"
711711- }
712712- fn def_name() -> &'static str {
713713- "entry"
714714- }
715715- fn lexicon_doc() -> ::jacquard_lexicon::lexicon::LexiconDoc<'static> {
716716- lexicon_doc_place_wisp_fs()
717717- }
718718- fn validate(
719719- &self,
720720- ) -> ::std::result::Result<(), ::jacquard_lexicon::validation::ConstraintError> {
721721- {
722722- let value = &self.name;
723723- #[allow(unused_comparisons)]
724724- if <str>::len(value.as_ref()) > 255usize {
725725- return Err(::jacquard_lexicon::validation::ConstraintError::MaxLength {
726726- path: ::jacquard_lexicon::validation::ValidationPath::from_field(
727727- "name",
728728- ),
729729- max: 255usize,
730730- actual: <str>::len(value.as_ref()),
731731- });
732732- }
733733- }
734734- Ok(())
735735- }
736736-}
737737-738738-#[jacquard_derive::lexicon]
739739-#[derive(
740740- serde::Serialize,
741741- serde::Deserialize,
742742- Debug,
743743- Clone,
744744- PartialEq,
745745- Eq,
746746- jacquard_derive::IntoStatic
747747-)]
748748-#[serde(rename_all = "camelCase")]
749749-pub struct File<'a> {
750750- /// True if blob content is base64-encoded (used to bypass PDS content sniffing)
751751- #[serde(skip_serializing_if = "std::option::Option::is_none")]
752752- pub base64: std::option::Option<bool>,
753753- /// Content blob ref
754754- #[serde(borrow)]
755755- pub blob: jacquard_common::types::blob::BlobRef<'a>,
756756- /// Content encoding (e.g., gzip for compressed files)
757757- #[serde(skip_serializing_if = "std::option::Option::is_none")]
758758- #[serde(borrow)]
759759- pub encoding: std::option::Option<jacquard_common::CowStr<'a>>,
760760- /// Original MIME type before compression
761761- #[serde(skip_serializing_if = "std::option::Option::is_none")]
762762- #[serde(borrow)]
763763- pub mime_type: std::option::Option<jacquard_common::CowStr<'a>>,
764764- #[serde(borrow)]
765765- pub r#type: jacquard_common::CowStr<'a>,
766766-}
767767-768768-pub mod file_state {
769769-770770- pub use crate::builder_types::{Set, Unset, IsSet, IsUnset};
771771- #[allow(unused)]
772772- use ::core::marker::PhantomData;
773773- mod sealed {
774774- pub trait Sealed {}
775775- }
776776- /// State trait tracking which required fields have been set
777777- pub trait State: sealed::Sealed {
778778- type Type;
779779- type Blob;
780780- }
781781- /// Empty state - all required fields are unset
782782- pub struct Empty(());
783783- impl sealed::Sealed for Empty {}
784784- impl State for Empty {
785785- type Type = Unset;
786786- type Blob = Unset;
787787- }
788788- ///State transition - sets the `type` field to Set
789789- pub struct SetType<S: State = Empty>(PhantomData<fn() -> S>);
790790- impl<S: State> sealed::Sealed for SetType<S> {}
791791- impl<S: State> State for SetType<S> {
792792- type Type = Set<members::r#type>;
793793- type Blob = S::Blob;
794794- }
795795- ///State transition - sets the `blob` field to Set
796796- pub struct SetBlob<S: State = Empty>(PhantomData<fn() -> S>);
797797- impl<S: State> sealed::Sealed for SetBlob<S> {}
798798- impl<S: State> State for SetBlob<S> {
799799- type Type = S::Type;
800800- type Blob = Set<members::blob>;
801801- }
802802- /// Marker types for field names
803803- #[allow(non_camel_case_types)]
804804- pub mod members {
805805- ///Marker type for the `type` field
806806- pub struct r#type(());
807807- ///Marker type for the `blob` field
808808- pub struct blob(());
809809- }
810810-}
811811-812812-/// Builder for constructing an instance of this type
813813-pub struct FileBuilder<'a, S: file_state::State> {
814814- _phantom_state: ::core::marker::PhantomData<fn() -> S>,
815815- __unsafe_private_named: (
816816- ::core::option::Option<bool>,
817817- ::core::option::Option<jacquard_common::types::blob::BlobRef<'a>>,
818818- ::core::option::Option<jacquard_common::CowStr<'a>>,
819819- ::core::option::Option<jacquard_common::CowStr<'a>>,
820820- ::core::option::Option<jacquard_common::CowStr<'a>>,
821821- ),
822822- _phantom: ::core::marker::PhantomData<&'a ()>,
823823-}
824824-825825-impl<'a> File<'a> {
826826- /// Create a new builder for this type
827827- pub fn new() -> FileBuilder<'a, file_state::Empty> {
828828- FileBuilder::new()
829829- }
830830-}
831831-832832-impl<'a> FileBuilder<'a, file_state::Empty> {
833833- /// Create a new builder with all fields unset
834834- pub fn new() -> Self {
835835- FileBuilder {
836836- _phantom_state: ::core::marker::PhantomData,
837837- __unsafe_private_named: (None, None, None, None, None),
838838- _phantom: ::core::marker::PhantomData,
839839- }
840840- }
841841-}
842842-843843-impl<'a, S: file_state::State> FileBuilder<'a, S> {
844844- /// Set the `base64` field (optional)
845845- pub fn base64(mut self, value: impl Into<Option<bool>>) -> Self {
846846- self.__unsafe_private_named.0 = value.into();
847847- self
848848- }
849849- /// Set the `base64` field to an Option value (optional)
850850- pub fn maybe_base64(mut self, value: Option<bool>) -> Self {
851851- self.__unsafe_private_named.0 = value;
852852- self
853853- }
854854-}
855855-856856-impl<'a, S> FileBuilder<'a, S>
857857-where
858858- S: file_state::State,
859859- S::Blob: file_state::IsUnset,
860860-{
861861- /// Set the `blob` field (required)
862862- pub fn blob(
863863- mut self,
864864- value: impl Into<jacquard_common::types::blob::BlobRef<'a>>,
865865- ) -> FileBuilder<'a, file_state::SetBlob<S>> {
866866- self.__unsafe_private_named.1 = ::core::option::Option::Some(value.into());
867867- FileBuilder {
868868- _phantom_state: ::core::marker::PhantomData,
869869- __unsafe_private_named: self.__unsafe_private_named,
870870- _phantom: ::core::marker::PhantomData,
871871- }
872872- }
873873-}
874874-875875-impl<'a, S: file_state::State> FileBuilder<'a, S> {
876876- /// Set the `encoding` field (optional)
877877- pub fn encoding(
878878- mut self,
879879- value: impl Into<Option<jacquard_common::CowStr<'a>>>,
880880- ) -> Self {
881881- self.__unsafe_private_named.2 = value.into();
882882- self
883883- }
884884- /// Set the `encoding` field to an Option value (optional)
885885- pub fn maybe_encoding(mut self, value: Option<jacquard_common::CowStr<'a>>) -> Self {
886886- self.__unsafe_private_named.2 = value;
887887- self
888888- }
889889-}
890890-891891-impl<'a, S: file_state::State> FileBuilder<'a, S> {
892892- /// Set the `mimeType` field (optional)
893893- pub fn mime_type(
894894- mut self,
895895- value: impl Into<Option<jacquard_common::CowStr<'a>>>,
896896- ) -> Self {
897897- self.__unsafe_private_named.3 = value.into();
898898- self
899899- }
900900- /// Set the `mimeType` field to an Option value (optional)
901901- pub fn maybe_mime_type(
902902- mut self,
903903- value: Option<jacquard_common::CowStr<'a>>,
904904- ) -> Self {
905905- self.__unsafe_private_named.3 = value;
906906- self
907907- }
908908-}
909909-910910-impl<'a, S> FileBuilder<'a, S>
911911-where
912912- S: file_state::State,
913913- S::Type: file_state::IsUnset,
914914-{
915915- /// Set the `type` field (required)
916916- pub fn r#type(
917917- mut self,
918918- value: impl Into<jacquard_common::CowStr<'a>>,
919919- ) -> FileBuilder<'a, file_state::SetType<S>> {
920920- self.__unsafe_private_named.4 = ::core::option::Option::Some(value.into());
921921- FileBuilder {
922922- _phantom_state: ::core::marker::PhantomData,
923923- __unsafe_private_named: self.__unsafe_private_named,
924924- _phantom: ::core::marker::PhantomData,
925925- }
926926- }
927927-}
928928-929929-impl<'a, S> FileBuilder<'a, S>
930930-where
931931- S: file_state::State,
932932- S::Type: file_state::IsSet,
933933- S::Blob: file_state::IsSet,
934934-{
935935- /// Build the final struct
936936- pub fn build(self) -> File<'a> {
937937- File {
938938- base64: self.__unsafe_private_named.0,
939939- blob: self.__unsafe_private_named.1.unwrap(),
940940- encoding: self.__unsafe_private_named.2,
941941- mime_type: self.__unsafe_private_named.3,
942942- r#type: self.__unsafe_private_named.4.unwrap(),
943943- extra_data: Default::default(),
944944- }
945945- }
946946- /// Build the final struct with custom extra_data
947947- pub fn build_with_data(
948948- self,
949949- extra_data: std::collections::BTreeMap<
950950- jacquard_common::smol_str::SmolStr,
951951- jacquard_common::types::value::Data<'a>,
952952- >,
953953- ) -> File<'a> {
954954- File {
955955- base64: self.__unsafe_private_named.0,
956956- blob: self.__unsafe_private_named.1.unwrap(),
957957- encoding: self.__unsafe_private_named.2,
958958- mime_type: self.__unsafe_private_named.3,
959959- r#type: self.__unsafe_private_named.4.unwrap(),
960960- extra_data: Some(extra_data),
961961- }
962962- }
963963-}
964964-965965-impl<'a> ::jacquard_lexicon::schema::LexiconSchema for File<'a> {
966966- fn nsid() -> &'static str {
967967- "place.wisp.fs"
968968- }
969969- fn def_name() -> &'static str {
970970- "file"
971971- }
972972- fn lexicon_doc() -> ::jacquard_lexicon::lexicon::LexiconDoc<'static> {
973973- lexicon_doc_place_wisp_fs()
974974- }
975975- fn validate(
976976- &self,
977977- ) -> ::std::result::Result<(), ::jacquard_lexicon::validation::ConstraintError> {
978978- Ok(())
979979- }
980980-}
981981-982982-/// Virtual filesystem manifest for a Wisp site
983983-#[jacquard_derive::lexicon]
984984-#[derive(
985985- serde::Serialize,
986986- serde::Deserialize,
987987- Debug,
988988- Clone,
989989- PartialEq,
990990- Eq,
991991- jacquard_derive::IntoStatic
992992-)]
993993-#[serde(rename_all = "camelCase")]
994994-pub struct Fs<'a> {
995995- pub created_at: jacquard_common::types::string::Datetime,
996996- #[serde(skip_serializing_if = "std::option::Option::is_none")]
997997- pub file_count: std::option::Option<i64>,
998998- #[serde(borrow)]
999999- pub root: crate::place_wisp::fs::Directory<'a>,
10001000- #[serde(borrow)]
10011001- pub site: jacquard_common::CowStr<'a>,
10021002-}
10031003-10041004-pub mod fs_state {
10051005-10061006- pub use crate::builder_types::{Set, Unset, IsSet, IsUnset};
10071007- #[allow(unused)]
10081008- use ::core::marker::PhantomData;
10091009- mod sealed {
10101010- pub trait Sealed {}
10111011- }
10121012- /// State trait tracking which required fields have been set
10131013- pub trait State: sealed::Sealed {
10141014- type CreatedAt;
10151015- type Site;
10161016- type Root;
10171017- }
10181018- /// Empty state - all required fields are unset
10191019- pub struct Empty(());
10201020- impl sealed::Sealed for Empty {}
10211021- impl State for Empty {
10221022- type CreatedAt = Unset;
10231023- type Site = Unset;
10241024- type Root = Unset;
10251025- }
10261026- ///State transition - sets the `created_at` field to Set
10271027- pub struct SetCreatedAt<S: State = Empty>(PhantomData<fn() -> S>);
10281028- impl<S: State> sealed::Sealed for SetCreatedAt<S> {}
10291029- impl<S: State> State for SetCreatedAt<S> {
10301030- type CreatedAt = Set<members::created_at>;
10311031- type Site = S::Site;
10321032- type Root = S::Root;
10331033- }
10341034- ///State transition - sets the `site` field to Set
10351035- pub struct SetSite<S: State = Empty>(PhantomData<fn() -> S>);
10361036- impl<S: State> sealed::Sealed for SetSite<S> {}
10371037- impl<S: State> State for SetSite<S> {
10381038- type CreatedAt = S::CreatedAt;
10391039- type Site = Set<members::site>;
10401040- type Root = S::Root;
10411041- }
10421042- ///State transition - sets the `root` field to Set
10431043- pub struct SetRoot<S: State = Empty>(PhantomData<fn() -> S>);
10441044- impl<S: State> sealed::Sealed for SetRoot<S> {}
10451045- impl<S: State> State for SetRoot<S> {
10461046- type CreatedAt = S::CreatedAt;
10471047- type Site = S::Site;
10481048- type Root = Set<members::root>;
10491049- }
10501050- /// Marker types for field names
10511051- #[allow(non_camel_case_types)]
10521052- pub mod members {
10531053- ///Marker type for the `created_at` field
10541054- pub struct created_at(());
10551055- ///Marker type for the `site` field
10561056- pub struct site(());
10571057- ///Marker type for the `root` field
10581058- pub struct root(());
10591059- }
10601060-}
10611061-10621062-/// Builder for constructing an instance of this type
10631063-pub struct FsBuilder<'a, S: fs_state::State> {
10641064- _phantom_state: ::core::marker::PhantomData<fn() -> S>,
10651065- __unsafe_private_named: (
10661066- ::core::option::Option<jacquard_common::types::string::Datetime>,
10671067- ::core::option::Option<i64>,
10681068- ::core::option::Option<crate::place_wisp::fs::Directory<'a>>,
10691069- ::core::option::Option<jacquard_common::CowStr<'a>>,
10701070- ),
10711071- _phantom: ::core::marker::PhantomData<&'a ()>,
10721072-}
10731073-10741074-impl<'a> Fs<'a> {
10751075- /// Create a new builder for this type
10761076- pub fn new() -> FsBuilder<'a, fs_state::Empty> {
10771077- FsBuilder::new()
10781078- }
10791079-}
10801080-10811081-impl<'a> FsBuilder<'a, fs_state::Empty> {
10821082- /// Create a new builder with all fields unset
10831083- pub fn new() -> Self {
10841084- FsBuilder {
10851085- _phantom_state: ::core::marker::PhantomData,
10861086- __unsafe_private_named: (None, None, None, None),
10871087- _phantom: ::core::marker::PhantomData,
10881088- }
10891089- }
10901090-}
10911091-10921092-impl<'a, S> FsBuilder<'a, S>
10931093-where
10941094- S: fs_state::State,
10951095- S::CreatedAt: fs_state::IsUnset,
10961096-{
10971097- /// Set the `createdAt` field (required)
10981098- pub fn created_at(
10991099- mut self,
11001100- value: impl Into<jacquard_common::types::string::Datetime>,
11011101- ) -> FsBuilder<'a, fs_state::SetCreatedAt<S>> {
11021102- self.__unsafe_private_named.0 = ::core::option::Option::Some(value.into());
11031103- FsBuilder {
11041104- _phantom_state: ::core::marker::PhantomData,
11051105- __unsafe_private_named: self.__unsafe_private_named,
11061106- _phantom: ::core::marker::PhantomData,
11071107- }
11081108- }
11091109-}
11101110-11111111-impl<'a, S: fs_state::State> FsBuilder<'a, S> {
11121112- /// Set the `fileCount` field (optional)
11131113- pub fn file_count(mut self, value: impl Into<Option<i64>>) -> Self {
11141114- self.__unsafe_private_named.1 = value.into();
11151115- self
11161116- }
11171117- /// Set the `fileCount` field to an Option value (optional)
11181118- pub fn maybe_file_count(mut self, value: Option<i64>) -> Self {
11191119- self.__unsafe_private_named.1 = value;
11201120- self
11211121- }
11221122-}
11231123-11241124-impl<'a, S> FsBuilder<'a, S>
11251125-where
11261126- S: fs_state::State,
11271127- S::Root: fs_state::IsUnset,
11281128-{
11291129- /// Set the `root` field (required)
11301130- pub fn root(
11311131- mut self,
11321132- value: impl Into<crate::place_wisp::fs::Directory<'a>>,
11331133- ) -> FsBuilder<'a, fs_state::SetRoot<S>> {
11341134- self.__unsafe_private_named.2 = ::core::option::Option::Some(value.into());
11351135- FsBuilder {
11361136- _phantom_state: ::core::marker::PhantomData,
11371137- __unsafe_private_named: self.__unsafe_private_named,
11381138- _phantom: ::core::marker::PhantomData,
11391139- }
11401140- }
11411141-}
11421142-11431143-impl<'a, S> FsBuilder<'a, S>
11441144-where
11451145- S: fs_state::State,
11461146- S::Site: fs_state::IsUnset,
11471147-{
11481148- /// Set the `site` field (required)
11491149- pub fn site(
11501150- mut self,
11511151- value: impl Into<jacquard_common::CowStr<'a>>,
11521152- ) -> FsBuilder<'a, fs_state::SetSite<S>> {
11531153- self.__unsafe_private_named.3 = ::core::option::Option::Some(value.into());
11541154- FsBuilder {
11551155- _phantom_state: ::core::marker::PhantomData,
11561156- __unsafe_private_named: self.__unsafe_private_named,
11571157- _phantom: ::core::marker::PhantomData,
11581158- }
11591159- }
11601160-}
11611161-11621162-impl<'a, S> FsBuilder<'a, S>
11631163-where
11641164- S: fs_state::State,
11651165- S::CreatedAt: fs_state::IsSet,
11661166- S::Site: fs_state::IsSet,
11671167- S::Root: fs_state::IsSet,
11681168-{
11691169- /// Build the final struct
11701170- pub fn build(self) -> Fs<'a> {
11711171- Fs {
11721172- created_at: self.__unsafe_private_named.0.unwrap(),
11731173- file_count: self.__unsafe_private_named.1,
11741174- root: self.__unsafe_private_named.2.unwrap(),
11751175- site: self.__unsafe_private_named.3.unwrap(),
11761176- extra_data: Default::default(),
11771177- }
11781178- }
11791179- /// Build the final struct with custom extra_data
11801180- pub fn build_with_data(
11811181- self,
11821182- extra_data: std::collections::BTreeMap<
11831183- jacquard_common::smol_str::SmolStr,
11841184- jacquard_common::types::value::Data<'a>,
11851185- >,
11861186- ) -> Fs<'a> {
11871187- Fs {
11881188- created_at: self.__unsafe_private_named.0.unwrap(),
11891189- file_count: self.__unsafe_private_named.1,
11901190- root: self.__unsafe_private_named.2.unwrap(),
11911191- site: self.__unsafe_private_named.3.unwrap(),
11921192- extra_data: Some(extra_data),
11931193- }
11941194- }
11951195-}
11961196-11971197-impl<'a> Fs<'a> {
11981198- pub fn uri(
11991199- uri: impl Into<jacquard_common::CowStr<'a>>,
12001200- ) -> Result<
12011201- jacquard_common::types::uri::RecordUri<'a, FsRecord>,
12021202- jacquard_common::types::uri::UriError,
12031203- > {
12041204- jacquard_common::types::uri::RecordUri::try_from_uri(
12051205- jacquard_common::types::string::AtUri::new_cow(uri.into())?,
12061206- )
12071207- }
12081208-}
12091209-12101210-/// Typed wrapper for GetRecord response with this collection's record type.
12111211-#[derive(
12121212- serde::Serialize,
12131213- serde::Deserialize,
12141214- Debug,
12151215- Clone,
12161216- PartialEq,
12171217- Eq,
12181218- jacquard_derive::IntoStatic
12191219-)]
12201220-#[serde(rename_all = "camelCase")]
12211221-pub struct FsGetRecordOutput<'a> {
12221222- #[serde(skip_serializing_if = "std::option::Option::is_none")]
12231223- #[serde(borrow)]
12241224- pub cid: std::option::Option<jacquard_common::types::string::Cid<'a>>,
12251225- #[serde(borrow)]
12261226- pub uri: jacquard_common::types::string::AtUri<'a>,
12271227- #[serde(borrow)]
12281228- pub value: Fs<'a>,
12291229-}
12301230-12311231-impl From<FsGetRecordOutput<'_>> for Fs<'_> {
12321232- fn from(output: FsGetRecordOutput<'_>) -> Self {
12331233- use jacquard_common::IntoStatic;
12341234- output.value.into_static()
12351235- }
12361236-}
12371237-12381238-impl jacquard_common::types::collection::Collection for Fs<'_> {
12391239- const NSID: &'static str = "place.wisp.fs";
12401240- type Record = FsRecord;
12411241-}
12421242-12431243-/// Marker type for deserializing records from this collection.
12441244-#[derive(Debug, serde::Serialize, serde::Deserialize)]
12451245-pub struct FsRecord;
12461246-impl jacquard_common::xrpc::XrpcResp for FsRecord {
12471247- const NSID: &'static str = "place.wisp.fs";
12481248- const ENCODING: &'static str = "application/json";
12491249- type Output<'de> = FsGetRecordOutput<'de>;
12501250- type Err<'de> = jacquard_common::types::collection::RecordError<'de>;
12511251-}
12521252-12531253-impl jacquard_common::types::collection::Collection for FsRecord {
12541254- const NSID: &'static str = "place.wisp.fs";
12551255- type Record = FsRecord;
12561256-}
12571257-12581258-impl<'a> ::jacquard_lexicon::schema::LexiconSchema for Fs<'a> {
12591259- fn nsid() -> &'static str {
12601260- "place.wisp.fs"
12611261- }
12621262- fn def_name() -> &'static str {
12631263- "main"
12641264- }
12651265- fn lexicon_doc() -> ::jacquard_lexicon::lexicon::LexiconDoc<'static> {
12661266- lexicon_doc_place_wisp_fs()
12671267- }
12681268- fn validate(
12691269- &self,
12701270- ) -> ::std::result::Result<(), ::jacquard_lexicon::validation::ConstraintError> {
12711271- if let Some(ref value) = self.file_count {
12721272- if *value > 1000i64 {
12731273- return Err(::jacquard_lexicon::validation::ConstraintError::Maximum {
12741274- path: ::jacquard_lexicon::validation::ValidationPath::from_field(
12751275- "file_count",
12761276- ),
12771277- max: 1000i64,
12781278- actual: *value,
12791279- });
12801280- }
12811281- }
12821282- if let Some(ref value) = self.file_count {
12831283- if *value < 0i64 {
12841284- return Err(::jacquard_lexicon::validation::ConstraintError::Minimum {
12851285- path: ::jacquard_lexicon::validation::ValidationPath::from_field(
12861286- "file_count",
12871287- ),
12881288- min: 0i64,
12891289- actual: *value,
12901290- });
12911291- }
12921292- }
12931293- Ok(())
12941294- }
12951295-}
12961296-12971297-#[jacquard_derive::lexicon]
12981298-#[derive(
12991299- serde::Serialize,
13001300- serde::Deserialize,
13011301- Debug,
13021302- Clone,
13031303- PartialEq,
13041304- Eq,
13051305- jacquard_derive::IntoStatic
13061306-)]
13071307-#[serde(rename_all = "camelCase")]
13081308-pub struct Subfs<'a> {
13091309- /// If true (default), the subfs record's root entries are merged (flattened) into the parent directory, replacing the subfs entry. If false, the subfs entries are placed in a subdirectory with the subfs entry's name. Flat merging is useful for splitting large directories across multiple records while maintaining a flat structure.
13101310- #[serde(skip_serializing_if = "std::option::Option::is_none")]
13111311- pub flat: std::option::Option<bool>,
13121312- /// AT-URI pointing to a place.wisp.subfs record containing this subtree.
13131313- #[serde(borrow)]
13141314- pub subject: jacquard_common::types::string::AtUri<'a>,
13151315- #[serde(borrow)]
13161316- pub r#type: jacquard_common::CowStr<'a>,
13171317-}
13181318-13191319-pub mod subfs_state {
13201320-13211321- pub use crate::builder_types::{Set, Unset, IsSet, IsUnset};
13221322- #[allow(unused)]
13231323- use ::core::marker::PhantomData;
13241324- mod sealed {
13251325- pub trait Sealed {}
13261326- }
13271327- /// State trait tracking which required fields have been set
13281328- pub trait State: sealed::Sealed {
13291329- type Type;
13301330- type Subject;
13311331- }
13321332- /// Empty state - all required fields are unset
13331333- pub struct Empty(());
13341334- impl sealed::Sealed for Empty {}
13351335- impl State for Empty {
13361336- type Type = Unset;
13371337- type Subject = Unset;
13381338- }
13391339- ///State transition - sets the `type` field to Set
13401340- pub struct SetType<S: State = Empty>(PhantomData<fn() -> S>);
13411341- impl<S: State> sealed::Sealed for SetType<S> {}
13421342- impl<S: State> State for SetType<S> {
13431343- type Type = Set<members::r#type>;
13441344- type Subject = S::Subject;
13451345- }
13461346- ///State transition - sets the `subject` field to Set
13471347- pub struct SetSubject<S: State = Empty>(PhantomData<fn() -> S>);
13481348- impl<S: State> sealed::Sealed for SetSubject<S> {}
13491349- impl<S: State> State for SetSubject<S> {
13501350- type Type = S::Type;
13511351- type Subject = Set<members::subject>;
13521352- }
13531353- /// Marker types for field names
13541354- #[allow(non_camel_case_types)]
13551355- pub mod members {
13561356- ///Marker type for the `type` field
13571357- pub struct r#type(());
13581358- ///Marker type for the `subject` field
13591359- pub struct subject(());
13601360- }
13611361-}
13621362-13631363-/// Builder for constructing an instance of this type
13641364-pub struct SubfsBuilder<'a, S: subfs_state::State> {
13651365- _phantom_state: ::core::marker::PhantomData<fn() -> S>,
13661366- __unsafe_private_named: (
13671367- ::core::option::Option<bool>,
13681368- ::core::option::Option<jacquard_common::types::string::AtUri<'a>>,
13691369- ::core::option::Option<jacquard_common::CowStr<'a>>,
13701370- ),
13711371- _phantom: ::core::marker::PhantomData<&'a ()>,
13721372-}
13731373-13741374-impl<'a> Subfs<'a> {
13751375- /// Create a new builder for this type
13761376- pub fn new() -> SubfsBuilder<'a, subfs_state::Empty> {
13771377- SubfsBuilder::new()
13781378- }
13791379-}
13801380-13811381-impl<'a> SubfsBuilder<'a, subfs_state::Empty> {
13821382- /// Create a new builder with all fields unset
13831383- pub fn new() -> Self {
13841384- SubfsBuilder {
13851385- _phantom_state: ::core::marker::PhantomData,
13861386- __unsafe_private_named: (None, None, None),
13871387- _phantom: ::core::marker::PhantomData,
13881388- }
13891389- }
13901390-}
13911391-13921392-impl<'a, S: subfs_state::State> SubfsBuilder<'a, S> {
13931393- /// Set the `flat` field (optional)
13941394- pub fn flat(mut self, value: impl Into<Option<bool>>) -> Self {
13951395- self.__unsafe_private_named.0 = value.into();
13961396- self
13971397- }
13981398- /// Set the `flat` field to an Option value (optional)
13991399- pub fn maybe_flat(mut self, value: Option<bool>) -> Self {
14001400- self.__unsafe_private_named.0 = value;
14011401- self
14021402- }
14031403-}
14041404-14051405-impl<'a, S> SubfsBuilder<'a, S>
14061406-where
14071407- S: subfs_state::State,
14081408- S::Subject: subfs_state::IsUnset,
14091409-{
14101410- /// Set the `subject` field (required)
14111411- pub fn subject(
14121412- mut self,
14131413- value: impl Into<jacquard_common::types::string::AtUri<'a>>,
14141414- ) -> SubfsBuilder<'a, subfs_state::SetSubject<S>> {
14151415- self.__unsafe_private_named.1 = ::core::option::Option::Some(value.into());
14161416- SubfsBuilder {
14171417- _phantom_state: ::core::marker::PhantomData,
14181418- __unsafe_private_named: self.__unsafe_private_named,
14191419- _phantom: ::core::marker::PhantomData,
14201420- }
14211421- }
14221422-}
14231423-14241424-impl<'a, S> SubfsBuilder<'a, S>
14251425-where
14261426- S: subfs_state::State,
14271427- S::Type: subfs_state::IsUnset,
14281428-{
14291429- /// Set the `type` field (required)
14301430- pub fn r#type(
14311431- mut self,
14321432- value: impl Into<jacquard_common::CowStr<'a>>,
14331433- ) -> SubfsBuilder<'a, subfs_state::SetType<S>> {
14341434- self.__unsafe_private_named.2 = ::core::option::Option::Some(value.into());
14351435- SubfsBuilder {
14361436- _phantom_state: ::core::marker::PhantomData,
14371437- __unsafe_private_named: self.__unsafe_private_named,
14381438- _phantom: ::core::marker::PhantomData,
14391439- }
14401440- }
14411441-}
14421442-14431443-impl<'a, S> SubfsBuilder<'a, S>
14441444-where
14451445- S: subfs_state::State,
14461446- S::Type: subfs_state::IsSet,
14471447- S::Subject: subfs_state::IsSet,
14481448-{
14491449- /// Build the final struct
14501450- pub fn build(self) -> Subfs<'a> {
14511451- Subfs {
14521452- flat: self.__unsafe_private_named.0,
14531453- subject: self.__unsafe_private_named.1.unwrap(),
14541454- r#type: self.__unsafe_private_named.2.unwrap(),
14551455- extra_data: Default::default(),
14561456- }
14571457- }
14581458- /// Build the final struct with custom extra_data
14591459- pub fn build_with_data(
14601460- self,
14611461- extra_data: std::collections::BTreeMap<
14621462- jacquard_common::smol_str::SmolStr,
14631463- jacquard_common::types::value::Data<'a>,
14641464- >,
14651465- ) -> Subfs<'a> {
14661466- Subfs {
14671467- flat: self.__unsafe_private_named.0,
14681468- subject: self.__unsafe_private_named.1.unwrap(),
14691469- r#type: self.__unsafe_private_named.2.unwrap(),
14701470- extra_data: Some(extra_data),
14711471- }
14721472- }
14731473-}
14741474-14751475-impl<'a> ::jacquard_lexicon::schema::LexiconSchema for Subfs<'a> {
14761476- fn nsid() -> &'static str {
14771477- "place.wisp.fs"
14781478- }
14791479- fn def_name() -> &'static str {
14801480- "subfs"
14811481- }
14821482- fn lexicon_doc() -> ::jacquard_lexicon::lexicon::LexiconDoc<'static> {
14831483- lexicon_doc_place_wisp_fs()
14841484- }
14851485- fn validate(
14861486- &self,
14871487- ) -> ::std::result::Result<(), ::jacquard_lexicon::validation::ConstraintError> {
14881488- Ok(())
14891489- }
14901490-}
···11-use jacquard_common::types::blob::BlobRef;
22-use jacquard_common::IntoStatic;
33-use std::collections::HashMap;
44-55-use wisp_lexicons::place_wisp::fs::{Directory, EntryNode};
66-77-/// Extract blob information from a directory tree
88-/// Returns a map of file paths to their blob refs and CIDs
99-///
1010-/// This mirrors the TypeScript implementation in src/lib/wisp-utils.ts lines 275-302
1111-pub fn extract_blob_map(
1212- directory: &Directory,
1313-) -> HashMap<String, (BlobRef<'static>, String)> {
1414- extract_blob_map_recursive(directory, String::new())
1515-}
1616-1717-fn extract_blob_map_recursive(
1818- directory: &Directory,
1919- current_path: String,
2020-) -> HashMap<String, (BlobRef<'static>, String)> {
2121- let mut blob_map = HashMap::new();
2222-2323- for entry in &directory.entries {
2424- let full_path = if current_path.is_empty() {
2525- entry.name.to_string()
2626- } else {
2727- format!("{}/{}", current_path, entry.name)
2828- };
2929-3030- match &entry.node {
3131- EntryNode::File(file_node) => {
3232- // Extract CID from blob ref
3333- // BlobRef is an enum with Blob variant, which has a ref field (CidLink)
3434- let blob_ref = &file_node.blob;
3535- let cid_string = blob_ref.blob().r#ref.to_string();
3636-3737- // Store with full path (mirrors TypeScript implementation)
3838- blob_map.insert(
3939- full_path,
4040- (blob_ref.clone().into_static(), cid_string)
4141- );
4242- }
4343- EntryNode::Directory(subdir) => {
4444- let sub_map = extract_blob_map_recursive(subdir, full_path);
4545- blob_map.extend(sub_map);
4646- }
4747- EntryNode::Subfs(_) => {
4848- // Subfs nodes don't contain blobs directly - they reference other records
4949- // Skip them in blob map extraction
5050- }
5151- EntryNode::Unknown(_) => {
5252- // Skip unknown node types
5353- }
5454- }
5555- }
5656-5757- blob_map
5858-}
5959-6060-/// Normalize file path by removing base folder prefix
6161-/// Example: "cobblemon/index.html" -> "index.html"
6262-///
6363-/// Note: This function is kept for reference but is no longer used in production code.
6464-/// The TypeScript server has a similar normalization (src/routes/wisp.ts line 291) to handle
6565-/// uploads that include a base folder prefix, but our CLI doesn't need this since we
6666-/// track full paths consistently.
6767-#[allow(dead_code)]
6868-pub fn normalize_path(path: &str) -> String {
6969- // Remove base folder prefix (everything before first /)
7070- if let Some(idx) = path.find('/') {
7171- path[idx + 1..].to_string()
7272- } else {
7373- path.to_string()
7474- }
7575-}
7676-7777-#[cfg(test)]
7878-mod tests {
7979- use super::*;
8080-8181- #[test]
8282- fn test_normalize_path() {
8383- assert_eq!(normalize_path("index.html"), "index.html");
8484- assert_eq!(normalize_path("cobblemon/index.html"), "index.html");
8585- assert_eq!(normalize_path("folder/subfolder/file.txt"), "subfolder/file.txt");
8686- assert_eq!(normalize_path("a/b/c/d.txt"), "b/c/d.txt");
8787- }
8888-}
8989-
-66
rust-cli/src/cid.rs
···11-use jacquard_common::types::cid::IpldCid;
22-use sha2::{Digest, Sha256};
33-44-/// Compute CID (Content Identifier) for blob content
55-/// Uses the same algorithm as AT Protocol: CIDv1 with raw codec (0x55) and SHA-256
66-///
77-/// CRITICAL: This must be called on BASE64-ENCODED GZIPPED content, not just gzipped content
88-///
99-/// Based on @atproto/common/src/ipld.ts sha256RawToCid implementation
1010-pub fn compute_cid(content: &[u8]) -> String {
1111- // Use node crypto to compute sha256 hash (same as AT Protocol)
1212- let hash = Sha256::digest(content);
1313-1414- // Create multihash (code 0x12 = sha2-256)
1515- let multihash = multihash::Multihash::wrap(0x12, &hash)
1616- .expect("SHA-256 hash should always fit in multihash");
1717-1818- // Create CIDv1 with raw codec (0x55)
1919- let cid = IpldCid::new_v1(0x55, multihash);
2020-2121- // Convert to base32 string representation
2222- cid.to_string_of_base(multibase::Base::Base32Lower)
2323- .unwrap_or_else(|_| cid.to_string())
2424-}
2525-2626-#[cfg(test)]
2727-mod tests {
2828- use super::*;
2929- use base64::Engine;
3030-3131- #[test]
3232- fn test_compute_cid() {
3333- // Test with a simple string: "hello"
3434- let content = b"hello";
3535- let cid = compute_cid(content);
3636-3737- // CID should start with 'baf' for raw codec base32
3838- assert!(cid.starts_with("baf"));
3939- }
4040-4141- #[test]
4242- fn test_compute_cid_base64_encoded() {
4343- // Simulate the actual use case: gzipped then base64 encoded
4444- use flate2::write::GzEncoder;
4545- use flate2::Compression;
4646- use std::io::Write;
4747-4848- let original = b"hello world";
4949-5050- // Gzip compress
5151- let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
5252- encoder.write_all(original).unwrap();
5353- let gzipped = encoder.finish().unwrap();
5454-5555- // Base64 encode the gzipped data
5656- let base64_bytes = base64::prelude::BASE64_STANDARD.encode(&gzipped).into_bytes();
5757-5858- // Compute CID on the base64 bytes
5959- let cid = compute_cid(&base64_bytes);
6060-6161- // Should be a valid CID
6262- assert!(cid.starts_with("baf"));
6363- assert!(cid.len() > 10);
6464- }
6565-}
6666-
-71
rust-cli/src/download.rs
···11-use base64::Engine;
22-use bytes::Bytes;
33-use flate2::read::GzDecoder;
44-use jacquard_common::types::blob::BlobRef;
55-use miette::IntoDiagnostic;
66-use std::io::Read;
77-use url::Url;
88-99-/// Download a blob from the PDS
1010-pub async fn download_blob(pds_url: &Url, blob_ref: &BlobRef<'_>, did: &str) -> miette::Result<Bytes> {
1111- // Extract CID from blob ref
1212- let cid = blob_ref.blob().r#ref.to_string();
1313-1414- // Construct blob download URL
1515- // The correct endpoint is: /xrpc/com.atproto.sync.getBlob?did={did}&cid={cid}
1616- let blob_url = pds_url
1717- .join(&format!("/xrpc/com.atproto.sync.getBlob?did={}&cid={}", did, cid))
1818- .into_diagnostic()?;
1919-2020- let client = reqwest::Client::new();
2121- let response = client
2222- .get(blob_url)
2323- .send()
2424- .await
2525- .into_diagnostic()?;
2626-2727- if !response.status().is_success() {
2828- return Err(miette::miette!(
2929- "Failed to download blob: {}",
3030- response.status()
3131- ));
3232- }
3333-3434- let bytes = response.bytes().await.into_diagnostic()?;
3535- Ok(bytes)
3636-}
3737-3838-/// Decompress and decode a blob (base64 + gzip)
3939-pub fn decompress_blob(data: &[u8], is_base64: bool, is_gzipped: bool) -> miette::Result<Vec<u8>> {
4040- let mut current_data = data.to_vec();
4141-4242- // First, decode base64 if needed
4343- if is_base64 {
4444- current_data = base64::prelude::BASE64_STANDARD
4545- .decode(¤t_data)
4646- .into_diagnostic()?;
4747- }
4848-4949- // Then, decompress gzip if needed
5050- if is_gzipped {
5151- let mut decoder = GzDecoder::new(¤t_data[..]);
5252- let mut decompressed = Vec::new();
5353- decoder.read_to_end(&mut decompressed).into_diagnostic()?;
5454- current_data = decompressed;
5555- }
5656-5757- Ok(current_data)
5858-}
5959-6060-/// Download and decompress a blob
6161-pub async fn download_and_decompress_blob(
6262- pds_url: &Url,
6363- blob_ref: &BlobRef<'_>,
6464- did: &str,
6565- is_base64: bool,
6666- is_gzipped: bool,
6767-) -> miette::Result<Vec<u8>> {
6868- let data = download_blob(pds_url, blob_ref, did).await?;
6969- decompress_blob(&data, is_base64, is_gzipped)
7070-}
7171-
-149
rust-cli/src/ignore_patterns.rs
···11-use globset::{Glob, GlobSet, GlobSetBuilder};
22-use serde::{Deserialize, Serialize};
33-use std::path::Path;
44-use miette::IntoDiagnostic;
55-66-#[derive(Debug, Deserialize, Serialize)]
77-struct IgnoreConfig {
88- patterns: Vec<String>,
99-}
1010-1111-/// Load ignore patterns from the default .wispignore.json file
1212-fn load_default_patterns() -> miette::Result<Vec<String>> {
1313- // Path to the default ignore patterns JSON file (in the monorepo root)
1414- let default_json_path = concat!(env!("CARGO_MANIFEST_DIR"), "/../.wispignore.json");
1515-1616- match std::fs::read_to_string(default_json_path) {
1717- Ok(contents) => {
1818- let config: IgnoreConfig = serde_json::from_str(&contents).into_diagnostic()?;
1919- Ok(config.patterns)
2020- }
2121- Err(_) => {
2222- // If the default file doesn't exist, return hardcoded patterns as fallback
2323- eprintln!("⚠️ Default .wispignore.json not found, using hardcoded patterns");
2424- Ok(get_hardcoded_patterns())
2525- }
2626- }
2727-}
2828-2929-/// Hardcoded fallback patterns (same as in .wispignore.json)
3030-fn get_hardcoded_patterns() -> Vec<String> {
3131- vec![
3232- ".git".to_string(),
3333- ".git/**".to_string(),
3434- ".github".to_string(),
3535- ".github/**".to_string(),
3636- ".gitlab".to_string(),
3737- ".gitlab/**".to_string(),
3838- ".DS_Store".to_string(),
3939- ".wisp.metadata.json".to_string(),
4040- ".env".to_string(),
4141- ".env.*".to_string(),
4242- "node_modules".to_string(),
4343- "node_modules/**".to_string(),
4444- "Thumbs.db".to_string(),
4545- "desktop.ini".to_string(),
4646- "._*".to_string(),
4747- ".Spotlight-V100".to_string(),
4848- ".Spotlight-V100/**".to_string(),
4949- ".Trashes".to_string(),
5050- ".Trashes/**".to_string(),
5151- ".fseventsd".to_string(),
5252- ".fseventsd/**".to_string(),
5353- ".cache".to_string(),
5454- ".cache/**".to_string(),
5555- ".temp".to_string(),
5656- ".temp/**".to_string(),
5757- ".tmp".to_string(),
5858- ".tmp/**".to_string(),
5959- "__pycache__".to_string(),
6060- "__pycache__/**".to_string(),
6161- "*.pyc".to_string(),
6262- ".venv".to_string(),
6363- ".venv/**".to_string(),
6464- "venv".to_string(),
6565- "venv/**".to_string(),
6666- "env".to_string(),
6767- "env/**".to_string(),
6868- "*.swp".to_string(),
6969- "*.swo".to_string(),
7070- "*~".to_string(),
7171- ".tangled".to_string(),
7272- ".tangled/**".to_string(),
7373- ]
7474-}
7575-7676-/// Load custom ignore patterns from a .wispignore file in the given directory
7777-fn load_wispignore_file(dir_path: &Path) -> miette::Result<Vec<String>> {
7878- let wispignore_path = dir_path.join(".wispignore");
7979-8080- if !wispignore_path.exists() {
8181- return Ok(Vec::new());
8282- }
8383-8484- let contents = std::fs::read_to_string(&wispignore_path).into_diagnostic()?;
8585-8686- // Parse gitignore-style file (one pattern per line, # for comments)
8787- let patterns: Vec<String> = contents
8888- .lines()
8989- .filter_map(|line| {
9090- let line = line.trim();
9191- // Skip empty lines and comments
9292- if line.is_empty() || line.starts_with('#') {
9393- None
9494- } else {
9595- Some(line.to_string())
9696- }
9797- })
9898- .collect();
9999-100100- if !patterns.is_empty() {
101101- println!("Loaded {} custom patterns from .wispignore", patterns.len());
102102- }
103103-104104- Ok(patterns)
105105-}
106106-107107-/// Build a GlobSet from a list of patterns
108108-fn build_globset(patterns: Vec<String>) -> miette::Result<GlobSet> {
109109- let mut builder = GlobSetBuilder::new();
110110-111111- for pattern in patterns {
112112- let glob = Glob::new(&pattern).into_diagnostic()?;
113113- builder.add(glob);
114114- }
115115-116116- let globset = builder.build().into_diagnostic()?;
117117- Ok(globset)
118118-}
119119-120120-/// IgnoreMatcher handles checking if paths should be ignored
121121-pub struct IgnoreMatcher {
122122- globset: GlobSet,
123123-}
124124-125125-impl IgnoreMatcher {
126126- /// Create a new IgnoreMatcher for the given directory
127127- /// Loads default patterns and any custom .wispignore file
128128- pub fn new(dir_path: &Path) -> miette::Result<Self> {
129129- let mut all_patterns = load_default_patterns()?;
130130-131131- // Load custom patterns from .wispignore
132132- let custom_patterns = load_wispignore_file(dir_path)?;
133133- all_patterns.extend(custom_patterns);
134134-135135- let globset = build_globset(all_patterns)?;
136136-137137- Ok(Self { globset })
138138- }
139139-140140- /// Check if the given path (relative to site root) should be ignored
141141- pub fn is_ignored(&self, path: &str) -> bool {
142142- self.globset.is_match(path)
143143- }
144144-145145- /// Check if a filename should be ignored (checks just the filename, not full path)
146146- pub fn is_filename_ignored(&self, filename: &str) -> bool {
147147- self.globset.is_match(filename)
148148- }
149149-}
-993
rust-cli/src/main.rs
···11-mod cid;
22-mod blob_map;
33-mod metadata;
44-mod download;
55-mod pull;
66-mod serve;
77-mod subfs_utils;
88-mod redirects;
99-mod ignore_patterns;
1010-1111-use clap::{Parser, Subcommand};
1212-use jacquard::CowStr;
1313-use jacquard::client::{Agent, FileAuthStore, AgentSessionExt, MemoryCredentialSession, AgentSession};
1414-use jacquard::oauth::client::OAuthClient;
1515-use jacquard::oauth::loopback::LoopbackConfig;
1616-use jacquard::prelude::IdentityResolver;
1717-use jacquard_common::types::string::{Datetime, Rkey, RecordKey, AtUri};
1818-use jacquard_common::types::blob::MimeType;
1919-use miette::IntoDiagnostic;
2020-use std::path::{Path, PathBuf};
2121-use std::collections::HashMap;
2222-use flate2::Compression;
2323-use flate2::write::GzEncoder;
2424-use std::io::Write;
2525-use base64::Engine;
2626-use futures::stream::{self, StreamExt};
2727-use indicatif::{ProgressBar, ProgressStyle, MultiProgress};
2828-2929-use wisp_lexicons::place_wisp::fs::*;
3030-use wisp_lexicons::place_wisp::settings::*;
3131-3232-/// Maximum number of concurrent file uploads to the PDS
3333-const MAX_CONCURRENT_UPLOADS: usize = 2;
3434-3535-/// Limits for caching on wisp.place (from @wisp/constants)
3636-const MAX_FILE_COUNT: usize = 1000;
3737-const MAX_SITE_SIZE: usize = 300 * 1024 * 1024; // 300MB
3838-3939-#[derive(Parser, Debug)]
4040-#[command(author, version, about = "wisp.place CLI tool")]
4141-struct Args {
4242- #[command(subcommand)]
4343- command: Option<Commands>,
4444-4545- // Deploy arguments (when no subcommand is specified)
4646- /// Handle (e.g., alice.bsky.social), DID, or PDS URL
4747- input: Option<CowStr<'static>>,
4848-4949- /// Path to the directory containing your static site
5050- #[arg(short, long)]
5151- path: Option<PathBuf>,
5252-5353- /// Site name (defaults to directory name)
5454- #[arg(short, long)]
5555- site: Option<String>,
5656-5757- /// Path to auth store file
5858- #[arg(long)]
5959- store: Option<String>,
6060-6161- /// App Password for authentication
6262- #[arg(long)]
6363- password: Option<CowStr<'static>>,
6464-6565- /// Enable directory listing mode for paths without index files
6666- #[arg(long)]
6767- directory: bool,
6868-6969- /// Enable SPA mode (serve index.html for all routes)
7070- #[arg(long)]
7171- spa: bool,
7272-7373- /// Skip confirmation prompts (automatically accept warnings)
7474- #[arg(short = 'y', long)]
7575- yes: bool,
7676-}
7777-7878-#[derive(Subcommand, Debug)]
7979-enum Commands {
8080- /// Deploy a static site to wisp.place (default command)
8181- Deploy {
8282- /// Handle (e.g., alice.bsky.social), DID, or PDS URL
8383- input: CowStr<'static>,
8484-8585- /// Path to the directory containing your static site
8686- #[arg(short, long, default_value = ".")]
8787- path: PathBuf,
8888-8989- /// Site name (defaults to directory name)
9090- #[arg(short, long)]
9191- site: Option<String>,
9292-9393- /// Path to auth store file (will be created if missing, only used with OAuth)
9494- #[arg(long, default_value = "/tmp/wisp-oauth-session.json")]
9595- store: String,
9696-9797- /// App Password for authentication (alternative to OAuth)
9898- #[arg(long)]
9999- password: Option<CowStr<'static>>,
100100-101101- /// Enable directory listing mode for paths without index files
102102- #[arg(long)]
103103- directory: bool,
104104-105105- /// Enable SPA mode (serve index.html for all routes)
106106- #[arg(long)]
107107- spa: bool,
108108-109109- /// Skip confirmation prompts (automatically accept warnings)
110110- #[arg(short = 'y', long)]
111111- yes: bool,
112112- },
113113- /// Pull a site from the PDS to a local directory
114114- Pull {
115115- /// Handle (e.g., alice.bsky.social) or DID
116116- input: CowStr<'static>,
117117-118118- /// Site name (record key)
119119- #[arg(short, long)]
120120- site: String,
121121-122122- /// Output directory for the downloaded site
123123- #[arg(short, long, default_value = ".")]
124124- path: PathBuf,
125125- },
126126- /// Serve a site locally with real-time firehose updates
127127- Serve {
128128- /// Handle (e.g., alice.bsky.social) or DID
129129- input: CowStr<'static>,
130130-131131- /// Site name (record key)
132132- #[arg(short, long)]
133133- site: String,
134134-135135- /// Output directory for the site files
136136- #[arg(short, long, default_value = ".")]
137137- path: PathBuf,
138138-139139- /// Port to serve on
140140- #[arg(short = 'P', long, default_value = "8080")]
141141- port: u16,
142142- },
143143-}
144144-145145-#[tokio::main]
146146-async fn main() -> miette::Result<()> {
147147- let args = Args::parse();
148148-149149- let result = match args.command {
150150- Some(Commands::Deploy { input, path, site, store, password, directory, spa, yes }) => {
151151- // Dispatch to appropriate authentication method
152152- if let Some(password) = password {
153153- run_with_app_password(input, password, path, site, directory, spa, yes).await
154154- } else {
155155- run_with_oauth(input, store, path, site, directory, spa, yes).await
156156- }
157157- }
158158- Some(Commands::Pull { input, site, path }) => {
159159- pull::pull_site(input, CowStr::from(site), path).await
160160- }
161161- Some(Commands::Serve { input, site, path, port }) => {
162162- serve::serve_site(input, CowStr::from(site), path, port).await
163163- }
164164- None => {
165165- // Legacy mode: if input is provided, assume deploy command
166166- if let Some(input) = args.input {
167167- let path = args.path.unwrap_or_else(|| PathBuf::from("."));
168168- let store = args.store.unwrap_or_else(|| "/tmp/wisp-oauth-session.json".to_string());
169169-170170- // Dispatch to appropriate authentication method
171171- if let Some(password) = args.password {
172172- run_with_app_password(input, password, path, args.site, args.directory, args.spa, args.yes).await
173173- } else {
174174- run_with_oauth(input, store, path, args.site, args.directory, args.spa, args.yes).await
175175- }
176176- } else {
177177- // No command and no input, show help
178178- use clap::CommandFactory;
179179- Args::command().print_help().into_diagnostic()?;
180180- Ok(())
181181- }
182182- }
183183- };
184184-185185- // Force exit to avoid hanging on background tasks/connections
186186- match result {
187187- Ok(_) => std::process::exit(0),
188188- Err(e) => {
189189- eprintln!("{:?}", e);
190190- std::process::exit(1)
191191- }
192192- }
193193-}
194194-195195-/// Run deployment with app password authentication
196196-async fn run_with_app_password(
197197- input: CowStr<'static>,
198198- password: CowStr<'static>,
199199- path: PathBuf,
200200- site: Option<String>,
201201- directory: bool,
202202- spa: bool,
203203- yes: bool,
204204-) -> miette::Result<()> {
205205- let (session, auth) =
206206- MemoryCredentialSession::authenticated(input, password, None, None).await?;
207207- println!("Signed in as {}", auth.handle);
208208-209209- let agent: Agent<_> = Agent::from(session);
210210- deploy_site(&agent, path, site, directory, spa, yes).await
211211-}
212212-213213-/// Run deployment with OAuth authentication
214214-async fn run_with_oauth(
215215- input: CowStr<'static>,
216216- store: String,
217217- path: PathBuf,
218218- site: Option<String>,
219219- directory: bool,
220220- spa: bool,
221221- yes: bool,
222222-) -> miette::Result<()> {
223223- use jacquard::oauth::scopes::Scope;
224224- use jacquard::oauth::atproto::AtprotoClientMetadata;
225225- use jacquard::oauth::session::ClientData;
226226- use url::Url;
227227-228228- // Request the necessary scopes for wisp.place (including settings)
229229- let scopes = Scope::parse_multiple("atproto repo:place.wisp.fs repo:place.wisp.subfs repo:place.wisp.settings blob:*/*")
230230- .map_err(|e| miette::miette!("Failed to parse scopes: {:?}", e))?;
231231-232232- // Create redirect URIs that match the loopback server (port 4000, path /oauth/callback)
233233- let redirect_uris = vec![
234234- Url::parse("http://127.0.0.1:4000/oauth/callback").into_diagnostic()?,
235235- Url::parse("http://[::1]:4000/oauth/callback").into_diagnostic()?,
236236- ];
237237-238238- // Create client metadata with matching redirect URIs and scopes
239239- let client_data = ClientData {
240240- keyset: None,
241241- config: AtprotoClientMetadata::new_localhost(
242242- Some(redirect_uris),
243243- Some(scopes),
244244- ),
245245- };
246246-247247- let oauth = OAuthClient::new(FileAuthStore::new(&store), client_data);
248248-249249- let session = oauth
250250- .login_with_local_server(input, Default::default(), LoopbackConfig::default())
251251- .await?;
252252-253253- let agent: Agent<_> = Agent::from(session);
254254- deploy_site(&agent, path, site, directory, spa, yes).await
255255-}
256256-257257-/// Scan directory to count files and calculate total size
258258-/// Returns (file_count, total_size_bytes)
259259-fn scan_directory_stats(
260260- dir_path: &Path,
261261- ignore_matcher: &ignore_patterns::IgnoreMatcher,
262262- current_path: String,
263263-) -> miette::Result<(usize, u64)> {
264264- let mut file_count = 0;
265265- let mut total_size = 0u64;
266266-267267- let dir_entries: Vec<_> = std::fs::read_dir(dir_path)
268268- .into_diagnostic()?
269269- .collect::<Result<Vec<_>, _>>()
270270- .into_diagnostic()?;
271271-272272- for entry in dir_entries {
273273- let path = entry.path();
274274- let name = entry.file_name();
275275- let name_str = name.to_str()
276276- .ok_or_else(|| miette::miette!("Invalid filename: {:?}", name))?
277277- .to_string();
278278-279279- let full_path = if current_path.is_empty() {
280280- name_str.clone()
281281- } else {
282282- format!("{}/{}", current_path, name_str)
283283- };
284284-285285- // Skip files/directories that match ignore patterns
286286- if ignore_matcher.is_ignored(&full_path) || ignore_matcher.is_filename_ignored(&name_str) {
287287- continue;
288288- }
289289-290290- let metadata = entry.metadata().into_diagnostic()?;
291291-292292- if metadata.is_file() {
293293- file_count += 1;
294294- total_size += metadata.len();
295295- } else if metadata.is_dir() {
296296- let subdir_path = if current_path.is_empty() {
297297- name_str
298298- } else {
299299- format!("{}/{}", current_path, name_str)
300300- };
301301- let (sub_count, sub_size) = scan_directory_stats(&path, ignore_matcher, subdir_path)?;
302302- file_count += sub_count;
303303- total_size += sub_size;
304304- }
305305- }
306306-307307- Ok((file_count, total_size))
308308-}
309309-310310-/// Deploy the site using the provided agent
311311-async fn deploy_site(
312312- agent: &Agent<impl jacquard::client::AgentSession + IdentityResolver>,
313313- path: PathBuf,
314314- site: Option<String>,
315315- directory_listing: bool,
316316- spa_mode: bool,
317317- skip_prompts: bool,
318318-) -> miette::Result<()> {
319319- // Verify the path exists
320320- if !path.exists() {
321321- return Err(miette::miette!("Path does not exist: {}", path.display()));
322322- }
323323-324324- // Get site name
325325- let site_name = site.unwrap_or_else(|| {
326326- path
327327- .file_name()
328328- .and_then(|n| n.to_str())
329329- .unwrap_or("site")
330330- .to_string()
331331- });
332332-333333- println!("Deploying site '{}'...", site_name);
334334-335335- // Scan directory to check file count and size
336336- let ignore_matcher = ignore_patterns::IgnoreMatcher::new(&path)?;
337337- let (file_count, total_size) = scan_directory_stats(&path, &ignore_matcher, String::new())?;
338338-339339- let size_mb = total_size as f64 / (1024.0 * 1024.0);
340340- println!("Scanned: {} files, {:.1} MB total", file_count, size_mb);
341341-342342- // Check if limits are exceeded
343343- let exceeds_file_count = file_count > MAX_FILE_COUNT;
344344- let exceeds_size = total_size > MAX_SITE_SIZE as u64;
345345-346346- if exceeds_file_count || exceeds_size {
347347- println!("\n⚠️ Warning: Your site exceeds wisp.place caching limits:");
348348-349349- if exceeds_file_count {
350350- println!(" • File count: {} (limit: {})", file_count, MAX_FILE_COUNT);
351351- }
352352-353353- if exceeds_size {
354354- let size_mb = total_size as f64 / (1024.0 * 1024.0);
355355- let limit_mb = MAX_SITE_SIZE as f64 / (1024.0 * 1024.0);
356356- println!(" • Total size: {:.1} MB (limit: {:.0} MB)", size_mb, limit_mb);
357357- }
358358-359359- println!("\nwisp.place will NOT serve your site if you proceed.");
360360- println!("Your site will be uploaded to your PDS, but will only be accessible via:");
361361- println!(" • wisp-cli serve (local hosting)");
362362- println!(" • Other hosting services with more generous limits");
363363-364364- if !skip_prompts {
365365- // Prompt for confirmation
366366- use std::io::{self, Write};
367367- print!("\nDo you want to upload anyway? (y/N): ");
368368- io::stdout().flush().into_diagnostic()?;
369369-370370- let mut input = String::new();
371371- io::stdin().read_line(&mut input).into_diagnostic()?;
372372- let input = input.trim().to_lowercase();
373373-374374- if input != "y" && input != "yes" {
375375- println!("Upload cancelled.");
376376- return Ok(());
377377- }
378378- } else {
379379- println!("\nSkipping confirmation (--yes flag set).");
380380- }
381381-382382- println!("\nProceeding with upload...\n");
383383- }
384384-385385- // Try to fetch existing manifest for incremental updates
386386- let (existing_blob_map, old_subfs_uris): (HashMap<String, (jacquard_common::types::blob::BlobRef<'static>, String)>, Vec<(String, String)>) = {
387387- use jacquard_common::types::string::AtUri;
388388-389389- // Get the DID for this session
390390- let session_info = agent.session_info().await;
391391- if let Some((did, _)) = session_info {
392392- // Construct the AT URI for the record
393393- let uri_string = format!("at://{}/place.wisp.fs/{}", did, site_name);
394394- if let Ok(uri) = AtUri::new(&uri_string) {
395395- match agent.get_record::<Fs>(&uri).await {
396396- Ok(response) => {
397397- match response.into_output() {
398398- Ok(record_output) => {
399399- let existing_manifest = record_output.value;
400400- let mut blob_map = blob_map::extract_blob_map(&existing_manifest.root);
401401- println!("Found existing manifest with {} files in main record", blob_map.len());
402402-403403- // Extract subfs URIs from main record
404404- let subfs_uris = subfs_utils::extract_subfs_uris(&existing_manifest.root, String::new());
405405-406406- if !subfs_uris.is_empty() {
407407- println!("Found {} subfs records, fetching for blob reuse...", subfs_uris.len());
408408-409409- // Merge blob maps from all subfs records
410410- match subfs_utils::merge_subfs_blob_maps(agent, subfs_uris.clone(), &mut blob_map).await {
411411- Ok(merged_count) => {
412412- println!("Total blob map: {} files (main + {} from subfs)", blob_map.len(), merged_count);
413413- }
414414- Err(e) => {
415415- eprintln!("⚠️ Failed to merge some subfs blob maps: {}", e);
416416- }
417417- }
418418-419419- (blob_map, subfs_uris)
420420- } else {
421421- (blob_map, Vec::new())
422422- }
423423- }
424424- Err(_) => {
425425- println!("No existing manifest found, uploading all files...");
426426- (HashMap::new(), Vec::new())
427427- }
428428- }
429429- }
430430- Err(_) => {
431431- // Record doesn't exist yet - this is a new site
432432- println!("No existing manifest found, uploading all files...");
433433- (HashMap::new(), Vec::new())
434434- }
435435- }
436436- } else {
437437- println!("No existing manifest found (invalid URI), uploading all files...");
438438- (HashMap::new(), Vec::new())
439439- }
440440- } else {
441441- println!("No existing manifest found (could not get DID), uploading all files...");
442442- (HashMap::new(), Vec::new())
443443- }
444444- };
445445-446446- // Create progress tracking (spinner style since we don't know total count upfront)
447447- let multi_progress = MultiProgress::new();
448448- let progress = multi_progress.add(ProgressBar::new_spinner());
449449- progress.set_style(
450450- ProgressStyle::default_spinner()
451451- .template("[{elapsed_precise}] {spinner:.cyan} {pos} files {msg}")
452452- .into_diagnostic()?
453453- .tick_chars("⠁⠂⠄⡀⢀⠠⠐⠈ ")
454454- );
455455- progress.set_message("Scanning files...");
456456- progress.enable_steady_tick(std::time::Duration::from_millis(100));
457457-458458- let (root_dir, total_files, reused_count) = build_directory(agent, &path, &existing_blob_map, String::new(), &ignore_matcher, &progress).await?;
459459- let uploaded_count = total_files - reused_count;
460460-461461- progress.finish_with_message(format!("✓ {} files ({} uploaded, {} reused)", total_files, uploaded_count, reused_count));
462462-463463- // Check if we need to split into subfs records
464464- const MAX_MANIFEST_SIZE: usize = 140 * 1024; // 140KB (PDS limit is 150KB)
465465- const FILE_COUNT_THRESHOLD: usize = 250; // Start splitting at this many files
466466- const TARGET_FILE_COUNT: usize = 200; // Keep main manifest under this
467467-468468- let mut working_directory = root_dir;
469469- let mut current_file_count = total_files;
470470- let mut new_subfs_uris: Vec<(String, String)> = Vec::new();
471471-472472- // Estimate initial manifest size
473473- let mut manifest_size = subfs_utils::estimate_directory_size(&working_directory);
474474-475475- if total_files >= FILE_COUNT_THRESHOLD || manifest_size > MAX_MANIFEST_SIZE {
476476- println!("\n⚠️ Large site detected ({} files, {:.1}KB manifest), splitting into subfs records...",
477477- total_files, manifest_size as f64 / 1024.0);
478478-479479- let mut attempts = 0;
480480- const MAX_SPLIT_ATTEMPTS: usize = 50;
481481-482482- while (manifest_size > MAX_MANIFEST_SIZE || current_file_count > TARGET_FILE_COUNT) && attempts < MAX_SPLIT_ATTEMPTS {
483483- attempts += 1;
484484-485485- // Find large directories to split
486486- let directories = subfs_utils::find_large_directories(&working_directory, String::new());
487487-488488- if let Some(largest_dir) = directories.first() {
489489- println!(" Split #{}: {} ({} files, {:.1}KB)",
490490- attempts, largest_dir.path, largest_dir.file_count, largest_dir.size as f64 / 1024.0);
491491-492492- // Check if this directory is itself too large for a single subfs record
493493- const MAX_SUBFS_SIZE: usize = 75 * 1024; // 75KB soft limit for safety
494494- let mut subfs_uri = String::new();
495495-496496- if largest_dir.size > MAX_SUBFS_SIZE {
497497- // Need to split this directory into multiple chunks
498498- println!(" → Directory too large, splitting into chunks...");
499499- let chunks = subfs_utils::split_directory_into_chunks(&largest_dir.directory, MAX_SUBFS_SIZE);
500500- println!(" → Created {} chunks", chunks.len());
501501-502502- // Upload each chunk as a subfs record
503503- let mut chunk_uris = Vec::new();
504504- for (i, chunk) in chunks.iter().enumerate() {
505505- use jacquard_common::types::string::Tid;
506506- let chunk_tid = Tid::now_0();
507507- let chunk_rkey = chunk_tid.to_string();
508508-509509- let chunk_file_count = subfs_utils::count_files_in_directory(chunk);
510510- let chunk_size = subfs_utils::estimate_directory_size(chunk);
511511-512512- let chunk_manifest = wisp_lexicons::place_wisp::subfs::SubfsRecord::new()
513513- .root(convert_fs_dir_to_subfs_dir(chunk.clone()))
514514- .file_count(Some(chunk_file_count as i64))
515515- .created_at(Datetime::now())
516516- .build();
517517-518518- println!(" → Uploading chunk {}/{} ({} files, {:.1}KB)...",
519519- i + 1, chunks.len(), chunk_file_count, chunk_size as f64 / 1024.0);
520520-521521- let chunk_output = agent.put_record(
522522- RecordKey::from(Rkey::new(&chunk_rkey).into_diagnostic()?),
523523- chunk_manifest
524524- ).await.into_diagnostic()?;
525525-526526- let chunk_uri = chunk_output.uri.to_string();
527527- chunk_uris.push((chunk_uri.clone(), format!("{}#{}", largest_dir.path, i)));
528528- new_subfs_uris.push((chunk_uri.clone(), format!("{}#{}", largest_dir.path, i)));
529529- }
530530-531531- // Create a parent subfs record that references all chunks
532532- // Each chunk reference MUST have flat: true to merge chunk contents
533533- println!(" → Creating parent subfs with {} chunk references...", chunk_uris.len());
534534- use jacquard_common::CowStr;
535535- use wisp_lexicons::place_wisp::fs::{Subfs};
536536-537537- // Convert to fs::Subfs (which has the 'flat' field) instead of subfs::Subfs
538538- let parent_entries_fs: Vec<Entry> = chunk_uris.iter().enumerate().map(|(i, (uri, _))| {
539539- let uri_string = uri.clone();
540540- let at_uri = AtUri::new_cow(CowStr::from(uri_string)).expect("valid URI");
541541- Entry::new()
542542- .name(CowStr::from(format!("chunk{}", i)))
543543- .node(EntryNode::Subfs(Box::new(
544544- Subfs::new()
545545- .r#type(CowStr::from("subfs"))
546546- .subject(at_uri)
547547- .flat(Some(true)) // EXPLICITLY TRUE - merge chunk contents
548548- .build()
549549- )))
550550- .build()
551551- }).collect();
552552-553553- let parent_root_fs = Directory::new()
554554- .r#type(CowStr::from("directory"))
555555- .entries(parent_entries_fs)
556556- .build();
557557-558558- // Convert to subfs::Directory for the parent subfs record
559559- let parent_root_subfs = convert_fs_dir_to_subfs_dir(parent_root_fs);
560560-561561- use jacquard_common::types::string::Tid;
562562- let parent_tid = Tid::now_0();
563563- let parent_rkey = parent_tid.to_string();
564564-565565- let parent_manifest = wisp_lexicons::place_wisp::subfs::SubfsRecord::new()
566566- .root(parent_root_subfs)
567567- .file_count(Some(largest_dir.file_count as i64))
568568- .created_at(Datetime::now())
569569- .build();
570570-571571- let parent_output = agent.put_record(
572572- RecordKey::from(Rkey::new(&parent_rkey).into_diagnostic()?),
573573- parent_manifest
574574- ).await.into_diagnostic()?;
575575-576576- subfs_uri = parent_output.uri.to_string();
577577- println!(" ✅ Created parent subfs with chunks (flat=true on each chunk): {}", subfs_uri);
578578- } else {
579579- // Directory fits in a single subfs record
580580- use jacquard_common::types::string::Tid;
581581- let subfs_tid = Tid::now_0();
582582- let subfs_rkey = subfs_tid.to_string();
583583-584584- let subfs_manifest = wisp_lexicons::place_wisp::subfs::SubfsRecord::new()
585585- .root(convert_fs_dir_to_subfs_dir(largest_dir.directory.clone()))
586586- .file_count(Some(largest_dir.file_count as i64))
587587- .created_at(Datetime::now())
588588- .build();
589589-590590- // Upload subfs record
591591- let subfs_output = agent.put_record(
592592- RecordKey::from(Rkey::new(&subfs_rkey).into_diagnostic()?),
593593- subfs_manifest
594594- ).await.into_diagnostic()?;
595595-596596- subfs_uri = subfs_output.uri.to_string();
597597- println!(" ✅ Created subfs: {}", subfs_uri);
598598- }
599599-600600- // Replace directory with subfs node (flat: false to preserve directory structure)
601601- working_directory = subfs_utils::replace_directory_with_subfs(
602602- working_directory,
603603- &largest_dir.path,
604604- &subfs_uri,
605605- false // Preserve directory - the chunks inside have flat=true
606606- )?;
607607-608608- new_subfs_uris.push((subfs_uri, largest_dir.path.clone()));
609609- current_file_count -= largest_dir.file_count;
610610-611611- // Recalculate manifest size
612612- manifest_size = subfs_utils::estimate_directory_size(&working_directory);
613613- println!(" → Manifest now {:.1}KB with {} files ({} subfs total)",
614614- manifest_size as f64 / 1024.0, current_file_count, new_subfs_uris.len());
615615-616616- if manifest_size <= MAX_MANIFEST_SIZE && current_file_count <= TARGET_FILE_COUNT {
617617- println!("✅ Manifest now fits within limits");
618618- break;
619619- }
620620- } else {
621621- println!(" No more subdirectories to split - stopping");
622622- break;
623623- }
624624- }
625625-626626- if attempts >= MAX_SPLIT_ATTEMPTS {
627627- return Err(miette::miette!(
628628- "Exceeded maximum split attempts ({}). Manifest still too large: {:.1}KB with {} files",
629629- MAX_SPLIT_ATTEMPTS,
630630- manifest_size as f64 / 1024.0,
631631- current_file_count
632632- ));
633633- }
634634-635635- println!("✅ Split complete: {} subfs records, {} files in main manifest, {:.1}KB",
636636- new_subfs_uris.len(), current_file_count, manifest_size as f64 / 1024.0);
637637- } else {
638638- println!("Manifest created ({} files, {:.1}KB) - no splitting needed",
639639- total_files, manifest_size as f64 / 1024.0);
640640- }
641641-642642- // Create the final Fs record
643643- let fs_record = Fs::new()
644644- .site(CowStr::from(site_name.clone()))
645645- .root(working_directory)
646646- .file_count(current_file_count as i64)
647647- .created_at(Datetime::now())
648648- .build();
649649-650650- // Use site name as the record key
651651- let rkey = Rkey::new(&site_name).map_err(|e| miette::miette!("Invalid rkey: {}", e))?;
652652- let output = agent.put_record(RecordKey::from(rkey), fs_record).await?;
653653-654654- // Extract DID from the AT URI (format: at://did:plc:xxx/collection/rkey)
655655- let uri_str = output.uri.to_string();
656656- let did = uri_str
657657- .strip_prefix("at://")
658658- .and_then(|s| s.split('/').next())
659659- .ok_or_else(|| miette::miette!("Failed to parse DID from URI"))?;
660660-661661- println!("\n✓ Deployed site '{}': {}", site_name, output.uri);
662662- println!(" Total files: {} ({} reused, {} uploaded)", total_files, reused_count, uploaded_count);
663663- println!(" Available at: https://sites.wisp.place/{}/{}", did, site_name);
664664-665665- // Clean up old subfs records
666666- if !old_subfs_uris.is_empty() {
667667- println!("\nCleaning up {} old subfs records...", old_subfs_uris.len());
668668-669669- let mut deleted_count = 0;
670670- let mut failed_count = 0;
671671-672672- for (uri, _path) in old_subfs_uris {
673673- match subfs_utils::delete_subfs_record(agent, &uri).await {
674674- Ok(_) => {
675675- deleted_count += 1;
676676- println!(" 🗑️ Deleted old subfs: {}", uri);
677677- }
678678- Err(e) => {
679679- failed_count += 1;
680680- eprintln!(" ⚠️ Failed to delete {}: {}", uri, e);
681681- }
682682- }
683683- }
684684-685685- if failed_count > 0 {
686686- eprintln!("⚠️ Cleanup completed with {} deleted, {} failed", deleted_count, failed_count);
687687- } else {
688688- println!("✅ Cleanup complete: {} old subfs records deleted", deleted_count);
689689- }
690690- }
691691-692692- // Upload settings if either flag is set
693693- if directory_listing || spa_mode {
694694- // Validate mutual exclusivity
695695- if directory_listing && spa_mode {
696696- return Err(miette::miette!("Cannot enable both --directory and --SPA modes"));
697697- }
698698-699699- println!("\n⚙️ Uploading site settings...");
700700-701701- // Build settings record
702702- let mut settings_builder = Settings::new();
703703-704704- if directory_listing {
705705- settings_builder = settings_builder.directory_listing(Some(true));
706706- println!(" • Directory listing: enabled");
707707- }
708708-709709- if spa_mode {
710710- settings_builder = settings_builder.spa_mode(Some(CowStr::from("index.html")));
711711- println!(" • SPA mode: enabled (serving index.html for all routes)");
712712- }
713713-714714- let settings_record = settings_builder.build();
715715-716716- // Upload settings record with same rkey as site
717717- let rkey = Rkey::new(&site_name).map_err(|e| miette::miette!("Invalid rkey: {}", e))?;
718718- match agent.put_record(RecordKey::from(rkey), settings_record).await {
719719- Ok(settings_output) => {
720720- println!("✅ Settings uploaded: {}", settings_output.uri);
721721- }
722722- Err(e) => {
723723- eprintln!("⚠️ Failed to upload settings: {}", e);
724724- eprintln!(" Site was deployed successfully, but settings may need to be configured manually.");
725725- }
726726- }
727727- }
728728-729729- Ok(())
730730-}
731731-732732-/// Recursively build a Directory from a filesystem path
733733-/// current_path is the path from the root of the site (e.g., "" for root, "config" for config dir)
734734-fn build_directory<'a>(
735735- agent: &'a Agent<impl jacquard::client::AgentSession + IdentityResolver + 'a>,
736736- dir_path: &'a Path,
737737- existing_blobs: &'a HashMap<String, (jacquard_common::types::blob::BlobRef<'static>, String)>,
738738- current_path: String,
739739- ignore_matcher: &'a ignore_patterns::IgnoreMatcher,
740740- progress: &'a ProgressBar,
741741-) -> std::pin::Pin<Box<dyn std::future::Future<Output = miette::Result<(Directory<'static>, usize, usize)>> + 'a>>
742742-{
743743- Box::pin(async move {
744744- // Collect all directory entries first
745745- let dir_entries: Vec<_> = std::fs::read_dir(dir_path)
746746- .into_diagnostic()?
747747- .collect::<Result<Vec<_>, _>>()
748748- .into_diagnostic()?;
749749-750750- // Separate files and directories
751751- let mut file_tasks = Vec::new();
752752- let mut dir_tasks = Vec::new();
753753-754754- for entry in dir_entries {
755755- let path = entry.path();
756756- let name = entry.file_name();
757757- let name_str = name.to_str()
758758- .ok_or_else(|| miette::miette!("Invalid filename: {:?}", name))?
759759- .to_string();
760760-761761- // Construct full path for ignore checking
762762- let full_path = if current_path.is_empty() {
763763- name_str.clone()
764764- } else {
765765- format!("{}/{}", current_path, name_str)
766766- };
767767-768768- // Skip files/directories that match ignore patterns
769769- if ignore_matcher.is_ignored(&full_path) || ignore_matcher.is_filename_ignored(&name_str) {
770770- continue;
771771- }
772772-773773- let metadata = entry.metadata().into_diagnostic()?;
774774-775775- if metadata.is_file() {
776776- // Construct full path for this file (for blob map lookup)
777777- let full_path = if current_path.is_empty() {
778778- name_str.clone()
779779- } else {
780780- format!("{}/{}", current_path, name_str)
781781- };
782782- file_tasks.push((name_str, path, full_path));
783783- } else if metadata.is_dir() {
784784- dir_tasks.push((name_str, path));
785785- }
786786- }
787787-788788- // Process files concurrently with a limit of 2
789789- let file_results: Vec<(Entry<'static>, bool)> = stream::iter(file_tasks)
790790- .map(|(name, path, full_path)| async move {
791791- let (file_node, reused) = process_file(agent, &path, &full_path, existing_blobs, progress).await?;
792792- progress.inc(1);
793793- let entry = Entry::new()
794794- .name(CowStr::from(name))
795795- .node(EntryNode::File(Box::new(file_node)))
796796- .build();
797797- Ok::<_, miette::Report>((entry, reused))
798798- })
799799- .buffer_unordered(MAX_CONCURRENT_UPLOADS)
800800- .collect::<Vec<_>>()
801801- .await
802802- .into_iter()
803803- .collect::<miette::Result<Vec<_>>>()?;
804804-805805- let mut file_entries = Vec::new();
806806- let mut reused_count = 0;
807807- let mut total_files = 0;
808808-809809- for (entry, reused) in file_results {
810810- file_entries.push(entry);
811811- total_files += 1;
812812- if reused {
813813- reused_count += 1;
814814- }
815815- }
816816-817817- // Process directories recursively (sequentially to avoid too much nesting)
818818- let mut dir_entries = Vec::new();
819819- for (name, path) in dir_tasks {
820820- // Construct full path for subdirectory
821821- let subdir_path = if current_path.is_empty() {
822822- name.clone()
823823- } else {
824824- format!("{}/{}", current_path, name)
825825- };
826826- let (subdir, sub_total, sub_reused) = build_directory(agent, &path, existing_blobs, subdir_path, ignore_matcher, progress).await?;
827827- dir_entries.push(Entry::new()
828828- .name(CowStr::from(name))
829829- .node(EntryNode::Directory(Box::new(subdir)))
830830- .build());
831831- total_files += sub_total;
832832- reused_count += sub_reused;
833833- }
834834-835835- // Combine file and directory entries
836836- let mut entries = file_entries;
837837- entries.extend(dir_entries);
838838-839839- let directory = Directory::new()
840840- .r#type(CowStr::from("directory"))
841841- .entries(entries)
842842- .build();
843843-844844- Ok((directory, total_files, reused_count))
845845- })
846846-}
847847-848848-/// Process a single file: gzip -> base64 -> upload blob (or reuse existing)
849849-/// Returns (File, reused: bool)
850850-/// file_path_key is the full path from the site root (e.g., "config/file.json") for blob map lookup
851851-///
852852-/// Special handling: _redirects files are NOT compressed (uploaded as-is)
853853-async fn process_file(
854854- agent: &Agent<impl jacquard::client::AgentSession + IdentityResolver>,
855855- file_path: &Path,
856856- file_path_key: &str,
857857- existing_blobs: &HashMap<String, (jacquard_common::types::blob::BlobRef<'static>, String)>,
858858- progress: &ProgressBar,
859859-) -> miette::Result<(File<'static>, bool)>
860860-{
861861- // Read file
862862- let file_data = std::fs::read(file_path).into_diagnostic()?;
863863-864864- // Detect original MIME type
865865- let original_mime = mime_guess::from_path(file_path)
866866- .first_or_octet_stream()
867867- .to_string();
868868-869869- // Check if this is a _redirects file (don't compress it)
870870- let is_redirects_file = file_path.file_name()
871871- .and_then(|n| n.to_str())
872872- .map(|n| n == "_redirects")
873873- .unwrap_or(false);
874874-875875- let (upload_bytes, encoding, is_base64) = if is_redirects_file {
876876- // Don't compress _redirects - upload as-is
877877- (file_data.clone(), None, false)
878878- } else {
879879- // Gzip compress
880880- let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
881881- encoder.write_all(&file_data).into_diagnostic()?;
882882- let gzipped = encoder.finish().into_diagnostic()?;
883883-884884- // Base64 encode the gzipped data
885885- let base64_bytes = base64::prelude::BASE64_STANDARD.encode(&gzipped).into_bytes();
886886- (base64_bytes, Some("gzip"), true)
887887- };
888888-889889- // Compute CID for this file
890890- let file_cid = cid::compute_cid(&upload_bytes);
891891-892892- // Check if we have an existing blob with the same CID
893893- let existing_blob = existing_blobs.get(file_path_key);
894894-895895- if let Some((existing_blob_ref, existing_cid)) = existing_blob {
896896- if existing_cid == &file_cid {
897897- // CIDs match - reuse existing blob
898898- progress.set_message(format!("✓ Reused {}", file_path_key));
899899- let mut file_builder = File::new()
900900- .r#type(CowStr::from("file"))
901901- .blob(existing_blob_ref.clone())
902902- .mime_type(CowStr::from(original_mime));
903903-904904- if let Some(enc) = encoding {
905905- file_builder = file_builder.encoding(CowStr::from(enc));
906906- }
907907- if is_base64 {
908908- file_builder = file_builder.base64(true);
909909- }
910910-911911- return Ok((file_builder.build(), true));
912912- }
913913- }
914914-915915- // File is new or changed - upload it
916916- let mime_type = if is_redirects_file {
917917- MimeType::new_static("text/plain")
918918- } else {
919919- MimeType::new_static("application/octet-stream")
920920- };
921921-922922- // Format file size nicely
923923- let size_str = if upload_bytes.len() < 1024 {
924924- format!("{} B", upload_bytes.len())
925925- } else if upload_bytes.len() < 1024 * 1024 {
926926- format!("{:.1} KB", upload_bytes.len() as f64 / 1024.0)
927927- } else {
928928- format!("{:.1} MB", upload_bytes.len() as f64 / (1024.0 * 1024.0))
929929- };
930930-931931- progress.set_message(format!("↑ Uploading {} ({})", file_path_key, size_str));
932932- let blob = agent.upload_blob(upload_bytes, mime_type).await?;
933933- progress.set_message(format!("✓ Uploaded {}", file_path_key));
934934-935935- let mut file_builder = File::new()
936936- .r#type(CowStr::from("file"))
937937- .blob(blob)
938938- .mime_type(CowStr::from(original_mime));
939939-940940- if let Some(enc) = encoding {
941941- file_builder = file_builder.encoding(CowStr::from(enc));
942942- }
943943- if is_base64 {
944944- file_builder = file_builder.base64(true);
945945- }
946946-947947- Ok((file_builder.build(), false))
948948-}
949949-950950-/// Convert fs::Directory to subfs::Directory
951951-/// They have the same structure, but different types
952952-fn convert_fs_dir_to_subfs_dir(fs_dir: wisp_lexicons::place_wisp::fs::Directory<'static>) -> wisp_lexicons::place_wisp::subfs::Directory<'static> {
953953- use wisp_lexicons::place_wisp::subfs::{Directory as SubfsDirectory, Entry as SubfsEntry, EntryNode as SubfsEntryNode, File as SubfsFile};
954954-955955- let subfs_entries: Vec<SubfsEntry> = fs_dir.entries.into_iter().map(|entry| {
956956- let node = match entry.node {
957957- wisp_lexicons::place_wisp::fs::EntryNode::File(file) => {
958958- SubfsEntryNode::File(Box::new(SubfsFile::new()
959959- .r#type(file.r#type)
960960- .blob(file.blob)
961961- .encoding(file.encoding)
962962- .mime_type(file.mime_type)
963963- .base64(file.base64)
964964- .build()))
965965- }
966966- wisp_lexicons::place_wisp::fs::EntryNode::Directory(dir) => {
967967- SubfsEntryNode::Directory(Box::new(convert_fs_dir_to_subfs_dir(*dir)))
968968- }
969969- wisp_lexicons::place_wisp::fs::EntryNode::Subfs(subfs) => {
970970- // Nested subfs in the directory we're converting
971971- // Note: subfs::Subfs doesn't have the 'flat' field - that's only in fs::Subfs
972972- SubfsEntryNode::Subfs(Box::new(wisp_lexicons::place_wisp::subfs::Subfs::new()
973973- .r#type(subfs.r#type)
974974- .subject(subfs.subject)
975975- .build()))
976976- }
977977- wisp_lexicons::place_wisp::fs::EntryNode::Unknown(unknown) => {
978978- SubfsEntryNode::Unknown(unknown)
979979- }
980980- };
981981-982982- SubfsEntry::new()
983983- .name(entry.name)
984984- .node(node)
985985- .build()
986986- }).collect();
987987-988988- SubfsDirectory::new()
989989- .r#type(fs_dir.r#type)
990990- .entries(subfs_entries)
991991- .build()
992992-}
993993-
-46
rust-cli/src/metadata.rs
···11-use serde::{Deserialize, Serialize};
22-use std::collections::HashMap;
33-use std::path::Path;
44-use miette::IntoDiagnostic;
55-66-/// Metadata tracking file CIDs for incremental updates
77-#[derive(Debug, Clone, Serialize, Deserialize)]
88-pub struct SiteMetadata {
99- /// Record CID from the PDS
1010- pub record_cid: String,
1111- /// Map of file paths to their blob CIDs
1212- pub file_cids: HashMap<String, String>,
1313- /// Timestamp when the site was last synced
1414- pub last_sync: i64,
1515-}
1616-1717-impl SiteMetadata {
1818- pub fn new(record_cid: String, file_cids: HashMap<String, String>) -> Self {
1919- Self {
2020- record_cid,
2121- file_cids,
2222- last_sync: chrono::Utc::now().timestamp(),
2323- }
2424- }
2525-2626- /// Load metadata from a directory
2727- pub fn load(dir: &Path) -> miette::Result<Option<Self>> {
2828- let metadata_path = dir.join(".wisp-metadata.json");
2929- if !metadata_path.exists() {
3030- return Ok(None);
3131- }
3232-3333- let contents = std::fs::read_to_string(&metadata_path).into_diagnostic()?;
3434- let metadata: SiteMetadata = serde_json::from_str(&contents).into_diagnostic()?;
3535- Ok(Some(metadata))
3636- }
3737-3838- /// Save metadata to a directory
3939- pub fn save(&self, dir: &Path) -> miette::Result<()> {
4040- let metadata_path = dir.join(".wisp-metadata.json");
4141- let contents = serde_json::to_string_pretty(self).into_diagnostic()?;
4242- std::fs::write(&metadata_path, contents).into_diagnostic()?;
4343- Ok(())
4444- }
4545-}
4646-
-681
rust-cli/src/pull.rs
···11-use crate::blob_map;
22-use crate::download;
33-use crate::metadata::SiteMetadata;
44-use wisp_lexicons::place_wisp::fs::*;
55-use crate::subfs_utils;
66-use jacquard::CowStr;
77-use jacquard::prelude::IdentityResolver;
88-use jacquard_common::types::string::Did;
99-use jacquard_common::xrpc::XrpcExt;
1010-use jacquard_identity::PublicResolver;
1111-use miette::IntoDiagnostic;
1212-use std::collections::HashMap;
1313-use std::path::{Path, PathBuf};
1414-use url::Url;
1515-1616-/// Pull a site from the PDS to a local directory
1717-pub async fn pull_site(
1818- input: CowStr<'static>,
1919- rkey: CowStr<'static>,
2020- output_dir: PathBuf,
2121-) -> miette::Result<()> {
2222- println!("Pulling site {} from {}...", rkey, input);
2323-2424- // Resolve handle to DID if needed
2525- let resolver = PublicResolver::default();
2626- let did = if input.starts_with("did:") {
2727- Did::new(&input).into_diagnostic()?
2828- } else {
2929- // It's a handle, resolve it
3030- let handle = jacquard_common::types::string::Handle::new(&input).into_diagnostic()?;
3131- resolver.resolve_handle(&handle).await.into_diagnostic()?
3232- };
3333-3434- // Resolve PDS endpoint for the DID
3535- let pds_url = resolver.pds_for_did(&did).await.into_diagnostic()?;
3636- println!("Resolved PDS: {}", pds_url);
3737-3838- // Create a temporary agent for fetching records (no auth needed for public reads)
3939- println!("Fetching record from PDS...");
4040- let client = reqwest::Client::new();
4141-4242- // Use com.atproto.repo.getRecord
4343- use jacquard::api::com_atproto::repo::get_record::GetRecord;
4444- use jacquard_common::types::string::Rkey as RkeyType;
4545- let rkey_parsed = RkeyType::new(&rkey).into_diagnostic()?;
4646-4747- use jacquard_common::types::ident::AtIdentifier;
4848- use jacquard_common::types::string::RecordKey;
4949- let request = GetRecord::new()
5050- .repo(AtIdentifier::Did(did.clone()))
5151- .collection(CowStr::from("place.wisp.fs"))
5252- .rkey(RecordKey::from(rkey_parsed))
5353- .build();
5454-5555- let response = client
5656- .xrpc(pds_url.clone())
5757- .send(&request)
5858- .await
5959- .into_diagnostic()?;
6060-6161- let record_output = response.into_output().into_diagnostic()?;
6262- let record_cid = record_output.cid.as_ref().map(|c| c.to_string()).unwrap_or_default();
6363-6464- // Parse the record value as Fs
6565- use jacquard_common::types::value::from_data;
6666- let fs_record: Fs = from_data(&record_output.value).into_diagnostic()?;
6767-6868- let file_count = fs_record.file_count.map(|c| c.to_string()).unwrap_or_else(|| "?".to_string());
6969- println!("Found site '{}' with {} files (in main record)", fs_record.site, file_count);
7070-7171- // Check for and expand subfs nodes
7272- // Note: We use a custom expand function for pull since we don't have an Agent
7373- let expanded_root = expand_subfs_in_pull_with_client(&fs_record.root, &client, &pds_url).await?;
7474- let total_file_count = subfs_utils::count_files_in_directory(&expanded_root);
7575-7676- if total_file_count as i64 != fs_record.file_count.unwrap_or(0) {
7777- println!("Total files after expanding subfs: {}", total_file_count);
7878- }
7979-8080- // Load existing metadata for incremental updates
8181- let existing_metadata = SiteMetadata::load(&output_dir)?;
8282- let existing_file_cids = existing_metadata
8383- .as_ref()
8484- .map(|m| m.file_cids.clone())
8585- .unwrap_or_default();
8686-8787- // Extract blob map from the expanded manifest
8888- let new_blob_map = blob_map::extract_blob_map(&expanded_root);
8989- let new_file_cids: HashMap<String, String> = new_blob_map
9090- .iter()
9191- .map(|(path, (_blob_ref, cid))| (path.clone(), cid.clone()))
9292- .collect();
9393-9494- // Clean up any leftover temp directories from previous failed attempts
9595- let parent = output_dir.parent().unwrap_or_else(|| std::path::Path::new("."));
9696- let output_name = output_dir.file_name().unwrap_or_else(|| std::ffi::OsStr::new("site")).to_string_lossy();
9797- let temp_prefix = format!(".tmp-{}-", output_name);
9898-9999- if let Ok(entries) = parent.read_dir() {
100100- for entry in entries.flatten() {
101101- let name = entry.file_name();
102102- if name.to_string_lossy().starts_with(&temp_prefix) {
103103- let _ = std::fs::remove_dir_all(entry.path());
104104- }
105105- }
106106- }
107107-108108- // Check if we need to update (verify files actually exist, not just metadata)
109109- if let Some(metadata) = &existing_metadata {
110110- if metadata.record_cid == record_cid {
111111- // Verify that the output directory actually exists and has the expected files
112112- let has_all_files = output_dir.exists() && {
113113- // Count actual files on disk (excluding metadata)
114114- let mut actual_file_count = 0;
115115- if let Ok(entries) = std::fs::read_dir(&output_dir) {
116116- for entry in entries.flatten() {
117117- let name = entry.file_name();
118118- if !name.to_string_lossy().starts_with(".wisp-metadata") {
119119- if entry.path().is_file() {
120120- actual_file_count += 1;
121121- }
122122- }
123123- }
124124- }
125125-126126- // Compare with expected file count from metadata
127127- let expected_count = metadata.file_cids.len();
128128- actual_file_count > 0 && actual_file_count >= expected_count
129129- };
130130-131131- if has_all_files {
132132- println!("Site is already up to date!");
133133- return Ok(());
134134- } else {
135135- println!("Site metadata exists but files are missing, re-downloading...");
136136- }
137137- }
138138- }
139139-140140- // Create temporary directory for atomic update
141141- // Place temp dir in parent directory to avoid issues with non-existent output_dir
142142- let parent = output_dir.parent().unwrap_or_else(|| std::path::Path::new("."));
143143- let temp_dir_name = format!(
144144- ".tmp-{}-{}",
145145- output_dir.file_name().unwrap_or_else(|| std::ffi::OsStr::new("site")).to_string_lossy(),
146146- chrono::Utc::now().timestamp()
147147- );
148148- let temp_dir = parent.join(temp_dir_name);
149149- std::fs::create_dir_all(&temp_dir).into_diagnostic()?;
150150-151151- println!("Downloading files...");
152152- let mut downloaded = 0;
153153- let mut reused = 0;
154154-155155- // Download files recursively (using expanded root)
156156- let download_result = download_directory(
157157- &expanded_root,
158158- &temp_dir,
159159- &pds_url,
160160- did.as_str(),
161161- &new_blob_map,
162162- &existing_file_cids,
163163- &output_dir,
164164- String::new(),
165165- &mut downloaded,
166166- &mut reused,
167167- )
168168- .await;
169169-170170- // If download failed, clean up temp directory
171171- if let Err(e) = download_result {
172172- let _ = std::fs::remove_dir_all(&temp_dir);
173173- return Err(e);
174174- }
175175-176176- println!(
177177- "Downloaded {} files, reused {} files",
178178- downloaded, reused
179179- );
180180-181181- // Save metadata
182182- let metadata = SiteMetadata::new(record_cid, new_file_cids);
183183- metadata.save(&temp_dir)?;
184184-185185- // Move files from temp to output directory
186186- let output_abs = std::fs::canonicalize(&output_dir).unwrap_or_else(|_| output_dir.clone());
187187- let current_dir = std::env::current_dir().into_diagnostic()?;
188188-189189- // Special handling for pulling to current directory
190190- if output_abs == current_dir {
191191- // Move files from temp to current directory
192192- for entry in std::fs::read_dir(&temp_dir).into_diagnostic()? {
193193- let entry = entry.into_diagnostic()?;
194194- let dest = current_dir.join(entry.file_name());
195195-196196- // Remove existing file/dir if it exists
197197- if dest.exists() {
198198- if dest.is_dir() {
199199- std::fs::remove_dir_all(&dest).into_diagnostic()?;
200200- } else {
201201- std::fs::remove_file(&dest).into_diagnostic()?;
202202- }
203203- }
204204-205205- // Move from temp to current dir
206206- std::fs::rename(entry.path(), dest).into_diagnostic()?;
207207- }
208208-209209- // Clean up temp directory
210210- std::fs::remove_dir_all(&temp_dir).into_diagnostic()?;
211211- } else {
212212- // If output directory exists and has content, remove it first
213213- if output_dir.exists() {
214214- std::fs::remove_dir_all(&output_dir).into_diagnostic()?;
215215- }
216216-217217- // Ensure parent directory exists
218218- if let Some(parent) = output_dir.parent() {
219219- if !parent.as_os_str().is_empty() && !parent.exists() {
220220- std::fs::create_dir_all(parent).into_diagnostic()?;
221221- }
222222- }
223223-224224- // Rename temp to final location
225225- match std::fs::rename(&temp_dir, &output_dir) {
226226- Ok(_) => {},
227227- Err(e) => {
228228- // Clean up temp directory on failure
229229- let _ = std::fs::remove_dir_all(&temp_dir);
230230- return Err(miette::miette!("Failed to move temp directory: {}", e));
231231- }
232232- }
233233- }
234234-235235- println!("✓ Site pulled successfully to {}", output_dir.display());
236236-237237- Ok(())
238238-}
239239-240240-/// Recursively download a directory with concurrent downloads
241241-fn download_directory<'a>(
242242- dir: &'a Directory<'_>,
243243- output_dir: &'a Path,
244244- pds_url: &'a Url,
245245- did: &'a str,
246246- new_blob_map: &'a HashMap<String, (jacquard_common::types::blob::BlobRef<'static>, String)>,
247247- existing_file_cids: &'a HashMap<String, String>,
248248- existing_output_dir: &'a Path,
249249- path_prefix: String,
250250- downloaded: &'a mut usize,
251251- reused: &'a mut usize,
252252-) -> std::pin::Pin<Box<dyn std::future::Future<Output = miette::Result<()>> + Send + 'a>> {
253253- Box::pin(async move {
254254- use futures::stream::{self, StreamExt};
255255-256256- // Collect download tasks and directory tasks separately
257257- struct DownloadTask {
258258- path: String,
259259- output_path: PathBuf,
260260- blob: jacquard_common::types::blob::BlobRef<'static>,
261261- base64: bool,
262262- gzip: bool,
263263- }
264264-265265- struct CopyTask {
266266- path: String,
267267- from: PathBuf,
268268- to: PathBuf,
269269- }
270270-271271- let mut download_tasks = Vec::new();
272272- let mut copy_tasks = Vec::new();
273273- let mut dir_tasks = Vec::new();
274274-275275- for entry in &dir.entries {
276276- let entry_name = entry.name.as_str();
277277- let current_path = if path_prefix.is_empty() {
278278- entry_name.to_string()
279279- } else {
280280- format!("{}/{}", path_prefix, entry_name)
281281- };
282282-283283- match &entry.node {
284284- EntryNode::File(file) => {
285285- let output_path = output_dir.join(entry_name);
286286-287287- // Check if file CID matches existing
288288- let should_copy = if let Some((_blob_ref, new_cid)) = new_blob_map.get(¤t_path) {
289289- if let Some(existing_cid) = existing_file_cids.get(¤t_path) {
290290- if existing_cid == new_cid {
291291- let existing_path = existing_output_dir.join(¤t_path);
292292- if existing_path.exists() {
293293- copy_tasks.push(CopyTask {
294294- path: current_path.clone(),
295295- from: existing_path,
296296- to: output_path.clone(),
297297- });
298298- true
299299- } else {
300300- false
301301- }
302302- } else {
303303- false
304304- }
305305- } else {
306306- false
307307- }
308308- } else {
309309- false
310310- };
311311-312312- if !should_copy {
313313- use jacquard_common::IntoStatic;
314314- // File needs to be downloaded
315315- download_tasks.push(DownloadTask {
316316- path: current_path,
317317- output_path,
318318- blob: file.blob.clone().into_static(),
319319- base64: file.base64.unwrap_or(false),
320320- gzip: file.encoding.as_ref().map(|e| e.as_str() == "gzip").unwrap_or(false),
321321- });
322322- }
323323- }
324324- EntryNode::Directory(subdir) => {
325325- let subdir_path = output_dir.join(entry_name);
326326- dir_tasks.push((subdir.as_ref().clone(), subdir_path, current_path));
327327- }
328328- EntryNode::Subfs(_) => {
329329- println!(" ⚠ Skipping subfs node at {} (should have been expanded)", current_path);
330330- }
331331- EntryNode::Unknown(_) => {
332332- println!(" ⚠ Skipping unknown node type for {}", current_path);
333333- }
334334- }
335335- }
336336-337337- // Execute copy tasks (fast, do them all)
338338- for task in copy_tasks {
339339- std::fs::copy(&task.from, &task.to).into_diagnostic()?;
340340- *reused += 1;
341341- println!(" ✓ Reused {}", task.path);
342342- }
343343-344344- // Execute download tasks with concurrency limit (20 concurrent downloads)
345345- const DOWNLOAD_CONCURRENCY: usize = 20;
346346-347347- let pds_url_clone = pds_url.clone();
348348- let did_str = did.to_string();
349349-350350- let download_results: Vec<miette::Result<(String, PathBuf, Vec<u8>)>> = stream::iter(download_tasks)
351351- .map(|task| {
352352- let pds = pds_url_clone.clone();
353353- let did_copy = did_str.clone();
354354-355355- async move {
356356- println!(" ↓ Downloading {}", task.path);
357357- let data = download::download_and_decompress_blob(
358358- &pds,
359359- &task.blob,
360360- &did_copy,
361361- task.base64,
362362- task.gzip,
363363- )
364364- .await?;
365365-366366- Ok::<_, miette::Report>((task.path, task.output_path, data))
367367- }
368368- })
369369- .buffer_unordered(DOWNLOAD_CONCURRENCY)
370370- .collect()
371371- .await;
372372-373373- // Write downloaded files to disk
374374- for result in download_results {
375375- let (path, output_path, data) = result?;
376376- std::fs::write(&output_path, data).into_diagnostic()?;
377377- *downloaded += 1;
378378- println!(" ✓ Downloaded {}", path);
379379- }
380380-381381- // Recursively process directories
382382- for (subdir, subdir_path, current_path) in dir_tasks {
383383- std::fs::create_dir_all(&subdir_path).into_diagnostic()?;
384384-385385- download_directory(
386386- &subdir,
387387- &subdir_path,
388388- pds_url,
389389- did,
390390- new_blob_map,
391391- existing_file_cids,
392392- existing_output_dir,
393393- current_path,
394394- downloaded,
395395- reused,
396396- )
397397- .await?;
398398- }
399399-400400- Ok(())
401401- })
402402-}
403403-404404-/// Expand subfs nodes in a directory tree by fetching and merging subfs records (RECURSIVELY)
405405-/// Uses reqwest client directly for pull command (no agent needed)
406406-async fn expand_subfs_in_pull_with_client<'a>(
407407- directory: &Directory<'a>,
408408- client: &reqwest::Client,
409409- pds_url: &Url,
410410-) -> miette::Result<Directory<'static>> {
411411- use jacquard_common::IntoStatic;
412412- use jacquard_common::types::value::from_data;
413413- use wisp_lexicons::place_wisp::subfs::SubfsRecord;
414414-415415- let mut all_subfs_map: HashMap<String, wisp_lexicons::place_wisp::subfs::Directory> = HashMap::new();
416416- let mut to_fetch = subfs_utils::extract_subfs_uris(directory, String::new());
417417-418418- if to_fetch.is_empty() {
419419- return Ok((*directory).clone().into_static());
420420- }
421421-422422- println!("Found {} subfs records, fetching recursively...", to_fetch.len());
423423-424424- let mut iteration = 0;
425425- const MAX_ITERATIONS: usize = 10;
426426-427427- while !to_fetch.is_empty() && iteration < MAX_ITERATIONS {
428428- iteration += 1;
429429- println!(" Iteration {}: fetching {} subfs records...", iteration, to_fetch.len());
430430-431431- let mut fetch_tasks = Vec::new();
432432-433433- for (uri, path) in to_fetch.clone() {
434434- let client = client.clone();
435435- let pds_url = pds_url.clone();
436436-437437- fetch_tasks.push(async move {
438438- // Parse URI
439439- let parts: Vec<&str> = uri.trim_start_matches("at://").split('/').collect();
440440- if parts.len() < 3 {
441441- return Err(miette::miette!("Invalid subfs URI: {}", uri));
442442- }
443443-444444- let did_str = parts[0];
445445- let collection = parts[1];
446446- let rkey_str = parts[2];
447447-448448- if collection != "place.wisp.subfs" {
449449- return Err(miette::miette!("Expected place.wisp.subfs collection, got: {}", collection));
450450- }
451451-452452- // Fetch using GetRecord
453453- use jacquard::api::com_atproto::repo::get_record::GetRecord;
454454- use jacquard_common::types::string::{Rkey as RkeyType, Did as DidType, RecordKey};
455455- use jacquard_common::types::ident::AtIdentifier;
456456-457457- let rkey_parsed = RkeyType::new(rkey_str).into_diagnostic()?;
458458- let did_parsed = DidType::new(did_str).into_diagnostic()?;
459459-460460- let request = GetRecord::new()
461461- .repo(AtIdentifier::Did(did_parsed))
462462- .collection(CowStr::from("place.wisp.subfs"))
463463- .rkey(RecordKey::from(rkey_parsed))
464464- .build();
465465-466466- let response = client
467467- .xrpc(pds_url)
468468- .send(&request)
469469- .await
470470- .into_diagnostic()?;
471471-472472- let record_output = response.into_output().into_diagnostic()?;
473473- let subfs_record: SubfsRecord = from_data(&record_output.value).into_diagnostic()?;
474474-475475- Ok::<_, miette::Report>((path, subfs_record.into_static()))
476476- });
477477- }
478478-479479- let results: Vec<_> = futures::future::join_all(fetch_tasks).await;
480480-481481- // Process results and find nested subfs
482482- let mut newly_found_uris = Vec::new();
483483- for result in results {
484484- match result {
485485- Ok((path, record)) => {
486486- println!(" ✓ Fetched subfs at {}", path);
487487-488488- // Extract nested subfs URIs
489489- let nested_uris = extract_subfs_uris_from_subfs_dir(&record.root, path.clone());
490490- newly_found_uris.extend(nested_uris);
491491-492492- all_subfs_map.insert(path, record.root);
493493- }
494494- Err(e) => {
495495- eprintln!(" ⚠️ Failed to fetch subfs: {}", e);
496496- }
497497- }
498498- }
499499-500500- // Filter out already-fetched paths
501501- to_fetch = newly_found_uris
502502- .into_iter()
503503- .filter(|(_, path)| !all_subfs_map.contains_key(path))
504504- .collect();
505505- }
506506-507507- if iteration >= MAX_ITERATIONS {
508508- eprintln!("⚠️ Max iterations reached while fetching nested subfs");
509509- }
510510-511511- println!(" Total subfs records fetched: {}", all_subfs_map.len());
512512-513513- // Now replace all subfs nodes with their content
514514- Ok(replace_subfs_with_content(directory.clone(), &all_subfs_map, String::new()))
515515-}
516516-517517-/// Extract subfs URIs from a subfs::Directory (helper for pull)
518518-fn extract_subfs_uris_from_subfs_dir(
519519- directory: &wisp_lexicons::place_wisp::subfs::Directory,
520520- current_path: String,
521521-) -> Vec<(String, String)> {
522522- let mut uris = Vec::new();
523523-524524- for entry in &directory.entries {
525525- let full_path = if current_path.is_empty() {
526526- entry.name.to_string()
527527- } else {
528528- format!("{}/{}", current_path, entry.name)
529529- };
530530-531531- match &entry.node {
532532- wisp_lexicons::place_wisp::subfs::EntryNode::Subfs(subfs_node) => {
533533- uris.push((subfs_node.subject.to_string(), full_path.clone()));
534534- }
535535- wisp_lexicons::place_wisp::subfs::EntryNode::Directory(subdir) => {
536536- let nested = extract_subfs_uris_from_subfs_dir(subdir, full_path);
537537- uris.extend(nested);
538538- }
539539- _ => {}
540540- }
541541- }
542542-543543- uris
544544-}
545545-546546-/// Recursively replace subfs nodes with their actual content
547547-fn replace_subfs_with_content(
548548- directory: Directory,
549549- subfs_map: &HashMap<String, wisp_lexicons::place_wisp::subfs::Directory>,
550550- current_path: String,
551551-) -> Directory<'static> {
552552- use jacquard_common::IntoStatic;
553553-554554- let new_entries: Vec<Entry<'static>> = directory
555555- .entries
556556- .into_iter()
557557- .flat_map(|entry| {
558558- let full_path = if current_path.is_empty() {
559559- entry.name.to_string()
560560- } else {
561561- format!("{}/{}", current_path, entry.name)
562562- };
563563-564564- match entry.node {
565565- EntryNode::Subfs(subfs_node) => {
566566- // Check if we have this subfs record
567567- if let Some(subfs_dir) = subfs_map.get(&full_path) {
568568- let flat = subfs_node.flat.unwrap_or(true); // Default to flat merge
569569-570570- if flat {
571571- // Flat merge: hoist subfs entries into parent
572572- println!(" Merging subfs {} (flat)", full_path);
573573- let converted_entries: Vec<Entry<'static>> = subfs_dir
574574- .entries
575575- .iter()
576576- .map(|subfs_entry| convert_subfs_entry_to_fs(subfs_entry.clone().into_static()))
577577- .collect();
578578-579579- converted_entries
580580- } else {
581581- // Nested: create a directory with the subfs name
582582- println!(" Merging subfs {} (nested)", full_path);
583583- let converted_entries: Vec<Entry<'static>> = subfs_dir
584584- .entries
585585- .iter()
586586- .map(|subfs_entry| convert_subfs_entry_to_fs(subfs_entry.clone().into_static()))
587587- .collect();
588588-589589- vec![Entry::new()
590590- .name(entry.name.into_static())
591591- .node(EntryNode::Directory(Box::new(
592592- Directory::new()
593593- .r#type(CowStr::from("directory"))
594594- .entries(converted_entries)
595595- .build()
596596- )))
597597- .build()]
598598- }
599599- } else {
600600- // Subfs not found, skip with warning
601601- eprintln!(" ⚠️ Subfs not found: {}", full_path);
602602- vec![]
603603- }
604604- }
605605- EntryNode::Directory(dir) => {
606606- // Recursively process subdirectories
607607- vec![Entry::new()
608608- .name(entry.name.into_static())
609609- .node(EntryNode::Directory(Box::new(
610610- replace_subfs_with_content(*dir, subfs_map, full_path)
611611- )))
612612- .build()]
613613- }
614614- EntryNode::File(_) => {
615615- vec![entry.into_static()]
616616- }
617617- EntryNode::Unknown(_) => {
618618- vec![entry.into_static()]
619619- }
620620- }
621621- })
622622- .collect();
623623-624624- Directory::new()
625625- .r#type(CowStr::from("directory"))
626626- .entries(new_entries)
627627- .build()
628628-}
629629-630630-/// Convert a subfs entry to a fs entry (they have the same structure but different types)
631631-fn convert_subfs_entry_to_fs(subfs_entry: wisp_lexicons::place_wisp::subfs::Entry<'static>) -> Entry<'static> {
632632- use jacquard_common::IntoStatic;
633633-634634- let node = match subfs_entry.node {
635635- wisp_lexicons::place_wisp::subfs::EntryNode::File(file) => {
636636- EntryNode::File(Box::new(
637637- File::new()
638638- .r#type(file.r#type.into_static())
639639- .blob(file.blob.into_static())
640640- .encoding(file.encoding.map(|e| e.into_static()))
641641- .mime_type(file.mime_type.map(|m| m.into_static()))
642642- .base64(file.base64)
643643- .build()
644644- ))
645645- }
646646- wisp_lexicons::place_wisp::subfs::EntryNode::Directory(dir) => {
647647- let converted_entries: Vec<Entry<'static>> = dir
648648- .entries
649649- .into_iter()
650650- .map(|e| convert_subfs_entry_to_fs(e.into_static()))
651651- .collect();
652652-653653- EntryNode::Directory(Box::new(
654654- Directory::new()
655655- .r#type(dir.r#type.into_static())
656656- .entries(converted_entries)
657657- .build()
658658- ))
659659- }
660660- wisp_lexicons::place_wisp::subfs::EntryNode::Subfs(_nested_subfs) => {
661661- // Nested subfs should have been expanded already - if we get here, it means expansion failed
662662- // Treat it like a directory reference that should have been expanded
663663- eprintln!(" ⚠️ Warning: unexpanded nested subfs at path, treating as empty directory");
664664- EntryNode::Directory(Box::new(
665665- Directory::new()
666666- .r#type(CowStr::from("directory"))
667667- .entries(vec![])
668668- .build()
669669- ))
670670- }
671671- wisp_lexicons::place_wisp::subfs::EntryNode::Unknown(unknown) => {
672672- EntryNode::Unknown(unknown)
673673- }
674674- };
675675-676676- Entry::new()
677677- .name(subfs_entry.name.into_static())
678678- .node(node)
679679- .build()
680680-}
681681-
-375
rust-cli/src/redirects.rs
···11-use regex::Regex;
22-use std::collections::HashMap;
33-use std::fs;
44-use std::path::Path;
55-66-/// Maximum number of redirect rules to prevent DoS attacks
77-const MAX_REDIRECT_RULES: usize = 1000;
88-99-#[derive(Debug, Clone)]
1010-pub struct RedirectRule {
1111- #[allow(dead_code)]
1212- pub from: String,
1313- pub to: String,
1414- pub status: u16,
1515- #[allow(dead_code)]
1616- pub force: bool,
1717- pub from_pattern: Regex,
1818- pub from_params: Vec<String>,
1919- pub query_params: Option<HashMap<String, String>>,
2020-}
2121-2222-#[derive(Debug)]
2323-pub struct RedirectMatch {
2424- pub target_path: String,
2525- pub status: u16,
2626- pub force: bool,
2727-}
2828-2929-/// Parse a _redirects file into an array of redirect rules
3030-pub fn parse_redirects_file(content: &str) -> Vec<RedirectRule> {
3131- let lines = content.lines();
3232- let mut rules = Vec::new();
3333-3434- for (line_num, line_raw) in lines.enumerate() {
3535- if line_raw.trim().is_empty() || line_raw.trim().starts_with('#') {
3636- continue;
3737- }
3838-3939- // Enforce max rules limit
4040- if rules.len() >= MAX_REDIRECT_RULES {
4141- eprintln!(
4242- "Redirect rules limit reached ({}), ignoring remaining rules",
4343- MAX_REDIRECT_RULES
4444- );
4545- break;
4646- }
4747-4848- match parse_redirect_line(line_raw.trim()) {
4949- Ok(Some(rule)) => rules.push(rule),
5050- Ok(None) => continue,
5151- Err(e) => {
5252- eprintln!(
5353- "Failed to parse redirect rule on line {}: {} ({})",
5454- line_num + 1,
5555- line_raw,
5656- e
5757- );
5858- }
5959- }
6060- }
6161-6262- rules
6363-}
6464-6565-/// Parse a single redirect rule line
6666-/// Format: /from [query_params] /to [status] [conditions]
6767-fn parse_redirect_line(line: &str) -> Result<Option<RedirectRule>, String> {
6868- let parts: Vec<&str> = line.split_whitespace().collect();
6969-7070- if parts.len() < 2 {
7171- return Ok(None);
7272- }
7373-7474- let mut idx = 0;
7575- let from = parts[idx];
7676- idx += 1;
7777-7878- let mut status = 301; // Default status
7979- let mut force = false;
8080- let mut query_params: HashMap<String, String> = HashMap::new();
8181-8282- // Parse query parameters that come before the destination path
8383- while idx < parts.len() {
8484- let part = parts[idx];
8585-8686- // If it starts with / or http, it's the destination path
8787- if part.starts_with('/') || part.starts_with("http://") || part.starts_with("https://") {
8888- break;
8989- }
9090-9191- // If it contains = and comes before the destination, it's a query param
9292- if part.contains('=') {
9393- let split_index = part.find('=').unwrap();
9494- let key = &part[..split_index];
9595- let value = &part[split_index + 1..];
9696-9797- if !key.is_empty() && !value.is_empty() {
9898- query_params.insert(key.to_string(), value.to_string());
9999- }
100100- idx += 1;
101101- } else {
102102- break;
103103- }
104104- }
105105-106106- // Next part should be the destination
107107- if idx >= parts.len() {
108108- return Ok(None);
109109- }
110110-111111- let to = parts[idx];
112112- idx += 1;
113113-114114- // Parse remaining parts for status code
115115- for part in parts.iter().skip(idx) {
116116- // Check for status code (with optional ! for force)
117117- if let Some(stripped) = part.strip_suffix('!') {
118118- if let Ok(s) = stripped.parse::<u16>() {
119119- force = true;
120120- status = s;
121121- }
122122- } else if let Ok(s) = part.parse::<u16>() {
123123- status = s;
124124- }
125125- // Note: We're ignoring conditional redirects (Country, Language, Cookie, Role) for now
126126- // They can be added later if needed
127127- }
128128-129129- // Parse the 'from' pattern
130130- let (pattern, params) = convert_path_to_regex(from)?;
131131-132132- Ok(Some(RedirectRule {
133133- from: from.to_string(),
134134- to: to.to_string(),
135135- status,
136136- force,
137137- from_pattern: pattern,
138138- from_params: params,
139139- query_params: if query_params.is_empty() {
140140- None
141141- } else {
142142- Some(query_params)
143143- },
144144- }))
145145-}
146146-147147-/// Convert a path pattern with placeholders and splats to a regex
148148-/// Examples:
149149-/// /blog/:year/:month/:day -> captures year, month, day
150150-/// /news/* -> captures splat
151151-fn convert_path_to_regex(pattern: &str) -> Result<(Regex, Vec<String>), String> {
152152- let mut params = Vec::new();
153153- let mut regex_str = String::from("^");
154154-155155- // Split by query string if present
156156- let path_part = pattern.split('?').next().unwrap_or(pattern);
157157-158158- // Escape special regex characters except * and :
159159- let mut escaped = String::new();
160160- for ch in path_part.chars() {
161161- match ch {
162162- '.' | '+' | '^' | '$' | '{' | '}' | '(' | ')' | '|' | '[' | ']' | '\\' => {
163163- escaped.push('\\');
164164- escaped.push(ch);
165165- }
166166- _ => escaped.push(ch),
167167- }
168168- }
169169-170170- // Replace :param with named capture groups
171171- let param_regex = Regex::new(r":([a-zA-Z_][a-zA-Z0-9_]*)").map_err(|e| e.to_string())?;
172172- let mut last_end = 0;
173173- let mut result = String::new();
174174-175175- for cap in param_regex.captures_iter(&escaped) {
176176- let m = cap.get(0).unwrap();
177177- result.push_str(&escaped[last_end..m.start()]);
178178- result.push_str("([^/?]+)");
179179- params.push(cap[1].to_string());
180180- last_end = m.end();
181181- }
182182- result.push_str(&escaped[last_end..]);
183183- escaped = result;
184184-185185- // Replace * with splat capture
186186- if escaped.contains('*') {
187187- escaped = escaped.replace('*', "(.*)");
188188- params.push("splat".to_string());
189189- }
190190-191191- regex_str.push_str(&escaped);
192192-193193- // Make trailing slash optional
194194- if !regex_str.ends_with(".*") {
195195- regex_str.push_str("/?");
196196- }
197197-198198- regex_str.push('$');
199199-200200- let pattern = Regex::new(®ex_str).map_err(|e| e.to_string())?;
201201-202202- Ok((pattern, params))
203203-}
204204-205205-/// Match a request path against redirect rules
206206-pub fn match_redirect_rule(
207207- request_path: &str,
208208- rules: &[RedirectRule],
209209- query_params: Option<&HashMap<String, String>>,
210210-) -> Option<RedirectMatch> {
211211- // Normalize path: ensure leading slash
212212- let normalized_path = if request_path.starts_with('/') {
213213- request_path.to_string()
214214- } else {
215215- format!("/{}", request_path)
216216- };
217217-218218- for rule in rules {
219219- // Check query parameter conditions first (if any)
220220- if let Some(required_params) = &rule.query_params {
221221- if let Some(actual_params) = query_params {
222222- let query_matches = required_params.iter().all(|(key, expected_value)| {
223223- if let Some(actual_value) = actual_params.get(key) {
224224- // If expected value is a placeholder (:name), any value is acceptable
225225- if expected_value.starts_with(':') {
226226- return true;
227227- }
228228- // Otherwise it must match exactly
229229- actual_value == expected_value
230230- } else {
231231- false
232232- }
233233- });
234234-235235- if !query_matches {
236236- continue;
237237- }
238238- } else {
239239- // Rule requires query params but none provided
240240- continue;
241241- }
242242- }
243243-244244- // Match the path pattern
245245- if let Some(captures) = rule.from_pattern.captures(&normalized_path) {
246246- let mut target_path = rule.to.clone();
247247-248248- // Replace captured parameters
249249- for (i, param_name) in rule.from_params.iter().enumerate() {
250250- if let Some(param_value) = captures.get(i + 1) {
251251- let value = param_value.as_str();
252252-253253- if param_name == "splat" {
254254- target_path = target_path.replace(":splat", value);
255255- } else {
256256- target_path = target_path.replace(&format!(":{}", param_name), value);
257257- }
258258- }
259259- }
260260-261261- // Handle query parameter replacements
262262- if let Some(required_params) = &rule.query_params {
263263- if let Some(actual_params) = query_params {
264264- for (key, placeholder) in required_params {
265265- if placeholder.starts_with(':') {
266266- if let Some(actual_value) = actual_params.get(key) {
267267- let param_name = &placeholder[1..];
268268- target_path = target_path.replace(
269269- &format!(":{}", param_name),
270270- actual_value,
271271- );
272272- }
273273- }
274274- }
275275- }
276276- }
277277-278278- // Preserve query string for 200, 301, 302 redirects (unless target already has one)
279279- if [200, 301, 302].contains(&rule.status)
280280- && query_params.is_some()
281281- && !target_path.contains('?')
282282- {
283283- if let Some(params) = query_params {
284284- if !params.is_empty() {
285285- let query_string: String = params
286286- .iter()
287287- .map(|(k, v)| format!("{}={}", k, v))
288288- .collect::<Vec<_>>()
289289- .join("&");
290290- target_path = format!("{}?{}", target_path, query_string);
291291- }
292292- }
293293- }
294294-295295- return Some(RedirectMatch {
296296- target_path,
297297- status: rule.status,
298298- force: rule.force,
299299- });
300300- }
301301- }
302302-303303- None
304304-}
305305-306306-/// Load redirect rules from a _redirects file
307307-pub fn load_redirect_rules(directory: &Path) -> Vec<RedirectRule> {
308308- let redirects_path = directory.join("_redirects");
309309-310310- if !redirects_path.exists() {
311311- return Vec::new();
312312- }
313313-314314- match fs::read_to_string(&redirects_path) {
315315- Ok(content) => parse_redirects_file(&content),
316316- Err(e) => {
317317- eprintln!("Failed to load _redirects file: {}", e);
318318- Vec::new()
319319- }
320320- }
321321-}
322322-323323-#[cfg(test)]
324324-mod tests {
325325- use super::*;
326326-327327- #[test]
328328- fn test_parse_simple_redirect() {
329329- let content = "/old-path /new-path";
330330- let rules = parse_redirects_file(content);
331331- assert_eq!(rules.len(), 1);
332332- assert_eq!(rules[0].from, "/old-path");
333333- assert_eq!(rules[0].to, "/new-path");
334334- assert_eq!(rules[0].status, 301);
335335- assert!(!rules[0].force);
336336- }
337337-338338- #[test]
339339- fn test_parse_with_status() {
340340- let content = "/temp /target 302";
341341- let rules = parse_redirects_file(content);
342342- assert_eq!(rules[0].status, 302);
343343- }
344344-345345- #[test]
346346- fn test_parse_force_redirect() {
347347- let content = "/force /target 301!";
348348- let rules = parse_redirects_file(content);
349349- assert!(rules[0].force);
350350- }
351351-352352- #[test]
353353- fn test_match_exact_path() {
354354- let rules = parse_redirects_file("/old-path /new-path");
355355- let m = match_redirect_rule("/old-path", &rules, None);
356356- assert!(m.is_some());
357357- assert_eq!(m.unwrap().target_path, "/new-path");
358358- }
359359-360360- #[test]
361361- fn test_match_splat() {
362362- let rules = parse_redirects_file("/news/* /blog/:splat");
363363- let m = match_redirect_rule("/news/2024/01/15/post", &rules, None);
364364- assert!(m.is_some());
365365- assert_eq!(m.unwrap().target_path, "/blog/2024/01/15/post");
366366- }
367367-368368- #[test]
369369- fn test_match_placeholders() {
370370- let rules = parse_redirects_file("/blog/:year/:month/:day /posts/:year-:month-:day");
371371- let m = match_redirect_rule("/blog/2024/01/15", &rules, None);
372372- assert!(m.is_some());
373373- assert_eq!(m.unwrap().target_path, "/posts/2024-01-15");
374374- }
375375-}
-623
rust-cli/src/serve.rs
···11-use crate::pull::pull_site;
22-use crate::redirects::{load_redirect_rules, match_redirect_rule, RedirectRule};
33-use wisp_lexicons::place_wisp::settings::Settings;
44-use axum::{
55- Router,
66- extract::Request,
77- response::{Response, IntoResponse, Redirect},
88- http::{StatusCode, Uri, header},
99- body::Body,
1010-};
1111-use jacquard::CowStr;
1212-use jacquard::api::com_atproto::sync::subscribe_repos::{SubscribeRepos, SubscribeReposMessage};
1313-use jacquard::api::com_atproto::repo::get_record::GetRecord;
1414-use jacquard_common::types::string::Did;
1515-use jacquard_common::xrpc::{SubscriptionClient, TungsteniteSubscriptionClient, XrpcExt};
1616-use jacquard_common::IntoStatic;
1717-use jacquard_common::types::value::from_data;
1818-use miette::IntoDiagnostic;
1919-use n0_future::StreamExt;
2020-use std::collections::HashMap;
2121-use std::path::{PathBuf, Path};
2222-use std::sync::Arc;
2323-use tokio::sync::RwLock;
2424-use tower::Service;
2525-use tower_http::compression::CompressionLayer;
2626-use tower_http::services::ServeDir;
2727-2828-/// Shared state for the server
2929-#[derive(Clone)]
3030-struct ServerState {
3131- did: CowStr<'static>,
3232- rkey: CowStr<'static>,
3333- output_dir: PathBuf,
3434- last_cid: Arc<RwLock<Option<String>>>,
3535- redirect_rules: Arc<RwLock<Vec<RedirectRule>>>,
3636- settings: Arc<RwLock<Option<Settings<'static>>>>,
3737-}
3838-3939-/// Fetch settings for a site from the PDS
4040-async fn fetch_settings(
4141- pds_url: &url::Url,
4242- did: &Did<'_>,
4343- rkey: &str,
4444-) -> miette::Result<Option<Settings<'static>>> {
4545- use jacquard_common::types::ident::AtIdentifier;
4646- use jacquard_common::types::string::{Rkey as RkeyType, RecordKey};
4747-4848- let client = reqwest::Client::new();
4949- let rkey_parsed = RkeyType::new(rkey).into_diagnostic()?;
5050-5151- let request = GetRecord::new()
5252- .repo(AtIdentifier::Did(did.clone()))
5353- .collection(CowStr::from("place.wisp.settings"))
5454- .rkey(RecordKey::from(rkey_parsed))
5555- .build();
5656-5757- match client.xrpc(pds_url.clone()).send(&request).await {
5858- Ok(response) => {
5959- let output = response.into_output().into_diagnostic()?;
6060-6161- // Parse the record value as Settings
6262- match from_data::<Settings>(&output.value) {
6363- Ok(settings) => {
6464- Ok(Some(settings.into_static()))
6565- }
6666- Err(_) => {
6767- // Settings record exists but couldn't parse - use defaults
6868- Ok(None)
6969- }
7070- }
7171- }
7272- Err(_) => {
7373- // Settings record doesn't exist
7474- Ok(None)
7575- }
7676- }
7777-}
7878-7979-/// Serve a site locally with real-time firehose updates
8080-pub async fn serve_site(
8181- input: CowStr<'static>,
8282- rkey: CowStr<'static>,
8383- output_dir: PathBuf,
8484- port: u16,
8585-) -> miette::Result<()> {
8686- println!("Serving site {} from {} on port {}...", rkey, input, port);
8787-8888- // Resolve handle to DID if needed
8989- use jacquard_identity::PublicResolver;
9090- use jacquard::prelude::IdentityResolver;
9191-9292- let resolver = PublicResolver::default();
9393- let did = if input.starts_with("did:") {
9494- Did::new(&input).into_diagnostic()?
9595- } else {
9696- // It's a handle, resolve it
9797- let handle = jacquard_common::types::string::Handle::new(&input).into_diagnostic()?;
9898- resolver.resolve_handle(&handle).await.into_diagnostic()?
9999- };
100100-101101- println!("Resolved to DID: {}", did.as_str());
102102-103103- // Resolve PDS URL (needed for settings fetch)
104104- let pds_url = resolver.pds_for_did(&did).await.into_diagnostic()?;
105105-106106- // Create output directory if it doesn't exist
107107- std::fs::create_dir_all(&output_dir).into_diagnostic()?;
108108-109109- // Initial pull of the site
110110- println!("Performing initial pull...");
111111- let did_str = CowStr::from(did.as_str().to_string());
112112- pull_site(did_str.clone(), rkey.clone(), output_dir.clone()).await?;
113113-114114- // Fetch settings
115115- let settings = fetch_settings(&pds_url, &did, rkey.as_ref()).await?;
116116- if let Some(ref s) = settings {
117117- println!("\nSettings loaded:");
118118- if let Some(true) = s.directory_listing {
119119- println!(" • Directory listing: enabled");
120120- }
121121- if let Some(ref spa_file) = s.spa_mode {
122122- println!(" • SPA mode: enabled ({})", spa_file);
123123- }
124124- if let Some(ref custom404) = s.custom404 {
125125- println!(" • Custom 404: {}", custom404);
126126- }
127127- } else {
128128- println!("No settings configured (using defaults)");
129129- }
130130-131131- // Load redirect rules
132132- let redirect_rules = load_redirect_rules(&output_dir);
133133- if !redirect_rules.is_empty() {
134134- println!("Loaded {} redirect rules from _redirects", redirect_rules.len());
135135- }
136136-137137- // Create shared state
138138- let state = ServerState {
139139- did: did_str.clone(),
140140- rkey: rkey.clone(),
141141- output_dir: output_dir.clone(),
142142- last_cid: Arc::new(RwLock::new(None)),
143143- redirect_rules: Arc::new(RwLock::new(redirect_rules)),
144144- settings: Arc::new(RwLock::new(settings)),
145145- };
146146-147147- // Start firehose listener in background
148148- let firehose_state = state.clone();
149149- tokio::spawn(async move {
150150- if let Err(e) = watch_firehose(firehose_state).await {
151151- eprintln!("Firehose error: {}", e);
152152- }
153153- });
154154-155155- // Create HTTP server with gzip compression and redirect handling
156156- let serve_dir = ServeDir::new(&output_dir).precompressed_gzip();
157157-158158- let app = Router::new()
159159- .fallback(move |req: Request| {
160160- let state = state.clone();
161161- let mut serve_dir = serve_dir.clone();
162162- async move {
163163- handle_request_with_redirects(req, state, &mut serve_dir).await
164164- }
165165- })
166166- .layer(CompressionLayer::new());
167167-168168- let addr = format!("0.0.0.0:{}", port);
169169- let listener = tokio::net::TcpListener::bind(&addr)
170170- .await
171171- .into_diagnostic()?;
172172-173173- println!("\n✓ Server running at http://localhost:{}", port);
174174- println!(" Watching for updates on the firehose...\n");
175175-176176- axum::serve(listener, app).await.into_diagnostic()?;
177177-178178- Ok(())
179179-}
180180-181181-/// Serve a file for SPA mode
182182-async fn serve_file_for_spa(output_dir: &Path, spa_file: &str) -> Response {
183183- let file_path = output_dir.join(spa_file.trim_start_matches('/'));
184184-185185- match tokio::fs::read(&file_path).await {
186186- Ok(contents) => {
187187- Response::builder()
188188- .status(StatusCode::OK)
189189- .header(header::CONTENT_TYPE, "text/html; charset=utf-8")
190190- .body(Body::from(contents))
191191- .unwrap()
192192- }
193193- Err(_) => {
194194- StatusCode::NOT_FOUND.into_response()
195195- }
196196- }
197197-}
198198-199199-/// Serve custom 404 page
200200-async fn serve_custom_404(output_dir: &Path, custom404_file: &str) -> Response {
201201- let file_path = output_dir.join(custom404_file.trim_start_matches('/'));
202202-203203- match tokio::fs::read(&file_path).await {
204204- Ok(contents) => {
205205- Response::builder()
206206- .status(StatusCode::NOT_FOUND)
207207- .header(header::CONTENT_TYPE, "text/html; charset=utf-8")
208208- .body(Body::from(contents))
209209- .unwrap()
210210- }
211211- Err(_) => {
212212- StatusCode::NOT_FOUND.into_response()
213213- }
214214- }
215215-}
216216-217217-/// Serve directory listing
218218-async fn serve_directory_listing(dir_path: &Path, url_path: &str) -> Response {
219219- match tokio::fs::read_dir(dir_path).await {
220220- Ok(mut entries) => {
221221- let mut html = String::from("<!DOCTYPE html><html><head><meta charset='utf-8'><title>Directory listing</title>");
222222- html.push_str("<style>body{font-family:sans-serif;margin:2em}a{display:block;padding:0.5em;text-decoration:none;color:#0066cc}a:hover{background:#f0f0f0}</style>");
223223- html.push_str("</head><body>");
224224- html.push_str(&format!("<h1>Index of {}</h1>", url_path));
225225- html.push_str("<hr>");
226226-227227- // Add parent directory link if not at root
228228- if url_path != "/" {
229229- let parent = if url_path.ends_with('/') {
230230- format!("{}../", url_path)
231231- } else {
232232- format!("{}/", url_path.rsplitn(2, '/').nth(1).unwrap_or("/"))
233233- };
234234- html.push_str(&format!("<a href='{}'>../</a>", parent));
235235- }
236236-237237- let mut items = Vec::new();
238238- while let Ok(Some(entry)) = entries.next_entry().await {
239239- if let Ok(name) = entry.file_name().into_string() {
240240- let is_dir = entry.path().is_dir();
241241- let display_name = if is_dir {
242242- format!("{}/", name)
243243- } else {
244244- name.clone()
245245- };
246246-247247- let link_path = if url_path.ends_with('/') {
248248- format!("{}{}", url_path, name)
249249- } else {
250250- format!("{}/{}", url_path, name)
251251- };
252252-253253- items.push((display_name, link_path, is_dir));
254254- }
255255- }
256256-257257- // Sort: directories first, then alphabetically
258258- items.sort_by(|a, b| {
259259- match (a.2, b.2) {
260260- (true, false) => std::cmp::Ordering::Less,
261261- (false, true) => std::cmp::Ordering::Greater,
262262- _ => a.0.cmp(&b.0),
263263- }
264264- });
265265-266266- for (display_name, link_path, _) in items {
267267- html.push_str(&format!("<a href='{}'>{}</a>", link_path, display_name));
268268- }
269269-270270- html.push_str("</body></html>");
271271-272272- Response::builder()
273273- .status(StatusCode::OK)
274274- .header(header::CONTENT_TYPE, "text/html; charset=utf-8")
275275- .body(Body::from(html))
276276- .unwrap()
277277- }
278278- Err(_) => {
279279- StatusCode::NOT_FOUND.into_response()
280280- }
281281- }
282282-}
283283-284284-/// Handle a request with redirect and settings support
285285-async fn handle_request_with_redirects(
286286- req: Request,
287287- state: ServerState,
288288- serve_dir: &mut ServeDir,
289289-) -> Response {
290290- let uri = req.uri().clone();
291291- let path = uri.path();
292292- let method = req.method().clone();
293293-294294- // Parse query parameters
295295- let query_params = uri.query().map(|q| {
296296- let mut params = HashMap::new();
297297- for pair in q.split('&') {
298298- if let Some((key, value)) = pair.split_once('=') {
299299- params.insert(key.to_string(), value.to_string());
300300- }
301301- }
302302- params
303303- });
304304-305305- // Get settings
306306- let settings = state.settings.read().await.clone();
307307-308308- // Check for redirect rules first
309309- let redirect_rules = state.redirect_rules.read().await;
310310- if let Some(redirect_match) = match_redirect_rule(path, &redirect_rules, query_params.as_ref()) {
311311- let is_force = redirect_match.force;
312312- drop(redirect_rules); // Release the lock
313313-314314- // If not forced, check if the file exists first
315315- if !is_force {
316316- // Try to serve the file normally first
317317- let test_req = Request::builder()
318318- .uri(uri.clone())
319319- .method(&method)
320320- .body(axum::body::Body::empty())
321321- .unwrap();
322322-323323- match serve_dir.call(test_req).await {
324324- Ok(response) if response.status().is_success() => {
325325- // File exists and was served successfully, return it
326326- return response.into_response();
327327- }
328328- _ => {
329329- // File doesn't exist or error, apply redirect
330330- }
331331- }
332332- }
333333-334334- // Handle different status codes
335335- match redirect_match.status {
336336- 200 => {
337337- // Rewrite: serve the target file but keep the URL the same
338338- if let Ok(target_uri) = redirect_match.target_path.parse::<Uri>() {
339339- let new_req = Request::builder()
340340- .uri(target_uri)
341341- .method(&method)
342342- .body(axum::body::Body::empty())
343343- .unwrap();
344344-345345- match serve_dir.call(new_req).await {
346346- Ok(response) => response.into_response(),
347347- Err(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
348348- }
349349- } else {
350350- StatusCode::INTERNAL_SERVER_ERROR.into_response()
351351- }
352352- }
353353- 301 => {
354354- // Permanent redirect
355355- Redirect::permanent(&redirect_match.target_path).into_response()
356356- }
357357- 302 => {
358358- // Temporary redirect
359359- Redirect::temporary(&redirect_match.target_path).into_response()
360360- }
361361- 404 => {
362362- // Custom 404 page
363363- if let Ok(target_uri) = redirect_match.target_path.parse::<Uri>() {
364364- let new_req = Request::builder()
365365- .uri(target_uri)
366366- .method(&method)
367367- .body(axum::body::Body::empty())
368368- .unwrap();
369369-370370- match serve_dir.call(new_req).await {
371371- Ok(mut response) => {
372372- *response.status_mut() = StatusCode::NOT_FOUND;
373373- response.into_response()
374374- }
375375- Err(_) => StatusCode::NOT_FOUND.into_response(),
376376- }
377377- } else {
378378- StatusCode::NOT_FOUND.into_response()
379379- }
380380- }
381381- _ => {
382382- // Unsupported status code, fall through to normal serving
383383- match serve_dir.call(req).await {
384384- Ok(response) => response.into_response(),
385385- Err(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
386386- }
387387- }
388388- }
389389- } else {
390390- drop(redirect_rules);
391391-392392- // No redirect match, try to serve the file
393393- let response_result = serve_dir.call(req).await;
394394-395395- match response_result {
396396- Ok(response) if response.status().is_success() => {
397397- // File served successfully
398398- response.into_response()
399399- }
400400- Ok(response) if response.status() == StatusCode::NOT_FOUND => {
401401- // File not found, check settings for fallback behavior
402402- if let Some(ref settings) = settings {
403403- // SPA mode takes precedence
404404- if let Some(ref spa_file) = settings.spa_mode {
405405- // Serve the SPA file for all non-file routes
406406- return serve_file_for_spa(&state.output_dir, spa_file.as_ref()).await;
407407- }
408408-409409- // Check if path is a directory and directory listing is enabled
410410- if let Some(true) = settings.directory_listing {
411411- let file_path = state.output_dir.join(path.trim_start_matches('/'));
412412- if file_path.is_dir() {
413413- return serve_directory_listing(&file_path, path).await;
414414- }
415415- }
416416-417417- // Check for custom 404
418418- if let Some(ref custom404) = settings.custom404 {
419419- return serve_custom_404(&state.output_dir, custom404.as_ref()).await;
420420- }
421421- }
422422-423423- // No special handling, return 404
424424- StatusCode::NOT_FOUND.into_response()
425425- }
426426- Ok(response) => response.into_response(),
427427- Err(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
428428- }
429429- }
430430-}
431431-432432-/// Watch the firehose for updates to the specific site
433433-fn watch_firehose(state: ServerState) -> std::pin::Pin<Box<dyn std::future::Future<Output = miette::Result<()>> + Send>> {
434434- Box::pin(async move {
435435- use jacquard_identity::PublicResolver;
436436- use jacquard::prelude::IdentityResolver;
437437-438438- // Resolve DID to PDS URL
439439- let resolver = PublicResolver::default();
440440- let did = Did::new(&state.did).into_diagnostic()?;
441441- let pds_url = resolver.pds_for_did(&did).await.into_diagnostic()?;
442442-443443- println!("[PDS] Resolved DID to PDS: {}", pds_url);
444444-445445- // Convert HTTP(S) URL to WebSocket URL
446446- let mut ws_url = pds_url.clone();
447447- let scheme = if pds_url.scheme() == "https" { "wss" } else { "ws" };
448448- ws_url.set_scheme(scheme)
449449- .map_err(|_| miette::miette!("Failed to set WebSocket scheme"))?;
450450-451451- println!("[PDS] Connecting to {}...", ws_url);
452452-453453- // Create subscription client
454454- let client = TungsteniteSubscriptionClient::from_base_uri(ws_url);
455455-456456- // Subscribe to the PDS firehose
457457- let params = SubscribeRepos::new().build();
458458-459459- let stream = client.subscribe(¶ms).await.into_diagnostic()?;
460460- println!("[PDS] Connected! Watching for updates...");
461461-462462- // Convert to typed message stream
463463- let (_sink, mut messages) = stream.into_stream();
464464-465465- loop {
466466- match messages.next().await {
467467- Some(Ok(msg)) => {
468468- if let Err(e) = handle_firehose_message(&state, msg).await {
469469- eprintln!("[PDS] Error handling message: {}", e);
470470- }
471471- }
472472- Some(Err(e)) => {
473473- eprintln!("[PDS] Stream error: {}", e);
474474- // Try to reconnect after a delay
475475- tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
476476- return Box::pin(watch_firehose(state)).await;
477477- }
478478- None => {
479479- println!("[PDS] Stream ended, reconnecting...");
480480- tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
481481- return Box::pin(watch_firehose(state)).await;
482482- }
483483- }
484484- }
485485- })
486486-}
487487-488488-/// Handle a firehose message
489489-async fn handle_firehose_message<'a>(
490490- state: &ServerState,
491491- msg: SubscribeReposMessage<'a>,
492492-) -> miette::Result<()> {
493493- match msg {
494494- SubscribeReposMessage::Commit(commit_msg) => {
495495- // Check if this commit is from our DID
496496- if commit_msg.repo.as_str() != state.did.as_str() {
497497- return Ok(());
498498- }
499499-500500- // Check if any operation affects our site or settings
501501- let site_path = format!("place.wisp.fs/{}", state.rkey);
502502- let settings_path = format!("place.wisp.settings/{}", state.rkey);
503503- let has_site_update = commit_msg.ops.iter().any(|op| op.path.as_ref() == site_path);
504504- let has_settings_update = commit_msg.ops.iter().any(|op| op.path.as_ref() == settings_path);
505505-506506- if has_site_update {
507507- // Debug: log all operations for this commit
508508- println!("[Debug] Commit has {} ops for {}", commit_msg.ops.len(), state.rkey);
509509- for op in &commit_msg.ops {
510510- if op.path.as_ref() == site_path {
511511- println!("[Debug] - {} {}", op.action.as_ref(), op.path.as_ref());
512512- }
513513- }
514514- }
515515-516516- if has_site_update {
517517- // Use the commit CID as the version tracker
518518- let commit_cid = commit_msg.commit.to_string();
519519-520520- // Check if this is a new commit
521521- let should_update = {
522522- let last_cid = state.last_cid.read().await;
523523- Some(commit_cid.clone()) != *last_cid
524524- };
525525-526526- if should_update {
527527- // Check operation types
528528- let has_create_or_update = commit_msg.ops.iter().any(|op| {
529529- op.path.as_ref() == site_path &&
530530- (op.action.as_ref() == "create" || op.action.as_ref() == "update")
531531- });
532532- let has_delete = commit_msg.ops.iter().any(|op| {
533533- op.path.as_ref() == site_path && op.action.as_ref() == "delete"
534534- });
535535-536536- // If there's a create/update, pull the site (even if there's also a delete in the same commit)
537537- if has_create_or_update {
538538- println!("\n[Update] Detected change to site {} (commit: {})", state.rkey, commit_cid);
539539- println!("[Update] Pulling latest version...");
540540-541541- // Pull the updated site
542542- match pull_site(
543543- state.did.clone(),
544544- state.rkey.clone(),
545545- state.output_dir.clone(),
546546- )
547547- .await
548548- {
549549- Ok(_) => {
550550- // Update last CID
551551- let mut last_cid = state.last_cid.write().await;
552552- *last_cid = Some(commit_cid);
553553-554554- // Reload redirect rules
555555- let new_redirect_rules = load_redirect_rules(&state.output_dir);
556556- let mut redirect_rules = state.redirect_rules.write().await;
557557- *redirect_rules = new_redirect_rules;
558558-559559- println!("[Update] ✓ Site updated successfully!\n");
560560- }
561561- Err(e) => {
562562- eprintln!("[Update] Failed to pull site: {}", e);
563563- }
564564- }
565565- } else if has_delete {
566566- // Only a delete, no create/update
567567- println!("\n[Update] Site {} was deleted", state.rkey);
568568-569569- // Update last CID so we don't process this commit again
570570- let mut last_cid = state.last_cid.write().await;
571571- *last_cid = Some(commit_cid);
572572- }
573573- }
574574- }
575575-576576- // Handle settings updates
577577- if has_settings_update {
578578- println!("\n[Settings] Detected change to settings");
579579-580580- // Resolve PDS URL
581581- use jacquard_identity::PublicResolver;
582582- use jacquard::prelude::IdentityResolver;
583583-584584- let resolver = PublicResolver::default();
585585- let did = Did::new(&state.did).into_diagnostic()?;
586586- let pds_url = resolver.pds_for_did(&did).await.into_diagnostic()?;
587587-588588- // Fetch updated settings
589589- match fetch_settings(&pds_url, &did, state.rkey.as_ref()).await {
590590- Ok(new_settings) => {
591591- let mut settings = state.settings.write().await;
592592- *settings = new_settings.clone();
593593- drop(settings);
594594-595595- if let Some(ref s) = new_settings {
596596- println!("[Settings] Updated:");
597597- if let Some(true) = s.directory_listing {
598598- println!(" • Directory listing: enabled");
599599- }
600600- if let Some(ref spa_file) = s.spa_mode {
601601- println!(" • SPA mode: enabled ({})", spa_file);
602602- }
603603- if let Some(ref custom404) = s.custom404 {
604604- println!(" • Custom 404: {}", custom404);
605605- }
606606- } else {
607607- println!("[Settings] Cleared (using defaults)");
608608- }
609609- }
610610- Err(e) => {
611611- eprintln!("[Settings] Failed to fetch updated settings: {}", e);
612612- }
613613- }
614614- }
615615- }
616616- _ => {
617617- // Ignore identity and account messages
618618- }
619619- }
620620-621621- Ok(())
622622-}
623623-
-497
rust-cli/src/subfs_utils.rs
···11-use jacquard_common::types::string::AtUri;
22-use jacquard_common::types::blob::BlobRef;
33-use jacquard_common::IntoStatic;
44-use jacquard::client::{Agent, AgentSession, AgentSessionExt};
55-use jacquard::prelude::IdentityResolver;
66-use miette::IntoDiagnostic;
77-use std::collections::HashMap;
88-99-use wisp_lexicons::place_wisp::fs::{Directory as FsDirectory, EntryNode as FsEntryNode};
1010-use wisp_lexicons::place_wisp::subfs::SubfsRecord;
1111-1212-/// Extract all subfs URIs from a directory tree with their mount paths
1313-pub fn extract_subfs_uris(directory: &FsDirectory, current_path: String) -> Vec<(String, String)> {
1414- let mut uris = Vec::new();
1515-1616- for entry in &directory.entries {
1717- let full_path = if current_path.is_empty() {
1818- entry.name.to_string()
1919- } else {
2020- format!("{}/{}", current_path, entry.name)
2121- };
2222-2323- match &entry.node {
2424- FsEntryNode::Subfs(subfs_node) => {
2525- // Found a subfs node - store its URI and mount path
2626- uris.push((subfs_node.subject.to_string(), full_path.clone()));
2727- }
2828- FsEntryNode::Directory(subdir) => {
2929- // Recursively search subdirectories
3030- let sub_uris = extract_subfs_uris(subdir, full_path);
3131- uris.extend(sub_uris);
3232- }
3333- FsEntryNode::File(_) => {
3434- // Files don't contain subfs
3535- }
3636- FsEntryNode::Unknown(_) => {
3737- // Skip unknown nodes
3838- }
3939- }
4040- }
4141-4242- uris
4343-}
4444-4545-/// Fetch a subfs record from the PDS
4646-pub async fn fetch_subfs_record(
4747- agent: &Agent<impl AgentSession + IdentityResolver>,
4848- uri: &str,
4949-) -> miette::Result<SubfsRecord<'static>> {
5050- // Parse URI: at://did/collection/rkey
5151- let parts: Vec<&str> = uri.trim_start_matches("at://").split('/').collect();
5252-5353- if parts.len() < 3 {
5454- return Err(miette::miette!("Invalid subfs URI: {}", uri));
5555- }
5656-5757- let _did = parts[0];
5858- let collection = parts[1];
5959- let _rkey = parts[2];
6060-6161- if collection != "place.wisp.subfs" {
6262- return Err(miette::miette!("Expected place.wisp.subfs collection, got: {}", collection));
6363- }
6464-6565- // Construct AT-URI for fetching
6666- let at_uri = AtUri::new(uri).into_diagnostic()?;
6767-6868- // Fetch the record
6969- let response = agent.get_record::<SubfsRecord>(&at_uri).await.into_diagnostic()?;
7070- let record_output = response.into_output().into_diagnostic()?;
7171-7272- Ok(record_output.value.into_static())
7373-}
7474-7575-/// Recursively fetch all subfs records (including nested ones)
7676-/// Returns a list of (mount_path, SubfsRecord) tuples
7777-/// Note: Multiple records can have the same mount_path (for flat-merged chunks)
7878-pub async fn fetch_all_subfs_records_recursive(
7979- agent: &Agent<impl AgentSession + IdentityResolver>,
8080- initial_uris: Vec<(String, String)>,
8181-) -> miette::Result<Vec<(String, SubfsRecord<'static>)>> {
8282- use futures::stream::{self, StreamExt};
8383-8484- let mut all_subfs: Vec<(String, SubfsRecord<'static>)> = Vec::new();
8585- let mut fetched_uris: std::collections::HashSet<String> = std::collections::HashSet::new();
8686- let mut to_fetch = initial_uris;
8787-8888- if to_fetch.is_empty() {
8989- return Ok(all_subfs);
9090- }
9191-9292- println!("Found {} subfs records, fetching recursively...", to_fetch.len());
9393-9494- let mut iteration = 0;
9595- const MAX_ITERATIONS: usize = 10;
9696-9797- while !to_fetch.is_empty() && iteration < MAX_ITERATIONS {
9898- iteration += 1;
9999- println!(" Iteration {}: fetching {} subfs records...", iteration, to_fetch.len());
100100-101101- let subfs_results: Vec<_> = stream::iter(to_fetch.clone())
102102- .map(|(uri, mount_path)| async move {
103103- match fetch_subfs_record(agent, &uri).await {
104104- Ok(record) => Some((mount_path, record, uri)),
105105- Err(e) => {
106106- eprintln!(" ⚠️ Failed to fetch subfs {}: {}", uri, e);
107107- None
108108- }
109109- }
110110- })
111111- .buffer_unordered(5)
112112- .collect()
113113- .await;
114114-115115- // Process results and find nested subfs
116116- let mut newly_found_uris = Vec::new();
117117- for result in subfs_results {
118118- if let Some((mount_path, record, uri)) = result {
119119- println!(" ✓ Fetched subfs at {}", mount_path);
120120-121121- // Extract nested subfs URIs from this record
122122- let nested_uris = extract_subfs_uris_from_subfs_dir(&record.root, mount_path.clone());
123123- newly_found_uris.extend(nested_uris);
124124-125125- all_subfs.push((mount_path, record));
126126- fetched_uris.insert(uri);
127127- }
128128- }
129129-130130- // Filter out already-fetched URIs (based on URI, not path)
131131- to_fetch = newly_found_uris
132132- .into_iter()
133133- .filter(|(uri, _)| !fetched_uris.contains(uri))
134134- .collect();
135135- }
136136-137137- if iteration >= MAX_ITERATIONS {
138138- eprintln!("⚠️ Max iterations reached while fetching nested subfs");
139139- }
140140-141141- println!(" Total subfs records fetched: {}", all_subfs.len());
142142-143143- Ok(all_subfs)
144144-}
145145-146146-/// Extract subfs URIs from a subfs::Directory
147147-fn extract_subfs_uris_from_subfs_dir(
148148- directory: &wisp_lexicons::place_wisp::subfs::Directory,
149149- current_path: String,
150150-) -> Vec<(String, String)> {
151151- let mut uris = Vec::new();
152152-153153- for entry in &directory.entries {
154154- match &entry.node {
155155- wisp_lexicons::place_wisp::subfs::EntryNode::Subfs(subfs_node) => {
156156- // Check if this is a chunk entry (chunk0, chunk1, etc.)
157157- // Chunks should be flat-merged, so use the parent's path
158158- let mount_path = if entry.name.starts_with("chunk") &&
159159- entry.name.chars().skip(5).all(|c| c.is_ascii_digit()) {
160160- // This is a chunk - use parent's path for flat merge
161161- println!(" → Found chunk {} at {}, will flat-merge to {}", entry.name, current_path, current_path);
162162- current_path.clone()
163163- } else {
164164- // Normal subfs - append name to path
165165- if current_path.is_empty() {
166166- entry.name.to_string()
167167- } else {
168168- format!("{}/{}", current_path, entry.name)
169169- }
170170- };
171171-172172- uris.push((subfs_node.subject.to_string(), mount_path));
173173- }
174174- wisp_lexicons::place_wisp::subfs::EntryNode::Directory(subdir) => {
175175- let full_path = if current_path.is_empty() {
176176- entry.name.to_string()
177177- } else {
178178- format!("{}/{}", current_path, entry.name)
179179- };
180180- let nested = extract_subfs_uris_from_subfs_dir(subdir, full_path);
181181- uris.extend(nested);
182182- }
183183- _ => {}
184184- }
185185- }
186186-187187- uris
188188-}
189189-190190-/// Merge blob maps from subfs records into the main blob map (RECURSIVE)
191191-/// Returns the total number of blobs merged from all subfs records
192192-pub async fn merge_subfs_blob_maps(
193193- agent: &Agent<impl AgentSession + IdentityResolver>,
194194- subfs_uris: Vec<(String, String)>,
195195- main_blob_map: &mut HashMap<String, (BlobRef<'static>, String)>,
196196-) -> miette::Result<usize> {
197197- // Fetch all subfs records recursively
198198- let all_subfs = fetch_all_subfs_records_recursive(agent, subfs_uris).await?;
199199-200200- let mut total_merged = 0;
201201-202202- // Extract blobs from all fetched subfs records
203203- // Skip parent records that only contain chunk references (no actual files)
204204- for (mount_path, subfs_record) in all_subfs {
205205- // Check if this record only contains chunk subfs references (no files)
206206- let only_has_chunks = subfs_record.root.entries.iter().all(|e| {
207207- matches!(&e.node, wisp_lexicons::place_wisp::subfs::EntryNode::Subfs(_)) &&
208208- e.name.starts_with("chunk") &&
209209- e.name.chars().skip(5).all(|c| c.is_ascii_digit())
210210- });
211211-212212- if only_has_chunks && !subfs_record.root.entries.is_empty() {
213213- // This is a parent containing only chunks - skip it, blobs are in the chunks
214214- println!(" → Skipping parent subfs at {} ({} chunks, no files)", mount_path, subfs_record.root.entries.len());
215215- continue;
216216- }
217217-218218- let subfs_blob_map = extract_subfs_blobs(&subfs_record.root, mount_path.clone());
219219- let count = subfs_blob_map.len();
220220-221221- for (path, blob_info) in subfs_blob_map {
222222- main_blob_map.insert(path, blob_info);
223223- }
224224-225225- total_merged += count;
226226- println!(" ✓ Merged {} blobs from subfs at {}", count, mount_path);
227227- }
228228-229229- Ok(total_merged)
230230-}
231231-232232-/// Extract blobs from a subfs directory (works with subfs::Directory)
233233-/// Returns a map of file paths to their blob refs and CIDs
234234-fn extract_subfs_blobs(
235235- directory: &wisp_lexicons::place_wisp::subfs::Directory,
236236- current_path: String,
237237-) -> HashMap<String, (BlobRef<'static>, String)> {
238238- let mut blob_map = HashMap::new();
239239-240240- for entry in &directory.entries {
241241- let full_path = if current_path.is_empty() {
242242- entry.name.to_string()
243243- } else {
244244- format!("{}/{}", current_path, entry.name)
245245- };
246246-247247- match &entry.node {
248248- wisp_lexicons::place_wisp::subfs::EntryNode::File(file_node) => {
249249- let blob_ref = &file_node.blob;
250250- let cid_string = blob_ref.blob().r#ref.to_string();
251251- blob_map.insert(
252252- full_path,
253253- (blob_ref.clone().into_static(), cid_string)
254254- );
255255- }
256256- wisp_lexicons::place_wisp::subfs::EntryNode::Directory(subdir) => {
257257- let sub_map = extract_subfs_blobs(subdir, full_path);
258258- blob_map.extend(sub_map);
259259- }
260260- wisp_lexicons::place_wisp::subfs::EntryNode::Subfs(_nested_subfs) => {
261261- // Nested subfs - these should be resolved recursively in the main flow
262262- // For now, we skip them (they'll be fetched separately)
263263- eprintln!(" ⚠️ Found nested subfs at {}, skipping (should be fetched separately)", full_path);
264264- }
265265- wisp_lexicons::place_wisp::subfs::EntryNode::Unknown(_) => {
266266- // Skip unknown nodes
267267- }
268268- }
269269- }
270270-271271- blob_map
272272-}
273273-274274-/// Count total files in a directory tree
275275-pub fn count_files_in_directory(directory: &FsDirectory) -> usize {
276276- let mut count = 0;
277277-278278- for entry in &directory.entries {
279279- match &entry.node {
280280- FsEntryNode::File(_) => count += 1,
281281- FsEntryNode::Directory(subdir) => {
282282- count += count_files_in_directory(subdir);
283283- }
284284- FsEntryNode::Subfs(_) => {
285285- // Subfs nodes don't count towards the main manifest file count
286286- }
287287- FsEntryNode::Unknown(_) => {}
288288- }
289289- }
290290-291291- count
292292-}
293293-294294-/// Estimate JSON size of a directory tree
295295-pub fn estimate_directory_size(directory: &FsDirectory) -> usize {
296296- // Serialize to JSON and measure
297297- match serde_json::to_string(directory) {
298298- Ok(json) => json.len(),
299299- Err(_) => 0,
300300- }
301301-}
302302-303303-/// Information about a directory that could be split into a subfs record
304304-#[derive(Debug)]
305305-pub struct SplittableDirectory {
306306- pub path: String,
307307- pub directory: FsDirectory<'static>,
308308- pub size: usize,
309309- pub file_count: usize,
310310-}
311311-312312-/// Find large directories that could be split into subfs records
313313-/// Returns directories sorted by size (largest first)
314314-pub fn find_large_directories(directory: &FsDirectory, current_path: String) -> Vec<SplittableDirectory> {
315315- let mut result = Vec::new();
316316-317317- for entry in &directory.entries {
318318- if let FsEntryNode::Directory(subdir) = &entry.node {
319319- let dir_path = if current_path.is_empty() {
320320- entry.name.to_string()
321321- } else {
322322- format!("{}/{}", current_path, entry.name)
323323- };
324324-325325- let size = estimate_directory_size(subdir);
326326- let file_count = count_files_in_directory(subdir);
327327-328328- result.push(SplittableDirectory {
329329- path: dir_path.clone(),
330330- directory: (*subdir.clone()).into_static(),
331331- size,
332332- file_count,
333333- });
334334-335335- // Recursively find subdirectories
336336- let subdirs = find_large_directories(subdir, dir_path);
337337- result.extend(subdirs);
338338- }
339339- }
340340-341341- // Sort by size (largest first)
342342- result.sort_by(|a, b| b.size.cmp(&a.size));
343343-344344- result
345345-}
346346-347347-/// Replace a directory with a subfs node in the tree
348348-pub fn replace_directory_with_subfs(
349349- directory: FsDirectory<'static>,
350350- target_path: &str,
351351- subfs_uri: &str,
352352- flat: bool,
353353-) -> miette::Result<FsDirectory<'static>> {
354354- use jacquard_common::CowStr;
355355- use wisp_lexicons::place_wisp::fs::{Entry, Subfs};
356356-357357- let path_parts: Vec<&str> = target_path.split('/').collect();
358358-359359- if path_parts.is_empty() {
360360- return Err(miette::miette!("Cannot replace root directory"));
361361- }
362362-363363- // Parse the subfs URI and make it owned/'static
364364- let at_uri = AtUri::new_cow(jacquard_common::CowStr::from(subfs_uri.to_string())).into_diagnostic()?;
365365-366366- // If this is a root-level directory
367367- if path_parts.len() == 1 {
368368- let target_name = path_parts[0];
369369- let new_entries: Vec<Entry> = directory.entries.into_iter().map(|entry| {
370370- if entry.name == target_name {
371371- // Replace this directory with a subfs node
372372- Entry::new()
373373- .name(entry.name)
374374- .node(FsEntryNode::Subfs(Box::new(
375375- Subfs::new()
376376- .r#type(CowStr::from("subfs"))
377377- .subject(at_uri.clone())
378378- .flat(Some(flat))
379379- .build()
380380- )))
381381- .build()
382382- } else {
383383- entry
384384- }
385385- }).collect();
386386-387387- return Ok(FsDirectory::new()
388388- .r#type(CowStr::from("directory"))
389389- .entries(new_entries)
390390- .build());
391391- }
392392-393393- // Recursively navigate to parent directory
394394- let first_part = path_parts[0];
395395- let remaining_path = path_parts[1..].join("/");
396396-397397- let new_entries: Vec<Entry> = directory.entries.into_iter().filter_map(|entry| {
398398- if entry.name == first_part {
399399- if let FsEntryNode::Directory(subdir) = entry.node {
400400- // Recursively process this subdirectory
401401- match replace_directory_with_subfs((*subdir).into_static(), &remaining_path, subfs_uri, flat) {
402402- Ok(updated_subdir) => {
403403- Some(Entry::new()
404404- .name(entry.name)
405405- .node(FsEntryNode::Directory(Box::new(updated_subdir)))
406406- .build())
407407- }
408408- Err(_) => None, // Skip entries that fail to update
409409- }
410410- } else {
411411- Some(entry)
412412- }
413413- } else {
414414- Some(entry)
415415- }
416416- }).collect();
417417-418418- Ok(FsDirectory::new()
419419- .r#type(CowStr::from("directory"))
420420- .entries(new_entries)
421421- .build())
422422-}
423423-424424-/// Delete a subfs record from the PDS
425425-pub async fn delete_subfs_record(
426426- agent: &Agent<impl AgentSession + IdentityResolver>,
427427- uri: &str,
428428-) -> miette::Result<()> {
429429- use jacquard_common::types::uri::RecordUri;
430430-431431- // Construct AT-URI and convert to RecordUri
432432- let at_uri = AtUri::new(uri).into_diagnostic()?;
433433- let record_uri: RecordUri<'_, wisp_lexicons::place_wisp::subfs::SubfsRecordRecord> = RecordUri::try_from_uri(at_uri).into_diagnostic()?;
434434-435435- let rkey = record_uri.rkey()
436436- .ok_or_else(|| miette::miette!("Invalid subfs URI: missing rkey"))?
437437- .clone();
438438-439439- agent.delete_record::<SubfsRecord>(rkey).await.into_diagnostic()?;
440440-441441- Ok(())
442442-}
443443-444444-/// Split a large directory into multiple smaller chunks
445445-/// Returns a list of chunk directories, each small enough to fit in a subfs record
446446-pub fn split_directory_into_chunks(
447447- directory: &FsDirectory,
448448- max_size: usize,
449449-) -> Vec<FsDirectory<'static>> {
450450- use jacquard_common::CowStr;
451451-452452- let mut chunks = Vec::new();
453453- let mut current_chunk_entries = Vec::new();
454454- let mut current_chunk_size = 100; // Base size for directory structure
455455-456456- for entry in &directory.entries {
457457- // Estimate the size of this entry
458458- let entry_size = estimate_entry_size(entry);
459459-460460- // If adding this entry would exceed the max size, start a new chunk
461461- if !current_chunk_entries.is_empty() && (current_chunk_size + entry_size > max_size) {
462462- // Create a chunk from current entries
463463- let chunk = FsDirectory::new()
464464- .r#type(CowStr::from("directory"))
465465- .entries(current_chunk_entries.clone())
466466- .build();
467467-468468- chunks.push(chunk);
469469-470470- // Start new chunk
471471- current_chunk_entries.clear();
472472- current_chunk_size = 100;
473473- }
474474-475475- current_chunk_entries.push(entry.clone().into_static());
476476- current_chunk_size += entry_size;
477477- }
478478-479479- // Add the last chunk if it has any entries
480480- if !current_chunk_entries.is_empty() {
481481- let chunk = FsDirectory::new()
482482- .r#type(CowStr::from("directory"))
483483- .entries(current_chunk_entries)
484484- .build();
485485- chunks.push(chunk);
486486- }
487487-488488- chunks
489489-}
490490-491491-/// Estimate the JSON size of a single entry
492492-fn estimate_entry_size(entry: &wisp_lexicons::place_wisp::fs::Entry) -> usize {
493493- match serde_json::to_string(entry) {
494494- Ok(json) => json.len(),
495495- Err(_) => 500, // Conservative estimate if serialization fails
496496- }
497497-}