forked from
tokono.ma/diffuse
A music player that connects to your cloud/distributed storage.
chore: https+json input plan
+246
+246
docs/plans/https-directory-listing.md
+246
docs/plans/https-directory-listing.md
···
1
+
# HTTPS directory listing (`https+json`)
2
+
3
+
## Context
4
+
5
+
Some HTTPS servers expose a directory listing API: appending `?ls` to a directory URL returns a JSON array describing its contents:
6
+
7
+
```json
8
+
[
9
+
{ "name": "my-directory", "type": "directory", "mtime": "2022-10-07T00:53:50Z" },
10
+
{ "name": "my_file.tar.gz", "type": "file", "mtime": "2022-09-27T22:44:34Z", "size": 332 }
11
+
]
12
+
```
13
+
14
+
Rather than bolting this onto the existing HTTPS input (which is designed for individual file URLs), this feature is implemented as a separate `https+json` input with its own URI scheme. Users add a server URL (e.g. `https+json://music.example.com`) and the input automatically discovers all audio files under it — recursively traversing subdirectories.
15
+
16
+
The existing HTTPS input is left untouched.
17
+
18
+
## Plan
19
+
20
+
### 1. Create `src/components/input/https-json/constants.js`
21
+
22
+
```js
23
+
export const SCHEME = "https+json";
24
+
```
25
+
26
+
### 2. Create `src/components/input/https-json/common.js`
27
+
28
+
Follows the same structure as `https/common.js`. Key additions are `buildURI` and `listDirectory`.
29
+
30
+
**`parseURI(uriString)`** — validates the `https+json:` scheme and returns components:
31
+
32
+
```js
33
+
export function parseURI(uriString) {
34
+
try {
35
+
const url = new URL(uriString);
36
+
if (url.protocol !== "https+json:") return undefined;
37
+
38
+
return {
39
+
// The real HTTPS URL (for fetch calls)
40
+
url: "https:" + uriString.slice("https+json:".length),
41
+
domain: url.hostname,
42
+
host: url.host,
43
+
path: url.pathname,
44
+
};
45
+
} catch {
46
+
return undefined;
47
+
}
48
+
}
49
+
```
50
+
51
+
**`buildURI(host, path)`** — constructs an `https+json://` URI:
52
+
53
+
```js
54
+
export function buildURI(host, path = "/") {
55
+
return `https+json://${host}${path}`;
56
+
}
57
+
```
58
+
59
+
**`listDirectory(httpsUrl)`** — fetches `url?ls`, parses the JSON listing, and recursively collects audio file URLs. Returns `https+json://` URIs (not plain `https://`):
60
+
61
+
```js
62
+
export async function listDirectory(httpsUrl) {
63
+
let entries;
64
+
65
+
try {
66
+
const response = await fetch(httpsUrl + "?ls");
67
+
if (!response.ok) return [];
68
+
entries = await response.json();
69
+
} catch {
70
+
return [];
71
+
}
72
+
73
+
if (!Array.isArray(entries)) return [];
74
+
75
+
const results = await Promise.all(
76
+
entries.map(async (entry) => {
77
+
if (entry.type === "file") {
78
+
if (!isAudioFile(entry.name)) return [];
79
+
const fileHttpsUrl = httpsUrl + entry.name;
80
+
// Return as https+json:// URI
81
+
return [fileHttpsUrl.replace(/^https:/, "https+json:")];
82
+
} else {
83
+
return listDirectory(httpsUrl + entry.name + "/");
84
+
}
85
+
}),
86
+
);
87
+
88
+
return results.flat(1);
89
+
}
90
+
```
91
+
92
+
Include `groupTracksByHost`, `groupUrisByHost`, `hostsFromTracks` — same implementations as in `https/common.js` but using `parseURI` from this module. Include `consultHostCached` — same implementation, caches per host.
93
+
94
+
### 3. Create `src/components/input/https-json/worker.js`
95
+
96
+
Five standard actions. Mirrors the S3 worker pattern for `list()`.
97
+
98
+
**`consult`** — same shape as HTTPS: scheme-only returns `undetermined`, full URI checks host reachability via `consultHostCached`.
99
+
100
+
**`detach`** — same shape as HTTPS: groups by host, removes the target host's group.
101
+
102
+
**`groupConsult`** — same shape as HTTPS: groups URIs by host, checks each host once.
103
+
104
+
**`list(cachedTracks)`** — the key new logic:
105
+
106
+
```js
107
+
export async function list(cachedTracks = []) {
108
+
// Build a URI → track cache for preserving metadata across relists.
109
+
/** @type {Record<string, Track>} */
110
+
const cacheByUri = {};
111
+
cachedTracks.forEach((t) => { cacheByUri[t.uri] = t; });
112
+
113
+
// Group by host; derive the scan root per host.
114
+
// If placeholder tracks exist for a host, use their path as the root.
115
+
// Otherwise, find the common path prefix of the audio file tracks.
116
+
const groups = groupTracksByHost(cachedTracks);
117
+
118
+
const promises = Object.values(groups).map(async ({ host, tracks }) => {
119
+
const root = scanRoot(tracks); // "/" or a shared directory prefix
120
+
const rootHttpsUrl = `https://${host}${root}`;
121
+
const now = new Date().toISOString();
122
+
123
+
const uris = await listDirectory(rootHttpsUrl);
124
+
125
+
let discovered = uris.map((uri) => {
126
+
const cached = cacheByUri[uri];
127
+
/** @type {Track} */
128
+
return {
129
+
$type: "sh.diffuse.output.track",
130
+
id: cached?.id ?? TID.now(),
131
+
createdAt: cached?.createdAt ?? now,
132
+
updatedAt: cached?.updatedAt ?? now,
133
+
stats: cached?.stats,
134
+
tags: cached?.tags,
135
+
uri,
136
+
};
137
+
});
138
+
139
+
if (!discovered.length) {
140
+
discovered = [{
141
+
$type: "sh.diffuse.output.track",
142
+
id: TID.now(),
143
+
createdAt: now,
144
+
updatedAt: now,
145
+
kind: "placeholder",
146
+
uri: buildURI(host, root),
147
+
}];
148
+
}
149
+
150
+
return discovered;
151
+
});
152
+
153
+
return (await Promise.all(promises)).flat(1);
154
+
}
155
+
```
156
+
157
+
`scanRoot(tracks)` is a small helper (defined in `worker.js` or `common.js`) that:
158
+
- Returns the path of the first placeholder-kind track if one exists
159
+
- Otherwise returns the longest common path prefix of all track paths, trimmed to the last `/`
160
+
161
+
**`resolve({ uri })`** — strips the scheme prefix and returns the `https://` URL with a one-year expiry (same as HTTPS input):
162
+
163
+
```js
164
+
export async function resolve({ uri }) {
165
+
const parsed = parseURI(uri);
166
+
if (!parsed) return undefined;
167
+
168
+
const expiresAt = Math.round(Date.now() / 1000) + 60 * 60 * 24 * 365;
169
+
return { url: parsed.url, expiresAt };
170
+
}
171
+
```
172
+
173
+
### 4. Create `src/components/input/https-json/element.js`
174
+
175
+
Minimal — same shape as `https/element.js`. `sources()` returns unique hosts from tracks.
176
+
177
+
```js
178
+
class HttpsJsonInput extends DiffuseElement {
179
+
static NAME = "diffuse/input/https-json";
180
+
static WORKER_URL = "components/input/https-json/worker.js";
181
+
182
+
SCHEME = SCHEME;
183
+
184
+
constructor() {
185
+
super();
186
+
this.proxy = this.workerProxy();
187
+
this.consult = this.proxy.consult;
188
+
this.detach = this.proxy.detach;
189
+
this.groupConsult = this.proxy.groupConsult;
190
+
this.list = this.proxy.list;
191
+
this.resolve = this.proxy.resolve;
192
+
}
193
+
194
+
sources(tracks) {
195
+
return Object.values(hostsFromTracks(tracks)).map((host) => ({
196
+
label: host,
197
+
uri: buildURI(host),
198
+
}));
199
+
}
200
+
}
201
+
202
+
export const CLASS = HttpsJsonInput;
203
+
export const NAME = "di-https-json";
204
+
205
+
customElements.define(NAME, CLASS);
206
+
```
207
+
208
+
### 5. Register in `src/components/orchestrator/input/element.js`
209
+
210
+
Add the import and the element to the rendered template:
211
+
212
+
```js
213
+
import "@components/input/https-json/element.js";
214
+
// ...
215
+
render({ html }) {
216
+
return html`
217
+
<dc-input>
218
+
<di-https></di-https>
219
+
<di-https-json></di-https-json>
220
+
<di-opensubsonic></di-opensubsonic>
221
+
<di-s3></di-s3>
222
+
</dc-input>
223
+
`;
224
+
}
225
+
```
226
+
227
+
## Files changed (5)
228
+
229
+
| File | Change | Difficulty |
230
+
|------|--------|------------|
231
+
| `src/components/input/https-json/constants.js` | New — define `SCHEME = "https+json"` | Trivial |
232
+
| `src/components/input/https-json/common.js` | New — `parseURI`, `buildURI`, `listDirectory`, grouping helpers, `consultHostCached` | Low |
233
+
| `src/components/input/https-json/worker.js` | New — all 5 actions; `list()` does recursive `?ls` fetching | Low-medium |
234
+
| `src/components/input/https-json/element.js` | New — `HttpsJsonInput` element, `di-https-json` | Trivial |
235
+
| `src/components/orchestrator/input/element.js` | Add import + `<di-https-json>` to template | Trivial |
236
+
237
+
**Overall difficulty: Low-medium.** All patterns are direct copies or adaptations of the existing HTTPS and S3 inputs. The only genuinely new logic is `listDirectory()` (recursive `?ls` fetch) and `scanRoot()` (derive listing root from cached tracks).
238
+
239
+
## Verification
240
+
241
+
- Add `https+json://music.example.com` as a source and trigger a library refresh
242
+
- Confirm audio files from the server's directory tree appear as tracks
243
+
- Confirm cached metadata (tags, stats) is preserved across relists
244
+
- Add a server with no audio files — confirm a placeholder track is produced
245
+
- Confirm `resolve()` returns a working `https://` URL for each track
246
+
- Confirm `di-https` (plain HTTPS) is unaffected