Sync articles marked toread in kipclip to Crosspoint Reader (Xteink X4)
Initial commit: crosspoint articles backend
Deno/Val Town backend for syncing and processing AT Protocol articles into EPUBs.
+1596
+19
.gitignore
+19
.gitignore
+7
.vtignore
+7
.vtignore
+290
AGENTS.md
+290
AGENTS.md
···
1
+
You are an advanced assistant specialized in generating Val Town code.
2
+
3
+
## Core Guidelines
4
+
5
+
- Ask clarifying questions when requirements are ambiguous
6
+
- Provide complete, functional solutions rather than skeleton implementations
7
+
- Test your logic against edge cases before presenting the final solution
8
+
- Ensure all code follows Val Town's specific platform requirements
9
+
- If a section of code that you're working on is getting too complex, consider refactoring it into subcomponents
10
+
11
+
## Code Standards
12
+
13
+
- Generate code in TypeScript or TSX
14
+
- Add appropriate TypeScript types and interfaces for all data structures
15
+
- Prefer official SDKs or libraries than writing API calls directly
16
+
- Ask the user to supply API or library documentation if you are at all unsure about it
17
+
- **Never bake in secrets into the code** - always use environment variables
18
+
- Include comments explaining complex logic (avoid commenting obvious operations)
19
+
- Follow modern ES6+ conventions and functional programming practices if possible
20
+
21
+
## Types of triggers
22
+
23
+
### 1. HTTP Trigger
24
+
25
+
- Create web APIs and endpoints
26
+
- Handle HTTP requests and responses
27
+
- Example structure:
28
+
29
+
```ts
30
+
export default async function (req: Request) {
31
+
return new Response("Hello World");
32
+
}
33
+
```
34
+
35
+
Files that are HTTP triggers have http in their name like `foobar.http.tsx`
36
+
37
+
### 2. Cron Triggers
38
+
39
+
- Run on a schedule
40
+
- Use cron expressions for timing
41
+
- Example structure:
42
+
43
+
```ts
44
+
export default async function () {
45
+
// Scheduled task code
46
+
}
47
+
```
48
+
49
+
Files that are Cron triggers have cron in their name like `foobar.cron.tsx`
50
+
51
+
### 3. Email Triggers
52
+
53
+
- Process incoming emails
54
+
- Handle email-based workflows
55
+
- Example structure:
56
+
57
+
```ts
58
+
export default async function (email: Email) {
59
+
// Process email
60
+
}
61
+
```
62
+
63
+
Files that are Email triggers have email in their name like `foobar.email.tsx`
64
+
65
+
66
+
## Val Town Standard Libraries
67
+
68
+
Val Town provides several hosted services and utility functions.
69
+
70
+
### Blob Storage
71
+
72
+
```ts
73
+
import { blob } from "https://esm.town/v/std/blob";
74
+
await blob.setJSON("myKey", { hello: "world" });
75
+
let blobDemo = await blob.getJSON("myKey");
76
+
let appKeys = await blob.list("app_");
77
+
await blob.delete("myKey");
78
+
```
79
+
80
+
### SQLite
81
+
82
+
```ts
83
+
import { sqlite } from "https://esm.town/v/stevekrouse/sqlite";
84
+
const TABLE_NAME = 'todo_app_users_2';
85
+
// Create table - do this before usage and change table name when modifying schema
86
+
await sqlite.execute(`CREATE TABLE IF NOT EXISTS ${TABLE_NAME} (
87
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
88
+
name TEXT NOT NULL
89
+
)`);
90
+
// Query data
91
+
const result = await sqlite.execute(`SELECT * FROM ${TABLE_NAME} WHERE id = ?`, [1]);
92
+
```
93
+
94
+
Note: When changing a SQLite table's schema, change the table's name (e.g., add _2 or _3) to create a fresh table.
95
+
96
+
### OpenAI
97
+
98
+
```ts
99
+
import { OpenAI } from "https://esm.town/v/std/openai";
100
+
const openai = new OpenAI();
101
+
const completion = await openai.chat.completions.create({
102
+
messages: [
103
+
{ role: "user", content: "Say hello in a creative way" },
104
+
],
105
+
model: "gpt-4o-mini",
106
+
max_tokens: 30,
107
+
});
108
+
```
109
+
110
+
### Email
111
+
112
+
```ts
113
+
import { email } from "https://esm.town/v/std/email";
114
+
// By default emails the owner of the val
115
+
await email({
116
+
subject: "Hi",
117
+
text: "Hi",
118
+
html: "<h1>Hi</h1>"
119
+
});
120
+
```
121
+
122
+
## Val Town Utility Functions
123
+
124
+
Val Town provides several utility functions to help with common project tasks.
125
+
126
+
### Importing Utilities
127
+
128
+
Always import utilities with version pins to avoid breaking changes:
129
+
130
+
```ts
131
+
import { parseProject, readFile, serveFile } from "https://esm.town/v/std/utils@85-main/index.ts";
132
+
```
133
+
134
+
### Available Utilities
135
+
136
+
137
+
#### **serveFile** - Serve project files with proper content types
138
+
139
+
For example, in Hono:
140
+
141
+
```ts
142
+
// serve all files in frontend/ and shared/
143
+
app.get("/frontend/*", c => serveFile(c.req.path, import.meta.url));
144
+
app.get("/shared/*", c => serveFile(c.req.path, import.meta.url));
145
+
```
146
+
147
+
#### **readFile** - Read files from within the project:
148
+
149
+
```ts
150
+
// Read a file from the project
151
+
const fileContent = await readFile("/frontend/index.html", import.meta.url);
152
+
```
153
+
154
+
#### **listFiles** - List all files in the project
155
+
156
+
```ts
157
+
const files = await listFiles(import.meta.url);
158
+
```
159
+
160
+
#### **parseProject** - Extract information about the current project from import.meta.url
161
+
162
+
This is useful for including info for linking back to a val, ie in "view source" urls:
163
+
164
+
```ts
165
+
const projectVal = parseProject(import.meta.url);
166
+
console.log(projectVal.username); // Owner of the project
167
+
console.log(projectVal.name); // Project name
168
+
console.log(projectVal.version); // Version number
169
+
console.log(projectVal.branch); // Branch name
170
+
console.log(projectVal.links.self.project); // URL to the project page
171
+
```
172
+
173
+
However, it's *extremely importing* to note that `parseProject` and other Standard Library utilities ONLY RUN ON THE SERVER.
174
+
If you need access to this data on the client, run it in the server and pass it to the client by splicing it into the HTML page
175
+
or by making an API request for it.
176
+
177
+
## Val Town Platform Specifics
178
+
179
+
- **Redirects:** Use `return new Response(null, { status: 302, headers: { Location: "/place/to/redirect" }})` instead of `Response.redirect` which is broken
180
+
- **Images:** Avoid external images or base64 images. Use emojis, unicode symbols, or icon fonts/libraries instead
181
+
- **AI Image:** To inline generate an AI image use: `<img src="https://maxm-imggenurl.web.val.run/the-description-of-your-image" />`
182
+
- **Storage:** DO NOT use the Deno KV module for storage
183
+
- **Browser APIs:** DO NOT use the `alert()`, `prompt()`, or `confirm()` methods
184
+
- **Weather Data:** Use open-meteo for weather data (doesn't require API keys) unless otherwise specified
185
+
- **View Source:** Add a view source link by importing & using `import.meta.url.replace("ems.sh", "val.town)"` (or passing this data to the client) and include `target="_top"` attribute
186
+
- **Error Debugging:** Add `<script src="https://esm.town/v/std/catch"></script>` to HTML to capture client-side errors
187
+
- **Error Handling:** Only use try...catch when there's a clear local resolution; Avoid catches that merely log or return 500s. Let errors bubble up with full context
188
+
- **Environment Variables:** Use `Deno.env.get('keyname')` when you need to, but generally prefer APIs that don't require keys
189
+
- **Imports:** Use `https://esm.sh` for npm and Deno dependencies to ensure compatibility on server and browser
190
+
- **Storage Strategy:** Only use backend storage if explicitly required; prefer simple static client-side sites
191
+
- **React Configuration:** When using React libraries, pin versions with `?deps=react@18.2.0,react-dom@18.2.0` and start the file with `/** @jsxImportSource https://esm.sh/react@18.2.0 */`
192
+
- Ensure all React dependencies and sub-dependencies are pinned to the same version
193
+
- **Styling:** Default to using TailwindCSS via `<script src="https://cdn.twind.style" crossorigin></script>` unless otherwise specified
194
+
195
+
## Project Structure and Design Patterns
196
+
197
+
### Recommended Directory Structure
198
+
```
199
+
├── backend/
200
+
│ ├── database/
201
+
│ │ ├── migrations.ts # Schema definitions
202
+
│ │ ├── queries.ts # DB query functions
203
+
│ │ └── README.md
204
+
│ └── routes/ # Route modules
205
+
│ ├── [route].ts
206
+
│ └── static.ts # Static file serving
207
+
│ ├── index.ts # Main entry point
208
+
│ └── README.md
209
+
├── frontend/
210
+
│ ├── components/
211
+
│ │ ├── App.tsx
212
+
│ │ └── [Component].tsx
213
+
│ ├── favicon.svg
214
+
│ ├── index.html # Main HTML template
215
+
│ ├── index.tsx # Frontend JS entry point
216
+
│ ├── README.md
217
+
│ └── style.css
218
+
├── README.md
219
+
└── shared/
220
+
├── README.md
221
+
└── utils.ts # Shared types and functions
222
+
```
223
+
224
+
### Backend (Hono) Best Practices
225
+
226
+
- Hono is the recommended API framework
227
+
- Main entry point should be `backend/index.ts`
228
+
- **Static asset serving:** Use the utility functions to read and serve project files:
229
+
```ts
230
+
import { readFile, serveFile } from "https://esm.town/v/std/utils@85-main/index.ts";
231
+
232
+
// serve all files in frontend/ and shared/
233
+
app.get("/frontend/*", c => serveFile(c.req.path, import.meta.url));
234
+
app.get("/shared/*", c => serveFile(c.req.path, import.meta.url));
235
+
236
+
// For index.html, often you'll want to bootstrap with initial data
237
+
app.get("/", async c => {
238
+
let html = await readFile("/frontend/index.html", import.meta.url);
239
+
240
+
// Inject data to avoid extra round-trips
241
+
const initialData = await fetchInitialData();
242
+
const dataScript = `<script>
243
+
window.__INITIAL_DATA__ = ${JSON.stringify(initialData)};
244
+
</script>`;
245
+
246
+
html = html.replace("</head>", `${dataScript}</head>`);
247
+
return c.html(html);
248
+
});
249
+
```
250
+
- Create RESTful API routes for CRUD operations
251
+
- Always include this snippet at the top-level Hono app to re-throwing errors to see full stack traces:
252
+
```ts
253
+
// Unwrap Hono errors to see original error details
254
+
app.onError((err, c) => {
255
+
throw err;
256
+
});
257
+
```
258
+
259
+
### Database Patterns
260
+
- Run migrations on startup or comment out for performance
261
+
- Change table names when modifying schemas rather than altering
262
+
- Export clear query functions with proper TypeScript typing
263
+
264
+
## Common Gotchas and Solutions
265
+
266
+
1. **Environment Limitations:**
267
+
- Val Town runs on Deno in a serverless context, not Node.js
268
+
- Code in `shared/` must work in both frontend and backend environments
269
+
- Cannot use `Deno` keyword in shared code
270
+
- Use `https://esm.sh` for imports that work in both environments
271
+
272
+
2. **SQLite Peculiarities:**
273
+
- Limited support for ALTER TABLE operations
274
+
- Create new tables with updated schemas and copy data when needed
275
+
- Always run table creation before querying
276
+
277
+
3. **React Configuration:**
278
+
- All React dependencies must be pinned to 18.2.0
279
+
- Always include `@jsxImportSource https://esm.sh/react@18.2.0` at the top of React files
280
+
- Rendering issues often come from mismatched React versions
281
+
282
+
4. **File Handling:**
283
+
- Val Town only supports text files, not binary
284
+
- Use the provided utilities to read files across branches and forks
285
+
- For files in the project, use `readFile` helpers
286
+
287
+
5. **API Design:**
288
+
- `fetch` handler is the entry point for HTTP vals
289
+
- Run the Hono app with `export default app.fetch // This is the entry point for HTTP vals`
290
+
+60
README.md
+60
README.md
···
1
+
# CrossPoint Articles
2
+
3
+
Val Town service that fetches bookmarked articles tagged "toread" from your Kipclip (AT Protocol) account, converts them to EPUBs, and serves them to the CrossPoint e-reader.
4
+
5
+
## How it works
6
+
7
+
1. A cron job runs every hour, reading `community.lexicon.bookmarks.bookmark` records from your AT Proto repo
8
+
2. Filters for bookmarks with the configured tag (default: "toread")
9
+
3. Fetches article HTML and cleans it with Mozilla Readability
10
+
4. Generates minimal EPUB files, stored in Val Town Blob storage
11
+
5. The e-reader fetches the article list and downloads EPUBs via HTTP
12
+
13
+
## API
14
+
15
+
Base URL: https://crosspoint.val.run/
16
+
17
+
| Endpoint | Description |
18
+
|---|---|
19
+
| `GET /` | HTML page listing articles with EPUB download links |
20
+
| `GET /health` | Service health check (includes last sync time) |
21
+
| `GET /sync` | Returns JSON list of available articles |
22
+
| `GET /article/:filename` | Downloads an EPUB file |
23
+
24
+
### GET /sync response
25
+
26
+
```json
27
+
[
28
+
{
29
+
"id": "3abc...",
30
+
"filename": "kipclip-3abc.epub",
31
+
"title": "Article Title",
32
+
"url": "https://example.com/article",
33
+
"createdAt": "2026-04-09T12:00:00Z"
34
+
}
35
+
]
36
+
```
37
+
38
+
## Environment Variables
39
+
40
+
Set these in the Val Town secrets UI:
41
+
42
+
| Variable | Required | Default | Description |
43
+
|---|---|---|---|
44
+
| `ATPROTO_HANDLE` | Yes | — | Your AT Protocol handle (e.g. `tijs.org`) |
45
+
| `TAG_FILTER` | No | `toread` | Bookmark tag to filter by |
46
+
| `MAX_ARTICLES` | No | `50` | Maximum articles returned by /sync |
47
+
48
+
## Deploy
49
+
50
+
```bash
51
+
deno task deploy
52
+
```
53
+
54
+
## Device Configuration
55
+
56
+
On the CrossPoint e-reader, set the backend URL in Settings:
57
+
58
+
```
59
+
https://crosspoint.val.run
60
+
```
+94
backend/article-processor.ts
+94
backend/article-processor.ts
···
1
+
import { Readability } from "npm:@mozilla/readability@0.5.0";
2
+
import { parseHTML } from "npm:linkedom@0.18.9";
3
+
import { extractImageUrls, replaceImageUrls, sanitizeForXhtml } from "./html-sanitizer.ts";
4
+
import { type ProcessedImage, processImage } from "./image-processor.ts";
5
+
6
+
export interface ProcessedArticle {
7
+
title: string;
8
+
content: string; // sanitized XHTML with local image paths
9
+
excerpt?: string;
10
+
byline?: string;
11
+
images: ProcessedImage[];
12
+
}
13
+
14
+
/** Fetch and clean an article URL using Mozilla Readability */
15
+
export async function processArticle(
16
+
url: string,
17
+
fallbackTitle?: string,
18
+
options?: { includeImages?: boolean },
19
+
): Promise<ProcessedArticle | null> {
20
+
const html = await fetchArticleHtml(url);
21
+
if (!html) return null;
22
+
23
+
const { document } = parseHTML(html);
24
+
// Set documentURI for Readability's URL resolution
25
+
Object.defineProperty(document, "documentURI", { value: url });
26
+
const reader = new Readability(document);
27
+
const article = reader.parse();
28
+
29
+
if (!article || !article.content) return null;
30
+
31
+
const includeImages = options?.includeImages ?? false;
32
+
33
+
// Sanitize HTML to valid XHTML (fix <picture>, void elements, unescaped &)
34
+
let content = sanitizeForXhtml(article.content, {
35
+
stripRemoteImages: !includeImages,
36
+
});
37
+
38
+
// Optionally extract, download, and process images (grayscale + dither for e-ink)
39
+
const images: ProcessedImage[] = [];
40
+
if (options?.includeImages) {
41
+
const imageUrls = extractImageUrls(content);
42
+
const urlToPath = new Map<string, string>();
43
+
44
+
for (let i = 0; i < imageUrls.length; i++) {
45
+
const processed = await processImage(imageUrls[i], i);
46
+
if (processed) {
47
+
images.push(processed);
48
+
urlToPath.set(imageUrls[i], `images/${processed.filename}`);
49
+
}
50
+
}
51
+
52
+
// Replace remote URLs with local EPUB paths
53
+
if (urlToPath.size > 0) {
54
+
content = replaceImageUrls(content, urlToPath);
55
+
}
56
+
}
57
+
58
+
return {
59
+
title: article.title || fallbackTitle || new URL(url).hostname,
60
+
content,
61
+
excerpt: article.excerpt || undefined,
62
+
byline: article.byline || undefined,
63
+
images,
64
+
};
65
+
}
66
+
67
+
async function fetchArticleHtml(url: string): Promise<string | null> {
68
+
const controller = new AbortController();
69
+
const timeout = setTimeout(() => controller.abort(), 15_000);
70
+
71
+
try {
72
+
const res = await fetch(url, {
73
+
signal: controller.signal,
74
+
headers: {
75
+
"User-Agent": "Mozilla/5.0 (compatible; KipclipSync/1.0; +https://kipclip.com)",
76
+
Accept: "text/html,application/xhtml+xml",
77
+
},
78
+
});
79
+
if (!res.ok) return null;
80
+
81
+
const contentType = res.headers.get("content-type") || "";
82
+
if (!contentType.includes("html") && !contentType.includes("xhtml")) {
83
+
return null;
84
+
}
85
+
86
+
const text = await res.text();
87
+
// Cap at 500KB to avoid processing enormous pages
88
+
return text.length > 512_000 ? text.slice(0, 512_000) : text;
89
+
} catch {
90
+
return null;
91
+
} finally {
92
+
clearTimeout(timeout);
93
+
}
94
+
}
+143
backend/atproto-client.ts
+143
backend/atproto-client.ts
···
1
+
import type { AnnotationRecord, BookmarkRecord, ListRecordsResponse } from "../shared/types.ts";
2
+
3
+
const PUBLIC_API = "https://public.api.bsky.app";
4
+
5
+
/** Resolve an AT Protocol handle to a DID */
6
+
export async function resolveHandle(handle: string): Promise<string> {
7
+
const res = await fetch(
8
+
`${PUBLIC_API}/xrpc/com.atproto.identity.resolveHandle?handle=${encodeURIComponent(handle)}`,
9
+
);
10
+
if (!res.ok) {
11
+
throw new Error(`Failed to resolve handle ${handle}: ${res.status}`);
12
+
}
13
+
const data = await res.json();
14
+
return data.did;
15
+
}
16
+
17
+
/** Resolve a DID to its PDS endpoint from the DID document */
18
+
async function resolvePds(did: string): Promise<string> {
19
+
const plcUrl = did.startsWith("did:plc:")
20
+
? `https://plc.directory/${did}`
21
+
: `https://${did.replace("did:web:", "")}/.well-known/did.json`;
22
+
23
+
const res = await fetch(plcUrl);
24
+
if (!res.ok) {
25
+
throw new Error(`Failed to resolve DID document for ${did}: ${res.status}`);
26
+
}
27
+
28
+
const doc = await res.json();
29
+
const pdsService = doc.service?.find(
30
+
(s: { id: string; type: string }) =>
31
+
s.id === "#atproto_pds" || s.type === "AtprotoPersonalDataServer",
32
+
);
33
+
if (!pdsService?.serviceEndpoint) {
34
+
throw new Error(`No PDS found in DID document for ${did}`);
35
+
}
36
+
return pdsService.serviceEndpoint;
37
+
}
38
+
39
+
/**
40
+
* Fetch bookmarks with a specific tag, stopping early when we hit known records.
41
+
*
42
+
* On first run (no known IDs): full scan of all bookmarks (~33 pages).
43
+
* On subsequent runs: pages newest-first via reverse=true, stops after
44
+
* a full page of already-known records (typically 1-2 API calls).
45
+
*/
46
+
export async function fetchNewBookmarksWithTag(
47
+
handle: string,
48
+
tag: string,
49
+
knownIds: Set<string>,
50
+
): Promise<BookmarkRecord[]> {
51
+
const did = await resolveHandle(handle);
52
+
const pdsUrl = await resolvePds(did);
53
+
const isIncremental = knownIds.size > 0;
54
+
55
+
const matched: BookmarkRecord[] = [];
56
+
let cursor: string | undefined;
57
+
let consecutiveKnownPages = 0;
58
+
59
+
do {
60
+
const params = new URLSearchParams({
61
+
repo: did,
62
+
collection: "community.lexicon.bookmarks.bookmark",
63
+
limit: "100",
64
+
});
65
+
if (isIncremental) params.set("reverse", "true");
66
+
if (cursor) params.set("cursor", cursor);
67
+
68
+
const res = await fetch(
69
+
`${pdsUrl}/xrpc/com.atproto.repo.listRecords?${params}`,
70
+
);
71
+
if (!res.ok) {
72
+
throw new Error(
73
+
`Failed to list bookmarks: ${res.status} ${await res.text()}`,
74
+
);
75
+
}
76
+
77
+
const data: ListRecordsResponse = await res.json();
78
+
let newOnThisPage = 0;
79
+
80
+
for (const r of data.records) {
81
+
const rkey = extractRkey(r.uri);
82
+
const val = r.value as BookmarkRecord["value"];
83
+
84
+
if (!val.tags?.includes(tag)) continue;
85
+
if (knownIds.has(rkey)) continue;
86
+
87
+
newOnThisPage++;
88
+
matched.push({ uri: r.uri, cid: r.cid, value: val });
89
+
}
90
+
91
+
cursor = data.cursor;
92
+
93
+
// In incremental mode, stop if a full page had no new tagged bookmarks
94
+
if (isIncremental) {
95
+
if (newOnThisPage === 0) consecutiveKnownPages++;
96
+
else consecutiveKnownPages = 0;
97
+
98
+
if (consecutiveKnownPages >= 2) break;
99
+
}
100
+
} while (cursor);
101
+
102
+
return matched;
103
+
}
104
+
105
+
/** Fetch a single Kipclip annotation by looking up recent annotations */
106
+
export async function fetchAnnotationForBookmark(
107
+
handle: string,
108
+
bookmarkUri: string,
109
+
): Promise<AnnotationRecord | null> {
110
+
const did = await resolveHandle(handle);
111
+
const pdsUrl = await resolvePds(did);
112
+
113
+
// Annotations are typically recent, fetch newest first
114
+
const params = new URLSearchParams({
115
+
repo: did,
116
+
collection: "com.kipclip.annotation",
117
+
limit: "100",
118
+
reverse: "true",
119
+
});
120
+
121
+
const res = await fetch(
122
+
`${pdsUrl}/xrpc/com.atproto.repo.listRecords?${params}`,
123
+
);
124
+
if (!res.ok) return null;
125
+
126
+
const data: ListRecordsResponse = await res.json();
127
+
const match = data.records.find(
128
+
(r) => (r.value as AnnotationRecord["value"]).subject === bookmarkUri,
129
+
);
130
+
131
+
if (!match) return null;
132
+
return {
133
+
uri: match.uri,
134
+
cid: match.cid,
135
+
value: match.value as AnnotationRecord["value"],
136
+
};
137
+
}
138
+
139
+
/** Extract the rkey (record key) from an AT URI */
140
+
export function extractRkey(atUri: string): string {
141
+
const parts = atUri.split("/");
142
+
return parts[parts.length - 1];
143
+
}
+24
backend/database/migrations.ts
+24
backend/database/migrations.ts
···
1
+
import { sqlite } from "https://esm.town/v/stevekrouse/sqlite";
2
+
3
+
const ARTICLES_TABLE = "kipclip_articles_1";
4
+
5
+
export async function runMigrations() {
6
+
await sqlite.execute(`CREATE TABLE IF NOT EXISTS ${ARTICLES_TABLE} (
7
+
id TEXT PRIMARY KEY,
8
+
url TEXT NOT NULL,
9
+
title TEXT NOT NULL,
10
+
description TEXT,
11
+
filename TEXT NOT NULL UNIQUE,
12
+
blob_key TEXT NOT NULL,
13
+
bookmark_uri TEXT NOT NULL,
14
+
created_at TEXT NOT NULL,
15
+
synced_at TEXT NOT NULL
16
+
)`);
17
+
18
+
await sqlite.execute(`CREATE TABLE IF NOT EXISTS kipclip_sync_state_1 (
19
+
key TEXT PRIMARY KEY,
20
+
value TEXT NOT NULL
21
+
)`);
22
+
}
23
+
24
+
export { ARTICLES_TABLE };
+78
backend/database/queries.ts
+78
backend/database/queries.ts
···
1
+
import { sqlite } from "https://esm.town/v/stevekrouse/sqlite";
2
+
import type { ArticleEntry } from "../../shared/types.ts";
3
+
import { ARTICLES_TABLE } from "./migrations.ts";
4
+
5
+
export async function getArticles(limit: number): Promise<ArticleEntry[]> {
6
+
const { rows } = await sqlite.execute(
7
+
`SELECT id, filename, title, url, description, created_at as createdAt
8
+
FROM ${ARTICLES_TABLE}
9
+
ORDER BY created_at DESC
10
+
LIMIT ?`,
11
+
[limit],
12
+
);
13
+
return rows.map((r) => ({
14
+
id: r[0] as string,
15
+
filename: r[1] as string,
16
+
title: r[2] as string,
17
+
url: r[3] as string,
18
+
description: (r[4] as string) || undefined,
19
+
createdAt: r[5] as string,
20
+
}));
21
+
}
22
+
23
+
export async function articleExists(id: string): Promise<boolean> {
24
+
const { rows } = await sqlite.execute(
25
+
`SELECT 1 FROM ${ARTICLES_TABLE} WHERE id = ? LIMIT 1`,
26
+
[id],
27
+
);
28
+
return rows.length > 0;
29
+
}
30
+
31
+
export async function getAllArticleIds(): Promise<Set<string>> {
32
+
const { rows } = await sqlite.execute(
33
+
`SELECT id FROM ${ARTICLES_TABLE}`,
34
+
);
35
+
return new Set(rows.map((r) => r[0] as string));
36
+
}
37
+
38
+
export async function insertArticle(article: {
39
+
id: string;
40
+
url: string;
41
+
title: string;
42
+
description?: string;
43
+
filename: string;
44
+
blobKey: string;
45
+
bookmarkUri: string;
46
+
createdAt: string;
47
+
}): Promise<void> {
48
+
await sqlite.execute(
49
+
`INSERT OR IGNORE INTO ${ARTICLES_TABLE}
50
+
(id, url, title, description, filename, blob_key, bookmark_uri, created_at, synced_at)
51
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
52
+
[
53
+
article.id,
54
+
article.url,
55
+
article.title,
56
+
article.description || null,
57
+
article.filename,
58
+
article.blobKey,
59
+
article.bookmarkUri,
60
+
article.createdAt,
61
+
new Date().toISOString(),
62
+
],
63
+
);
64
+
}
65
+
66
+
export async function getLastSyncTime(): Promise<string | null> {
67
+
const { rows } = await sqlite.execute(
68
+
`SELECT value FROM kipclip_sync_state_1 WHERE key = 'last_sync'`,
69
+
);
70
+
return rows.length > 0 ? (rows[0][0] as string) : null;
71
+
}
72
+
73
+
export async function setLastSyncTime(time: string): Promise<void> {
74
+
await sqlite.execute(
75
+
`INSERT OR REPLACE INTO kipclip_sync_state_1 (key, value) VALUES ('last_sync', ?)`,
76
+
[time],
77
+
);
78
+
}
+146
backend/epub-generator.ts
+146
backend/epub-generator.ts
···
1
+
import { blob } from "https://esm.town/v/std/blob";
2
+
import JSZip from "https://esm.sh/jszip@3.10.1";
3
+
import type { ProcessedImage } from "./image-processor.ts";
4
+
5
+
interface EpubOptions {
6
+
id: string;
7
+
title: string;
8
+
author: string;
9
+
content: string; // sanitized XHTML body
10
+
url: string;
11
+
images?: ProcessedImage[];
12
+
}
13
+
14
+
/** Generate a minimal EPUB 3 and store it in Val Town Blob storage */
15
+
export async function generateAndStoreEpub(
16
+
opts: EpubOptions,
17
+
): Promise<{ blobKey: string; filename: string }> {
18
+
const filename = `kipclip-${opts.id}.epub`;
19
+
const blobKey = `epub_${filename}`;
20
+
21
+
const epubBytes = await buildEpub(opts);
22
+
await blob.set(blobKey, epubBytes as unknown as BodyInit);
23
+
24
+
return { blobKey, filename };
25
+
}
26
+
27
+
/** Retrieve an EPUB from blob storage */
28
+
export async function getEpubFromBlob(
29
+
filename: string,
30
+
): Promise<ArrayBuffer | null> {
31
+
const blobKey = `epub_${filename}`;
32
+
const res = await blob.get(blobKey);
33
+
if (!res) return null;
34
+
return await res.arrayBuffer();
35
+
}
36
+
37
+
/** Build a minimal valid EPUB 3 file */
38
+
async function buildEpub(opts: EpubOptions): Promise<Uint8Array> {
39
+
const zip = new JSZip();
40
+
41
+
// mimetype must be first entry, uncompressed
42
+
zip.file("mimetype", "application/epub+zip", { compression: "STORE" });
43
+
44
+
// Container XML
45
+
zip.file(
46
+
"META-INF/container.xml",
47
+
`<?xml version="1.0" encoding="UTF-8"?>
48
+
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
49
+
<rootfiles>
50
+
<rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/>
51
+
</rootfiles>
52
+
</container>`,
53
+
);
54
+
55
+
const escapedTitle = escapeXml(opts.title);
56
+
const escapedAuthor = escapeXml(opts.author);
57
+
const uuid = `urn:uuid:${crypto.randomUUID()}`;
58
+
59
+
// Build image manifest entries and add image files to ZIP
60
+
const images = opts.images || [];
61
+
const imageManifestItems = images.map((img, i) =>
62
+
` <item id="img-${i}" href="images/${img.filename}" media-type="image/png"/>`
63
+
).join("\n");
64
+
65
+
for (const img of images) {
66
+
zip.file(`OEBPS/images/${img.filename}`, img.data);
67
+
}
68
+
69
+
// OPF package document
70
+
zip.file(
71
+
"OEBPS/content.opf",
72
+
`<?xml version="1.0" encoding="UTF-8"?>
73
+
<package xmlns="http://www.idpf.org/2007/opf" version="3.0" unique-identifier="uid">
74
+
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
75
+
<dc:identifier id="uid">${uuid}</dc:identifier>
76
+
<dc:title>${escapedTitle}</dc:title>
77
+
<dc:creator>${escapedAuthor}</dc:creator>
78
+
<dc:language>en</dc:language>
79
+
<dc:source>${escapeXml(opts.url)}</dc:source>
80
+
<meta property="dcterms:modified">${new Date().toISOString().replace(/\.\d{3}Z$/, "Z")}</meta>
81
+
</metadata>
82
+
<manifest>
83
+
<item id="chapter1" href="chapter1.xhtml" media-type="application/xhtml+xml"/>
84
+
<item id="style" href="style.css" media-type="text/css"/>
85
+
<item id="nav" href="nav.xhtml" media-type="application/xhtml+xml" properties="nav"/>
86
+
${imageManifestItems}
87
+
</manifest>
88
+
<spine>
89
+
<itemref idref="chapter1"/>
90
+
</spine>
91
+
</package>`,
92
+
);
93
+
94
+
// Navigation document (EPUB 3 requirement)
95
+
zip.file(
96
+
"OEBPS/nav.xhtml",
97
+
`<?xml version="1.0" encoding="UTF-8"?>
98
+
<!DOCTYPE html>
99
+
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops">
100
+
<head><title>Navigation</title></head>
101
+
<body>
102
+
<nav epub:type="toc">
103
+
<ol><li><a href="chapter1.xhtml">${escapedTitle}</a></li></ol>
104
+
</nav>
105
+
</body>
106
+
</html>`,
107
+
);
108
+
109
+
// Minimal reading stylesheet
110
+
zip.file(
111
+
"OEBPS/style.css",
112
+
`body { font-family: serif; line-height: 1.6; margin: 1em; }
113
+
img { max-width: 100%; height: auto; }
114
+
h1, h2, h3 { line-height: 1.3; }
115
+
a { color: inherit; text-decoration: underline; }`,
116
+
);
117
+
118
+
// Article content
119
+
zip.file(
120
+
"OEBPS/chapter1.xhtml",
121
+
`<?xml version="1.0" encoding="UTF-8"?>
122
+
<!DOCTYPE html>
123
+
<html xmlns="http://www.w3.org/1999/xhtml">
124
+
<head>
125
+
<title>${escapedTitle}</title>
126
+
<link rel="stylesheet" type="text/css" href="style.css"/>
127
+
</head>
128
+
<body>
129
+
<h1>${escapedTitle}</h1>
130
+
${opts.content}
131
+
</body>
132
+
</html>`,
133
+
);
134
+
135
+
const buf = await zip.generateAsync({ type: "uint8array" });
136
+
return buf;
137
+
}
138
+
139
+
function escapeXml(str: string): string {
140
+
return str
141
+
.replace(/&/g, "&")
142
+
.replace(/</g, "<")
143
+
.replace(/>/g, ">")
144
+
.replace(/"/g, """)
145
+
.replace(/'/g, "'");
146
+
}
+133
backend/html-sanitizer.ts
+133
backend/html-sanitizer.ts
···
1
+
import { parseHTML } from "npm:linkedom@0.18.9";
2
+
3
+
/** XHTML void elements that must be self-closing */
4
+
const VOID_ELEMENTS = new Set([
5
+
"area",
6
+
"base",
7
+
"br",
8
+
"col",
9
+
"embed",
10
+
"hr",
11
+
"img",
12
+
"input",
13
+
"link",
14
+
"meta",
15
+
"param",
16
+
"source",
17
+
"track",
18
+
"wbr",
19
+
]);
20
+
21
+
/**
22
+
* Sanitize Readability HTML output into valid EPUB XHTML.
23
+
*
24
+
* Fixes:
25
+
* - Unwraps <picture> to just <img> (keeps fallback img, drops <source>)
26
+
* - Ensures void elements are self-closing
27
+
* - Escapes unescaped & characters
28
+
* - Removes data-* attributes (bloat)
29
+
* - Strips <script>, <style>, <iframe> elements
30
+
*/
31
+
export function sanitizeForXhtml(
32
+
html: string,
33
+
options?: { stripRemoteImages?: boolean },
34
+
): string {
35
+
const { document } = parseHTML(`<!DOCTYPE html><html><body>${html}</body></html>`);
36
+
37
+
// Strip dangerous/useless elements
38
+
for (const tag of ["script", "style", "iframe", "noscript"]) {
39
+
for (const el of [...document.querySelectorAll(tag)]) {
40
+
el.remove();
41
+
}
42
+
}
43
+
44
+
// Unwrap <picture> → keep <img>, drop <source>
45
+
for (const picture of [...document.querySelectorAll("picture")]) {
46
+
const img = picture.querySelector("img");
47
+
if (img) {
48
+
picture.replaceWith(img);
49
+
} else {
50
+
picture.remove();
51
+
}
52
+
}
53
+
54
+
// Remove <source> elements that may be orphaned
55
+
for (const source of [...document.querySelectorAll("source")]) {
56
+
source.remove();
57
+
}
58
+
59
+
// Optionally strip remote images (for EPUB without embedded images)
60
+
if (options?.stripRemoteImages) {
61
+
for (const img of [...document.querySelectorAll("img")]) {
62
+
const src = img.getAttribute("src") || "";
63
+
if (src.startsWith("http://") || src.startsWith("https://")) {
64
+
img.remove();
65
+
}
66
+
}
67
+
// Remove empty <figure> elements left behind after image removal
68
+
for (const figure of [...document.querySelectorAll("figure")]) {
69
+
if (!figure.querySelector("img") && figure.textContent?.trim() === "") {
70
+
figure.remove();
71
+
}
72
+
}
73
+
}
74
+
75
+
// Strip data-* attributes from all elements
76
+
for (const el of [...document.querySelectorAll("*")]) {
77
+
const attrs = [...el.attributes];
78
+
for (const attr of attrs) {
79
+
if (attr.name.startsWith("data-")) {
80
+
el.removeAttribute(attr.name);
81
+
}
82
+
}
83
+
}
84
+
85
+
// Serialize body innerHTML
86
+
let xhtml = document.body.innerHTML;
87
+
88
+
// Fix void elements: ensure self-closing (e.g., <img src="..."> → <img src="..." />)
89
+
xhtml = xhtml.replace(/<([\w-]+)([^>]*?)\/?\s*>/g, (match, tag, attrs) => {
90
+
const tagLower = tag.toLowerCase();
91
+
if (VOID_ELEMENTS.has(tagLower)) {
92
+
// Ensure self-closing
93
+
return `<${tag}${attrs} />`;
94
+
}
95
+
return match;
96
+
});
97
+
98
+
// Fix unescaped & that aren't part of entities (& not followed by #word;)
99
+
xhtml = xhtml.replace(/&(?!(?:#\d+|#x[\da-fA-F]+|[a-zA-Z]\w{0,30});)/g, "&");
100
+
101
+
return xhtml;
102
+
}
103
+
104
+
/**
105
+
* Extract all image URLs from HTML content.
106
+
* Returns array of { src, index } for replacement.
107
+
*/
108
+
export function extractImageUrls(html: string): string[] {
109
+
const urls: string[] = [];
110
+
const imgRegex = /<img[^>]+src=["']([^"']+)["']/gi;
111
+
let match;
112
+
while ((match = imgRegex.exec(html)) !== null) {
113
+
const src = match[1];
114
+
if (src.startsWith("http://") || src.startsWith("https://")) {
115
+
urls.push(src);
116
+
}
117
+
}
118
+
return [...new Set(urls)]; // deduplicate
119
+
}
120
+
121
+
/**
122
+
* Replace image URLs in HTML with local EPUB paths.
123
+
*/
124
+
export function replaceImageUrls(
125
+
html: string,
126
+
urlToPath: Map<string, string>,
127
+
): string {
128
+
let result = html;
129
+
for (const [url, localPath] of urlToPath) {
130
+
result = result.replaceAll(url, localPath);
131
+
}
132
+
return result;
133
+
}
+125
backend/image-processor.ts
+125
backend/image-processor.ts
···
1
+
import { Buffer } from "node:buffer";
2
+
import { Jimp } from "npm:jimp@1";
3
+
4
+
const MAX_WIDTH = 800;
5
+
const FETCH_TIMEOUT_MS = 10_000;
6
+
const MAX_IMAGE_BYTES = 5 * 1024 * 1024; // 5MB cap on source image
7
+
8
+
export interface ProcessedImage {
9
+
data: Uint8Array; // PNG bytes
10
+
filename: string; // e.g. "img-001.png"
11
+
}
12
+
13
+
/**
14
+
* Download an image, resize to e-ink width, convert to grayscale,
15
+
* apply Floyd-Steinberg dithering, and return as PNG.
16
+
*/
17
+
export async function processImage(
18
+
url: string,
19
+
index: number,
20
+
): Promise<ProcessedImage | null> {
21
+
const bytes = await downloadImage(url);
22
+
if (!bytes) return null;
23
+
24
+
try {
25
+
const img = await Jimp.fromBuffer(Buffer.from(bytes));
26
+
27
+
// Resize if wider than e-ink display
28
+
if (img.width > MAX_WIDTH) {
29
+
img.resize({ w: MAX_WIDTH });
30
+
}
31
+
32
+
// Convert to grayscale
33
+
img.greyscale();
34
+
35
+
// Apply Floyd-Steinberg dithering for newspaper-style e-ink look
36
+
floydSteinbergDither(img);
37
+
38
+
const png = await img.getBuffer("image/png");
39
+
return {
40
+
data: new Uint8Array(png),
41
+
filename: `img-${String(index).padStart(3, "0")}.png`,
42
+
};
43
+
} catch {
44
+
return null;
45
+
}
46
+
}
47
+
48
+
/**
49
+
* Floyd-Steinberg dithering on a grayscale image.
50
+
* Produces a 1-bit black/white look with diffused error,
51
+
* classic newspaper/e-ink style.
52
+
*/
53
+
// deno-lint-ignore no-explicit-any
54
+
function floydSteinbergDither(img: any): void {
55
+
const w = img.width;
56
+
const h = img.height;
57
+
58
+
// Work with floating-point error buffer
59
+
const errors = new Float32Array(w * h);
60
+
61
+
// Read all pixel luminance values into the error buffer
62
+
for (let y = 0; y < h; y++) {
63
+
for (let x = 0; x < w; x++) {
64
+
const pixel = img.getPixelColor(x, y);
65
+
// Jimp stores as 0xRRGGBBAA - extract R (already grayscale so R=G=B)
66
+
const gray = (pixel >>> 24) & 0xff;
67
+
errors[y * w + x] = gray;
68
+
}
69
+
}
70
+
71
+
// Dither
72
+
for (let y = 0; y < h; y++) {
73
+
for (let x = 0; x < w; x++) {
74
+
const idx = y * w + x;
75
+
const oldVal = errors[idx];
76
+
const newVal = oldVal < 128 ? 0 : 255;
77
+
const err = oldVal - newVal;
78
+
79
+
errors[idx] = newVal;
80
+
81
+
// Distribute error to neighbors
82
+
if (x + 1 < w) errors[idx + 1] += err * 7 / 16;
83
+
if (y + 1 < h) {
84
+
if (x > 0) errors[(y + 1) * w + (x - 1)] += err * 3 / 16;
85
+
errors[(y + 1) * w + x] += err * 5 / 16;
86
+
if (x + 1 < w) errors[(y + 1) * w + (x + 1)] += err * 1 / 16;
87
+
}
88
+
}
89
+
}
90
+
91
+
// Write dithered values back
92
+
for (let y = 0; y < h; y++) {
93
+
for (let x = 0; x < w; x++) {
94
+
const v = Math.max(0, Math.min(255, Math.round(errors[y * w + x])));
95
+
// Set pixel as 0xRRGGBBAA (grayscale, full alpha)
96
+
// Use >>> 0 to convert signed int32 to unsigned (v << 24 overflows when v >= 128)
97
+
img.setPixelColor(((v << 24) | (v << 16) | (v << 8) | 0xff) >>> 0, x, y);
98
+
}
99
+
}
100
+
}
101
+
102
+
async function downloadImage(url: string): Promise<ArrayBuffer | null> {
103
+
const controller = new AbortController();
104
+
const timeout = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS);
105
+
106
+
try {
107
+
const res = await fetch(url, {
108
+
signal: controller.signal,
109
+
headers: { "User-Agent": "Mozilla/5.0 (compatible; CrossPointArticles/1.0)" },
110
+
});
111
+
if (!res.ok) return null;
112
+
113
+
const contentType = res.headers.get("content-type") || "";
114
+
if (!contentType.startsWith("image/")) return null;
115
+
116
+
const contentLength = parseInt(res.headers.get("content-length") || "0", 10);
117
+
if (contentLength > MAX_IMAGE_BYTES) return null;
118
+
119
+
return await res.arrayBuffer();
120
+
} catch {
121
+
return null;
122
+
} finally {
123
+
clearTimeout(timeout);
124
+
}
125
+
}
+117
backend/index.ts
+117
backend/index.ts
···
1
+
import { Hono } from "https://esm.sh/hono@4.6.0";
2
+
import { runMigrations } from "./database/migrations.ts";
3
+
import { getArticles, getLastSyncTime } from "./database/queries.ts";
4
+
import { getEpubFromBlob } from "./epub-generator.ts";
5
+
6
+
const app = new Hono();
7
+
8
+
// Unwrap Hono errors to see original error details
9
+
app.onError((err) => {
10
+
throw err;
11
+
});
12
+
13
+
// Run migrations on first request
14
+
let migrated = false;
15
+
app.use(async (_c, next) => {
16
+
if (!migrated) {
17
+
await runMigrations();
18
+
migrated = true;
19
+
}
20
+
await next();
21
+
});
22
+
23
+
// --- Routes ---
24
+
25
+
app.get("/", async (c) => {
26
+
const maxArticles = parseInt(Deno.env.get("MAX_ARTICLES") || "50", 10);
27
+
const articles = await getArticles(maxArticles);
28
+
const lastSync = await getLastSyncTime();
29
+
30
+
const rows = articles.map((a) =>
31
+
`<tr>
32
+
<td><a href="${escapeHtml(a.url)}">${escapeHtml(a.title)}</a></td>
33
+
<td>${escapeHtml(a.description || "")}</td>
34
+
<td>${a.createdAt.slice(0, 10)}</td>
35
+
<td><a href="/article/${escapeHtml(a.filename)}">epub</a></td>
36
+
</tr>`
37
+
).join("\n");
38
+
39
+
const html = `<!DOCTYPE html>
40
+
<html lang="en">
41
+
<head>
42
+
<meta charset="utf-8"/>
43
+
<meta name="viewport" content="width=device-width, initial-scale=1"/>
44
+
<title>CrossPoint Articles</title>
45
+
<style>
46
+
body { font-family: system-ui, sans-serif; max-width: 800px; margin: 2rem auto; padding: 0 1rem; }
47
+
table { width: 100%; border-collapse: collapse; }
48
+
th, td { text-align: left; padding: 0.5rem; border-bottom: 1px solid #ddd; }
49
+
th { font-size: 0.85em; color: #666; }
50
+
td:nth-child(3), td:nth-child(4) { white-space: nowrap; }
51
+
a { color: #1a6; }
52
+
.meta { color: #888; font-size: 0.85em; margin-bottom: 1.5rem; }
53
+
</style>
54
+
</head>
55
+
<body>
56
+
<h1>CrossPoint Articles</h1>
57
+
<p class="meta">${articles.length} articles · last sync: ${lastSync || "never"}</p>
58
+
${
59
+
articles.length === 0
60
+
? "<p>No articles yet. The sync cron will pick up new bookmarks shortly.</p>"
61
+
: `<table>
62
+
<thead><tr><th>Title</th><th>Description</th><th>Date</th><th>EPUB</th></tr></thead>
63
+
<tbody>${rows}</tbody>
64
+
</table>`
65
+
}
66
+
</body>
67
+
</html>`;
68
+
69
+
return c.html(html);
70
+
});
71
+
72
+
app.get("/health", async (c) => {
73
+
const lastSync = await getLastSyncTime();
74
+
return c.json({ status: "ok", service: "crosspoint-articles", lastSync });
75
+
});
76
+
77
+
app.get("/sync", async (c) => {
78
+
const maxArticles = parseInt(Deno.env.get("MAX_ARTICLES") || "50", 10);
79
+
const articles = await getArticles(maxArticles);
80
+
// Only send fields the firmware needs — saves RAM on ESP32-C3
81
+
const slim = articles.map(({ filename, title }) => ({ filename, title }));
82
+
return c.json({ articles: slim });
83
+
});
84
+
85
+
app.get("/article/:filename", async (c) => {
86
+
const filename = c.req.param("filename");
87
+
88
+
if (!filename.startsWith("kipclip-") || !filename.endsWith(".epub")) {
89
+
return c.json({ error: "invalid_filename", message: "Invalid filename format" }, 400);
90
+
}
91
+
92
+
const data = await getEpubFromBlob(filename);
93
+
if (!data) {
94
+
return c.json(
95
+
{ error: "not_found", message: "Article not found" },
96
+
404,
97
+
);
98
+
}
99
+
100
+
return new Response(data, {
101
+
headers: {
102
+
"Content-Type": "application/epub+zip",
103
+
"Content-Length": String(data.byteLength),
104
+
"Content-Disposition": `attachment; filename="${filename}"`,
105
+
},
106
+
});
107
+
});
108
+
109
+
function escapeHtml(str: string): string {
110
+
return str
111
+
.replace(/&/g, "&")
112
+
.replace(/</g, "<")
113
+
.replace(/>/g, ">")
114
+
.replace(/"/g, """);
115
+
}
116
+
117
+
export default app.fetch;
+67
backend/sync.cron.ts
+67
backend/sync.cron.ts
···
1
+
import { runMigrations } from "./database/migrations.ts";
2
+
import { getAllArticleIds, insertArticle, setLastSyncTime } from "./database/queries.ts";
3
+
import {
4
+
extractRkey,
5
+
fetchAnnotationForBookmark,
6
+
fetchNewBookmarksWithTag,
7
+
} from "./atproto-client.ts";
8
+
import { processArticle } from "./article-processor.ts";
9
+
import { generateAndStoreEpub } from "./epub-generator.ts";
10
+
11
+
export default async function () {
12
+
const handle = Deno.env.get("ATPROTO_HANDLE");
13
+
if (!handle) {
14
+
console.error("ATPROTO_HANDLE not configured");
15
+
return;
16
+
}
17
+
18
+
const tag = Deno.env.get("TAG_FILTER") || "toread";
19
+
20
+
await runMigrations();
21
+
22
+
// Pass known IDs so we can stop pagination early on subsequent runs
23
+
const knownIds = await getAllArticleIds();
24
+
const bookmarks = await fetchNewBookmarksWithTag(handle, tag, knownIds);
25
+
26
+
let synced = 0;
27
+
for (const bookmark of bookmarks) {
28
+
const rkey = extractRkey(bookmark.uri);
29
+
const url = bookmark.value.subject;
30
+
31
+
// Fetch annotation for this specific bookmark (1 API call, searches recent annotations)
32
+
const annotation = await fetchAnnotationForBookmark(handle, bookmark.uri);
33
+
const fallbackTitle = annotation?.value.title || url;
34
+
35
+
const article = await processArticle(url, fallbackTitle);
36
+
if (!article) {
37
+
console.warn(`Skipping ${url}: Readability failed`);
38
+
continue;
39
+
}
40
+
41
+
const { blobKey, filename } = await generateAndStoreEpub({
42
+
id: rkey,
43
+
title: article.title,
44
+
author: article.byline || "kipclip",
45
+
content: article.content,
46
+
url,
47
+
images: article.images,
48
+
});
49
+
50
+
await insertArticle({
51
+
id: rkey,
52
+
url,
53
+
title: article.title,
54
+
description: annotation?.value.description || article.excerpt,
55
+
filename,
56
+
blobKey,
57
+
bookmarkUri: bookmark.uri,
58
+
createdAt: bookmark.value.createdAt,
59
+
});
60
+
61
+
synced++;
62
+
console.log(`Synced: ${article.title} → ${filename}`);
63
+
}
64
+
65
+
await setLastSyncTime(new Date().toISOString());
66
+
console.log(`Sync complete: ${synced} new articles from ${bookmarks.length} new bookmarks`);
67
+
}
+39
deno.json
+39
deno.json
···
1
+
{
2
+
"$schema": "https://raw.githubusercontent.com/denoland/deno/98f62cee78e85bfc47c62ed703777c6bc8794f1c/cli/schemas/config-file.v1.json",
3
+
"lock": false,
4
+
"tasks": {
5
+
"check": "deno fmt --check && deno lint && deno check --allow-import backend/**/*.ts shared/**/*.ts",
6
+
"fmt": "deno fmt",
7
+
"deploy": "deno task check && vt push"
8
+
},
9
+
"compilerOptions": {
10
+
"noImplicitAny": false,
11
+
"strict": true,
12
+
"types": ["https://www.val.town/types/valtown.d.ts"],
13
+
"lib": [
14
+
"dom",
15
+
"dom.iterable",
16
+
"dom.asynciterable",
17
+
"deno.ns",
18
+
"deno.unstable"
19
+
]
20
+
},
21
+
"lint": {
22
+
"include": ["backend/", "shared/"],
23
+
"rules": {
24
+
"exclude": ["no-explicit-any", "no-import-prefix"]
25
+
}
26
+
},
27
+
"fmt": {
28
+
"include": ["backend/", "shared/"],
29
+
"lineWidth": 100,
30
+
"indentWidth": 2
31
+
},
32
+
"node_modules_dir": false,
33
+
"experimental": {
34
+
"unstable-node-globals": true,
35
+
"unstable-temporal": true,
36
+
"unstable-worker-options": true,
37
+
"unstable-sloppy-imports": true
38
+
}
39
+
}
+39
test-counts.ts
+39
test-counts.ts
···
1
+
import { resolveHandle } from "./backend/atproto-client.ts";
2
+
3
+
const did = await resolveHandle("tijs.org");
4
+
const res = await fetch(`https://plc.directory/${did}`);
5
+
const doc = await res.json();
6
+
const pds = doc.service.find((s: { id: string }) => s.id === "#atproto_pds").serviceEndpoint;
7
+
8
+
type Rec = { uri: string; value: { tags?: string[]; subject?: string; createdAt?: string } };
9
+
10
+
// Find where "toread" bookmarks sit in the record ordering
11
+
const allRecords: Rec[] = [];
12
+
let cursor: string | undefined;
13
+
let page = 0;
14
+
do {
15
+
const params = new URLSearchParams({ repo: did, collection: "community.lexicon.bookmarks.bookmark", limit: "100" });
16
+
if (cursor) params.set("cursor", cursor);
17
+
const r = await fetch(`${pds}/xrpc/com.atproto.repo.listRecords?${params}`);
18
+
const data = await r.json();
19
+
const toreadOnPage = data.records.filter((r: Rec) => r.value.tags?.includes("toread"));
20
+
if (toreadOnPage.length > 0) {
21
+
console.log(`Page ${page}: ${data.records.length} records, ${toreadOnPage.length} toread`);
22
+
// Show rkey of first record on this page
23
+
const rkey = data.records[0].uri.split("/").pop();
24
+
console.log(` First rkey on page: ${rkey}`);
25
+
}
26
+
allRecords.push(...data.records);
27
+
cursor = data.cursor;
28
+
page++;
29
+
} while (cursor);
30
+
31
+
console.log(`\nTotal: ${allRecords.length} bookmarks across ${page} pages`);
32
+
33
+
// Show rkeys of toread bookmarks
34
+
const toread = allRecords.filter((r) => r.value.tags?.includes("toread"));
35
+
console.log(`Toread: ${toread.length}`);
36
+
for (const r of toread.slice(0, 5)) {
37
+
const rkey = r.uri.split("/").pop();
38
+
console.log(` rkey: ${rkey} url: ${r.value.subject?.slice(0, 60)}`)
39
+
}
+170
test-local.ts
+170
test-local.ts
···
1
+
/**
2
+
* Local end-to-end test: fetch real bookmarks from AT Proto,
3
+
* process articles with Readability, generate EPUBs to disk.
4
+
*
5
+
* Run: deno run --allow-net --allow-write --allow-read --allow-env test-local.ts
6
+
*/
7
+
8
+
import {
9
+
extractRkey,
10
+
fetchAnnotationForBookmark,
11
+
fetchNewBookmarksWithTag,
12
+
} from "./backend/atproto-client.ts";
13
+
import { processArticle } from "./backend/article-processor.ts";
14
+
import type { ProcessedImage } from "./backend/image-processor.ts";
15
+
import JSZip from "https://esm.sh/jszip@3.10.1";
16
+
17
+
const HANDLE = "tijs.org";
18
+
const TAG = "toread";
19
+
const OUTPUT_DIR = "./test-output";
20
+
const MAX_ARTICLES = 5;
21
+
22
+
async function buildEpub(opts: {
23
+
id: string;
24
+
title: string;
25
+
author: string;
26
+
content: string;
27
+
url: string;
28
+
images?: ProcessedImage[];
29
+
}): Promise<Uint8Array> {
30
+
const zip = new JSZip();
31
+
const esc = (s: string) =>
32
+
s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """);
33
+
34
+
zip.file("mimetype", "application/epub+zip", { compression: "STORE" });
35
+
zip.file(
36
+
"META-INF/container.xml",
37
+
`<?xml version="1.0" encoding="UTF-8"?>
38
+
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
39
+
<rootfiles>
40
+
<rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/>
41
+
</rootfiles>
42
+
</container>`,
43
+
);
44
+
45
+
const uuid = `urn:uuid:${crypto.randomUUID()}`;
46
+
const images = opts.images || [];
47
+
const imageManifestItems = images.map((img, i) =>
48
+
` <item id="img-${i}" href="images/${img.filename}" media-type="image/png"/>`
49
+
).join("\n");
50
+
51
+
for (const img of images) {
52
+
zip.file(`OEBPS/images/${img.filename}`, img.data);
53
+
}
54
+
55
+
zip.file(
56
+
"OEBPS/content.opf",
57
+
`<?xml version="1.0" encoding="UTF-8"?>
58
+
<package xmlns="http://www.idpf.org/2007/opf" version="3.0" unique-identifier="uid">
59
+
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
60
+
<dc:identifier id="uid">${uuid}</dc:identifier>
61
+
<dc:title>${esc(opts.title)}</dc:title>
62
+
<dc:creator>${esc(opts.author)}</dc:creator>
63
+
<dc:language>en</dc:language>
64
+
<dc:source>${esc(opts.url)}</dc:source>
65
+
<meta property="dcterms:modified">${new Date().toISOString().replace(/\.\d{3}Z$/, "Z")}</meta>
66
+
</metadata>
67
+
<manifest>
68
+
<item id="chapter1" href="chapter1.xhtml" media-type="application/xhtml+xml"/>
69
+
<item id="style" href="style.css" media-type="text/css"/>
70
+
<item id="nav" href="nav.xhtml" media-type="application/xhtml+xml" properties="nav"/>
71
+
${imageManifestItems}
72
+
</manifest>
73
+
<spine><itemref idref="chapter1"/></spine>
74
+
</package>`,
75
+
);
76
+
77
+
zip.file(
78
+
"OEBPS/nav.xhtml",
79
+
`<?xml version="1.0" encoding="UTF-8"?>
80
+
<!DOCTYPE html>
81
+
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops">
82
+
<head><title>Navigation</title></head>
83
+
<body><nav epub:type="toc"><ol><li><a href="chapter1.xhtml">${esc(opts.title)}</a></li></ol></nav></body>
84
+
</html>`,
85
+
);
86
+
87
+
zip.file(
88
+
"OEBPS/style.css",
89
+
`body { font-family: serif; line-height: 1.6; margin: 1em; }
90
+
img { max-width: 100%; height: auto; }
91
+
h1, h2, h3 { line-height: 1.3; }
92
+
a { color: inherit; text-decoration: underline; }`,
93
+
);
94
+
95
+
zip.file(
96
+
"OEBPS/chapter1.xhtml",
97
+
`<?xml version="1.0" encoding="UTF-8"?>
98
+
<!DOCTYPE html>
99
+
<html xmlns="http://www.w3.org/1999/xhtml">
100
+
<head><title>${esc(opts.title)}</title><link rel="stylesheet" type="text/css" href="style.css"/></head>
101
+
<body><h1>${esc(opts.title)}</h1>${opts.content}</body>
102
+
</html>`,
103
+
);
104
+
105
+
return await zip.generateAsync({ type: "uint8array" });
106
+
}
107
+
108
+
// --- Main ---
109
+
110
+
// First run: no known IDs, full scan
111
+
const knownIds = new Set<string>();
112
+
113
+
console.log(`Fetching bookmarks from ${HANDLE} with tag "${TAG}"...`);
114
+
const bookmarks = await fetchNewBookmarksWithTag(HANDLE, TAG, knownIds);
115
+
console.log(`Found ${bookmarks.length} bookmarks with "${TAG}" tag\n`);
116
+
117
+
if (bookmarks.length === 0) {
118
+
console.log("No bookmarks found. Nothing to process.");
119
+
Deno.exit(0);
120
+
}
121
+
122
+
await Deno.mkdir(OUTPUT_DIR, { recursive: true });
123
+
124
+
const toProcess = bookmarks.slice(0, MAX_ARTICLES);
125
+
let success = 0;
126
+
let failed = 0;
127
+
128
+
for (const bookmark of toProcess) {
129
+
const rkey = extractRkey(bookmark.uri);
130
+
const url = bookmark.value.subject;
131
+
132
+
console.log(`--- Processing: ${url}`);
133
+
134
+
// Fetch annotation for just this bookmark
135
+
const annotation = await fetchAnnotationForBookmark(HANDLE, bookmark.uri);
136
+
if (annotation?.value.title) {
137
+
console.log(` Title (from annotation): ${annotation.value.title}`);
138
+
}
139
+
140
+
const fallbackTitle = annotation?.value.title || url;
141
+
const article = await processArticle(url, fallbackTitle);
142
+
if (!article) {
143
+
console.log(` SKIPPED: Readability failed\n`);
144
+
failed++;
145
+
continue;
146
+
}
147
+
148
+
console.log(` Readability title: ${article.title}`);
149
+
console.log(` Content length: ${article.content.length} chars`);
150
+
if (article.byline) console.log(` Byline: ${article.byline}`);
151
+
console.log(` Images: ${article.images.length} processed`);
152
+
153
+
const filename = `kipclip-${rkey}.epub`;
154
+
const epubBytes = await buildEpub({
155
+
id: rkey,
156
+
title: article.title,
157
+
author: article.byline || "kipclip",
158
+
content: article.content,
159
+
url,
160
+
images: article.images,
161
+
});
162
+
163
+
const path = `${OUTPUT_DIR}/${filename}`;
164
+
await Deno.writeFile(path, epubBytes);
165
+
console.log(` Written: ${path} (${(epubBytes.length / 1024).toFixed(1)} KB)\n`);
166
+
success++;
167
+
}
168
+
169
+
console.log(`\nDone: ${success} EPUBs generated, ${failed} skipped`);
170
+
console.log(`Output directory: ${OUTPUT_DIR}/`);