Durable Streams over Toon
Incorporate glm.md ideas into state-protocol-toon.opus.md
- Add Key Optimization section in abstract highlighting tabular batching
- Use application/toon+json media type with IANA registration section
- Add content negotiation examples
- Add named tabular arrays (users[], messages[], changes[])
- Add batching strategies (by operation, entity type, transaction, time)
- Add when-to-use guidance for single vs tabular messages
- Add resource limits section (array size, row length, nesting depth)
- Expand Appendix A with field-by-field analysis and real-world bandwidth
- Add Appendix B implementation guidelines (streaming, error handling)
- Add Appendix C migration guide with phased rollout strategy
+4020
README.md
README.md
This is a binary file and will not be displayed.
+1046
doc/discovery/durable-state-protocol.md
+1046
doc/discovery/durable-state-protocol.md
···
1
+
# Durable Streams State Protocol - Deep Dive
2
+
3
+
A comprehensive exploration of Durable Streams State Protocol, the higher-level protocol for database-style synchronization built on Durable Streams.
4
+
5
+
---
6
+
7
+
# journal - Initial Investigation
8
+
9
+
I started by fetching and reviewing the primary source material:
10
+
11
+
1. **Electric SQL Blog Post** - "Durable Streams 0.1.0 and State Protocol" (December 23, 2025)
12
+
- https://electric-sql.com/blog/2025/12/23/durable-streams-0.1.0
13
+
14
+
2. **State Protocol Specification** - Official spec document
15
+
- https://github.com/durable-streams/durable-streams/blob/main/packages/state/STATE-PROTOCOL.md
16
+
17
+
3. **Existing Documentation**:
18
+
- `~/doc/durable-streams-1.md` - Comprehensive Durable Streams protocol guide
19
+
- `~/doc/durable-streams-2.md` - Implementation guide for DUGH-SERVER
20
+
- `~/wiki/durable-streams/` - Directory exists but empty
21
+
22
+
4. **Cloned Reference Implementation**:
23
+
- `.test-agent/durable-streams/` - Complete repository with examples and implementations
24
+
25
+
From these sources, I learned that State Protocol is:
26
+
- An **extension of Durable Streams Protocol** (PROTOCOL)
27
+
- A **composable schema** for state change events
28
+
- Extracted from **Electric SQL's production Postgres sync protocol** (18 months of production use)
29
+
- Designed for **database-style sync** without prescribing storage or querying
30
+
31
+
## Tech Stack Overview
32
+
33
+
| Component | Purpose | Known For |
34
+
|-----------|-----------|-------------|
35
+
| **Durable Streams Protocol** | Base protocol for byte-level streaming with offset-based resumption | Ordered, replayable data delivery over HTTP |
36
+
| **State Protocol** | Higher-level protocol defining semantic meaning (insert/update/delete) | Database-style sync, entity management, CRUD operations |
37
+
| **@durable-streams/state** | NPM package providing State Protocol primitives | MaterializedState, createStreamDB, schema validation |
38
+
| **@tanstack/db** | Optional integration for reactive queries, filtering, joins | Real-time reactive UI, differential dataflow |
39
+
| **Standard Schema** | Vendor-neutral schema format for validation | Works with Zod, Valibot, ArkType |
40
+
41
+
## Key Design Principles
42
+
43
+
State Protocol follows these core design principles:
44
+
45
+
1. **Composability** - Build on Durable Streams without requiring changes to base protocol
46
+
2. **Type-Safe** - Discriminated unions enable multi-type streams
47
+
3. **Decoupled** - Separates event processing from persistence decisions
48
+
4. **Schema-Agnostic** - Uses Standard Schema for validation, supporting multiple schema libraries
49
+
5. **Protocol Neutrality** - Works across different transport layers (HTTP, SSE, WebSocket)
50
+
51
+
---
52
+
53
+
# journal - Protocol Overview and Architecture
54
+
55
+
State Protocol is a **message format specification** that sits on top of Durable Streams Protocol. It defines the structure of state change events that get appended to durable streams.
56
+
57
+
## Protocol Stack
58
+
59
+
```
60
+
┌─────────────────────────────────────────────────────────────┐
61
+
│ Application Layer │
62
+
│ (React, Vue, Native Apps, AI Agents) │
63
+
└────────────────────┬────────────────────────────────────────┘
64
+
│
65
+
┌───────────▼─────────────┐
66
+
│ State Protocol │
67
+
│ (Event Format Specification) │
68
+
└───────────┬────────────────┘
69
+
│
70
+
┌───────────▼────────────────┐
71
+
│ Durable Streams Protocol │
72
+
│ (Base HTTP Streaming Protocol) │
73
+
└───────────┬───────────────────┘
74
+
│
75
+
┌───────────▼───────────────────────────────┐
76
+
│ Storage Backend │
77
+
│ (Postgres, SQLite, Files, Object Store) │
78
+
└───────────────────────────────────────────────┘
79
+
```
80
+
81
+
## Relationship to Durable Streams Protocol
82
+
83
+
State Protocol **extends** Durable Streams Protocol (PROTOCOL):
84
+
85
+
| Aspect | Durable Streams Protocol | State Protocol |
86
+
|---------|------------------------|----------------|
87
+
| **Purpose** | Byte-level ordered streaming | Semantic state changes (insert/update/delete) |
88
+
| **Content-Type** | Any MIME type | `application/json` required |
89
+
| **Message Structure** | Raw bytes or JSON arrays | JSON objects with type/key/value/headers |
90
+
| **Operation Semantics** | Append bytes | Entity mutations |
91
+
| **Transport Layer** | HTTP (PUT, POST, GET) | Same, with State-specific messages |
92
+
93
+
---
94
+
95
+
# journal - Message Types and Operations
96
+
97
+
State Protocol defines two primary message categories: **Change Messages** and **Control Messages**.
98
+
99
+
## Change Messages
100
+
101
+
Change messages represent state mutations (insert, update, delete operations) on entities.
102
+
103
+
### Message Structure
104
+
105
+
```json
106
+
{
107
+
"type": "<entity-type>",
108
+
"key": "<entity-key>",
109
+
"value": <any-json-value>,
110
+
"old_value": <any-json-value>, // Optional
111
+
"headers": {
112
+
"operation": "insert" | "update" | "delete",
113
+
"txid": "<transaction-id>", // Optional
114
+
"timestamp": "<rfc3339-timestamp>" // Optional
115
+
}
116
+
}
117
+
```
118
+
119
+
### Field Requirements
120
+
121
+
| Field | Required | Description |
122
+
|--------|-----------|-------------|
123
+
| `type` | **MUST** | Non-empty string discriminator for entity type |
124
+
| `key` | **MUST** | Non-empty string unique identifier within type |
125
+
| `value` | **MUST** for insert/update | The new entity data (any JSON value) |
126
+
| `old_value` | **MAY** | Previous value for conflict detection or audit logging |
127
+
| `headers.operation` | **MUST** | One of: `"insert"`, `"update"`, `"delete"` |
128
+
| `headers.txid` | **MAY** | Transaction ID for grouping related changes |
129
+
| `headers.timestamp` | **MAY** | RFC3339 timestamp of when change occurred |
130
+
131
+
## Insert Operation
132
+
133
+
Creates a new entity in the state.
134
+
135
+
### Requirements
136
+
137
+
- `value` field **MUST** be present and contain entity data
138
+
- `old_value` field **SHOULD NOT** be present for insert operations
139
+
140
+
### Example
141
+
142
+
```json
143
+
{
144
+
"type": "user",
145
+
"key": "user:123",
146
+
"value": {
147
+
"name": "Alice",
148
+
"email": "alice@example.com",
149
+
"role": "admin"
150
+
},
151
+
"headers": {
152
+
"operation": "insert",
153
+
"timestamp": "2025-12-23T10:30:00Z"
154
+
}
155
+
}
156
+
```
157
+
158
+
### Semantics
159
+
160
+
- Creates new entity at `type`/`key` path
161
+
- If entity already exists at same key, behavior depends on conflict resolution
162
+
- `old_value` not included (no previous state exists)
163
+
164
+
## Update Operation
165
+
166
+
Modifies an existing entity in the state.
167
+
168
+
### Requirements
169
+
170
+
- `value` field **MUST** be present and contain new entity data
171
+
- `old_value` field **MAY** be present to enable conflict detection
172
+
173
+
### Example
174
+
175
+
```json
176
+
{
177
+
"type": "user",
178
+
"key": "user:123",
179
+
"value": {
180
+
"name": "Alice Smith",
181
+
"email": "alice.new@example.com"
182
+
},
183
+
"old_value": {
184
+
"name": "Alice",
185
+
"email": "alice@example.com"
186
+
},
187
+
"headers": {
188
+
"operation": "update",
189
+
"timestamp": "2025-12-23T10:35:00Z"
190
+
}
191
+
}
192
+
```
193
+
194
+
### Semantics
195
+
196
+
- Replaces entity at `type`/`key` path with new `value`
197
+
- `old_value` contains previous state for comparison/auditing
198
+
- Enables optimistic locking and conflict detection patterns
199
+
200
+
## Delete Operation
201
+
202
+
Removes an entity from the state.
203
+
204
+
### Requirements
205
+
206
+
- `value` field **MAY** be present (typically `null` or omitted entirely)
207
+
- `old_value` field **MAY** be present to preserve deleted entity data
208
+
209
+
### Example (value as null)
210
+
211
+
```json
212
+
{
213
+
"type": "user",
214
+
"key": "user:123",
215
+
"value": null,
216
+
"old_value": {
217
+
"name": "Alice",
218
+
"email": "alice@example.com"
219
+
},
220
+
"headers": {
221
+
"operation": "delete",
222
+
"timestamp": "2025-12-23T10:40:00Z"
223
+
}
224
+
}
225
+
```
226
+
227
+
### Example (value omitted)
228
+
229
+
```json
230
+
{
231
+
"type": "user",
232
+
"key": "user:123",
233
+
"old_value": {
234
+
"name": "Alice",
235
+
"email": "alice@example.com"
236
+
},
237
+
"headers": {
238
+
"operation": "delete",
239
+
"timestamp": "2025-12-23T10:40:00Z"
240
+
}
241
+
}
242
+
```
243
+
244
+
### Semantics
245
+
246
+
- Removes entity at `type`/`key` path from state
247
+
- `old_value` preserves deleted data for soft delete scenarios
248
+
- Value may be `null` or omitted - both patterns are valid
249
+
250
+
## Control Messages
251
+
252
+
Control messages provide stream management signals separate from data changes.
253
+
254
+
### Message Structure
255
+
256
+
```json
257
+
{
258
+
"headers": {
259
+
"control": "snapshot-start" | "snapshot-end" | "reset",
260
+
"offset": "<stream-offset>" // Optional
261
+
}
262
+
}
263
+
```
264
+
265
+
### Control Types
266
+
267
+
| Control Type | Description | Use Case |
268
+
|-------------|-------------|-----------|
269
+
| `snapshot-start` | Marks beginning of a snapshot boundary | Indicates that following messages represent complete state snapshot |
270
+
| `snapshot-end` | Marks end of a snapshot boundary | Enables snapshot extraction and replay |
271
+
| `reset` | Signals client to clear state and restart from offset | Used for state resets or schema migrations |
272
+
273
+
### Snapshot Boundaries
274
+
275
+
Snapshot boundaries allow clients to extract coherent state snapshots from an event stream:
276
+
277
+
```json
278
+
// Snapshot start
279
+
{
280
+
"headers": {
281
+
"control": "snapshot-start",
282
+
"offset": "0000000000000000_0000000000000500"
283
+
}
284
+
}
285
+
286
+
// Subsequent change messages...
287
+
288
+
// Snapshot end
289
+
{
290
+
"headers": {
291
+
"control": "snapshot-end",
292
+
"offset": "0000000000000000_0000000000001500"
293
+
}
294
+
}
295
+
```
296
+
297
+
### Reset Control
298
+
299
+
Reset signals clients to clear materialized state and restart from specified offset:
300
+
301
+
```json
302
+
{
303
+
"headers": {
304
+
"control": "reset",
305
+
"offset": "0000000000000000_0000000000000200"
306
+
}
307
+
}
308
+
```
309
+
310
+
**Use Cases:**
311
+
- State reset after schema migration
312
+
- Clear client-side cache
313
+
- Recovery from inconsistency
314
+
315
+
---
316
+
317
+
# journal - State Materialization
318
+
319
+
State Protocol defines how clients should materialize (construct) state from change events.
320
+
321
+
## Materialization Process
322
+
323
+
1. **Process messages sequentially** - Apply changes in stream order
324
+
2. **Apply change operations**:
325
+
- `insert`: Store entity at `type`/`key` path with `value`
326
+
- `update`: Replace entity at `type`/`key` path with `value`
327
+
- `delete`: Remove entity at `type`/`key` path
328
+
3. **Handle control messages**:
329
+
- `snapshot-start` / `snapshot-end`: Delimit snapshot boundaries
330
+
- `reset`: Clear state and restart from offset
331
+
4. **Maintain entity collections** - Organize state by entity type
332
+
333
+
## Storage Abstractions
334
+
335
+
State Protocol doesn't prescribe storage. Implementations **MAY** use:
336
+
337
+
| Storage | Characteristics | Best For |
338
+
|----------|----------------|-----------|
339
+
| **In-memory Map** | Fast, simple, non-persistent | Temporary state, server-side materialization |
340
+
| **IndexedDB** | Browser persistence, indexed | Web apps, offline-first functionality |
341
+
| **SQLite** | Structured query capabilities, ACID | Local databases, mobile apps |
342
+
| **TanStack DB** | Reactive queries, filtering, joins | Complex UI state management |
343
+
| **Custom Backend** | Any persistence layer | Production applications with database |
344
+
345
+
## TanStack DB Integration
346
+
347
+
TanStack DB provides built-in State Protocol integration:
348
+
349
+
```typescript
350
+
import { createStateSchema, createStreamDB } from "@durable-streams/state"
351
+
import { useLiveQuery } from "@tanstack/react-db"
352
+
import { eq } from "@tanstack/db"
353
+
354
+
// Define schema
355
+
const schema = createStateSchema({
356
+
users: {
357
+
schema: userSchema,
358
+
type: "user",
359
+
primaryKey: "id"
360
+
},
361
+
messages: {
362
+
schema: messageSchema,
363
+
type: "message",
364
+
primaryKey: "id"
365
+
}
366
+
})
367
+
368
+
// Create database backed by durable stream
369
+
const db = createStreamDB({
370
+
streamOptions: {
371
+
url: streamUrl,
372
+
contentType: "application/json"
373
+
},
374
+
state: schema
375
+
})
376
+
377
+
// Reactive query that updates automatically as state events arrive
378
+
const activeUsers = useLiveQuery((q) =>
379
+
q.from({ users: db.collections.users })
380
+
.where(({ users }) => eq(users.active, true))
381
+
)
382
+
```
383
+
384
+
**Benefits:**
385
+
- Differential dataflow for incremental updates
386
+
- Optimized query execution
387
+
- Automatic state management
388
+
- Efficient reactivity without manual diffing
389
+
390
+
---
391
+
392
+
# journal - Multi-Type Streams
393
+
394
+
State Protocol supports multiple entity types in a single stream using discriminated unions.
395
+
396
+
## Type and Key Pattern
397
+
398
+
**Composite Key**: `{type}:{key}`
399
+
400
+
- `type`: Entity type discriminator (routes to collection)
401
+
- `key`: Unique identifier within that type
402
+
- Together: Forms fully unique entity identifier
403
+
404
+
Example composite keys:
405
+
- `"user:user:123"` - User entity with ID 123
406
+
- `"message:msg:456"` - Message entity with ID 456
407
+
- `"presence:device:abc"` - Presence entity for device ABC
408
+
409
+
## Multi-Type Stream Example
410
+
411
+
```json
412
+
[
413
+
{
414
+
"type": "user",
415
+
"key": "user:123",
416
+
"value": { "name": "Alice" },
417
+
"headers": { "operation": "insert" }
418
+
},
419
+
{
420
+
"type": "message",
421
+
"key": "msg:456",
422
+
"value": { "userId": "user:123", "text": "Hello!" },
423
+
"headers": { "operation": "insert" }
424
+
},
425
+
{
426
+
"type": "reaction",
427
+
"key": "reaction:789",
428
+
"value": { "messageId": "msg:456", "emoji": "👍" },
429
+
"headers": { "operation": "insert" }
430
+
},
431
+
{
432
+
"type": "user",
433
+
"key": "user:123",
434
+
"value": { "name": "Alice Smith" },
435
+
"headers": { "operation": "update" }
436
+
}
437
+
]
438
+
```
439
+
440
+
## Use Cases
441
+
442
+
| Use Case | Entity Types | Description |
443
+
|-----------|--------------|-------------|
444
+
| **Chat Rooms** | users, messages, reactions | Real-time messaging with user accounts and message threading |
445
+
| **Collaborative Editing** | documents, cursors, operations | Google Docs-style collaboration with conflict resolution |
446
+
| **Presence Tracking** | users, sessions, devices | Online/offline status across multiple devices |
447
+
| **Feature Flags** | flags, configurations | Real-time configuration rollout |
448
+
| **AI Workflows** | tasks, results, artifacts | Streaming AI agent outputs with progress tracking |
449
+
450
+
---
451
+
452
+
# journal - Schema Validation
453
+
454
+
State Protocol supports schema validation using **Standard Schema** format.
455
+
456
+
## Standard Schema Integration
457
+
458
+
```typescript
459
+
import { z } from "zod"
460
+
461
+
// Define user schema
462
+
const userSchema = z.object({
463
+
name: z.string().min(1),
464
+
email: z.string().email(),
465
+
age: z.number().int().positive()
466
+
})
467
+
468
+
// Create typed stream
469
+
const db = createStreamDB({
470
+
streamOptions: { url: streamUrl },
471
+
state: {
472
+
users: {
473
+
schema: userSchema,
474
+
type: "user",
475
+
primaryKey: "id"
476
+
}
477
+
}
478
+
}
479
+
```
480
+
481
+
## Validation Behavior
482
+
483
+
When schema validation is enabled:
484
+
485
+
1. **Pre-materialization validation** - Change messages validated before application
486
+
2. **Type safety** - Invalid values rejected with clear error messages
487
+
3. **Multiple schema libraries** - Zod, Valibot, ArkType, etc. supported
488
+
4. **Selective validation** - Can validate only certain entity types
489
+
490
+
## Error Handling
491
+
492
+
Implementations **MAY**:
493
+
- Reject invalid messages
494
+
- Log validation errors
495
+
- Continue processing valid messages
496
+
- Provide detailed error information
497
+
498
+
---
499
+
500
+
# journal - API Surfaces and Usage
501
+
502
+
## @durable-streams/state Package
503
+
504
+
The main NPM package providing State Protocol primitives.
505
+
506
+
### Installation
507
+
508
+
```bash
509
+
npm install @durable-streams/state
510
+
```
511
+
512
+
### MaterializedState
513
+
514
+
Simple in-memory key-value store for basic state management:
515
+
516
+
```typescript
517
+
import { MaterializedState } from "@durable-streams/state"
518
+
519
+
const state = new MaterializedState()
520
+
521
+
// Apply change events
522
+
state.apply({
523
+
type: "user",
524
+
key: "1",
525
+
value: { name: "Kyle" },
526
+
headers: { operation: "insert" }
527
+
})
528
+
529
+
// Retrieve entity
530
+
const user = state.get("user", "1")
531
+
// Returns: { name: "Kyle" }
532
+
```
533
+
534
+
### createStreamDB
535
+
536
+
Create a reactive database backed by durable stream with TanStack DB:
537
+
538
+
```typescript
539
+
import { createStateSchema, createStreamDB } from "@durable-streams/state"
540
+
import { useLiveQuery } from "@tanstack/react-db"
541
+
542
+
const schema = createStateSchema({
543
+
users: {
544
+
schema: userSchema,
545
+
type: "user",
546
+
primaryKey: "id"
547
+
},
548
+
products: {
549
+
schema: productSchema,
550
+
type: "product",
551
+
primaryKey: "sku"
552
+
}
553
+
})
554
+
555
+
const db = createStreamDB({
556
+
streamOptions: {
557
+
url: streamUrl,
558
+
contentType: "application/json"
559
+
},
560
+
state: schema
561
+
})
562
+
563
+
// Query with live updates
564
+
const activeProducts = useLiveQuery((q) =>
565
+
q.from({ products: db.collections.products })
566
+
.where(({ products }) => eq(products.active, true))
567
+
)
568
+
```
569
+
570
+
## @durable-streams/client Integration
571
+
572
+
State Protocol works seamlessly with the Durable Streams client:
573
+
574
+
```typescript
575
+
import { DurableStream } from "@durable-streams/client"
576
+
577
+
// Create stream
578
+
const stream = await DurableStream.create({
579
+
url: "https://server.com/v1/stream/my-app",
580
+
contentType: "application/json"
581
+
})
582
+
583
+
// Write state changes
584
+
await stream.append({
585
+
type: "user",
586
+
key: "123",
587
+
value: { name: "Alice" },
588
+
headers: { operation: "insert" }
589
+
})
590
+
591
+
// Read state changes
592
+
const res = await stream.stream<Change>({ live: "auto" })
593
+
res.subscribeJson(async (batch) => {
594
+
for (const change of batch.items) {
595
+
// Apply change to state
596
+
}
597
+
})
598
+
```
599
+
600
+
---
601
+
602
+
# journal - Examples and Use Cases
603
+
604
+
## Example 1: Key/Value Store
605
+
606
+
Simple configuration management with optimistic updates:
607
+
608
+
```typescript
609
+
import { MaterializedState } from "@durable-streams/state"
610
+
611
+
const configState = new MaterializedState()
612
+
613
+
// Set theme
614
+
configState.apply({
615
+
type: "config",
616
+
key: "theme",
617
+
value: "dark",
618
+
headers: { operation: "insert" }
619
+
})
620
+
621
+
// Get current theme
622
+
const theme = configState.get("config", "theme")
623
+
// Returns: "dark"
624
+
```
625
+
626
+
## Example 2: Chat Application
627
+
628
+
Full chat room with users, messages, and reactions:
629
+
630
+
```typescript
631
+
import { createStreamDB, useLiveQuery } from "@durable-streams/state"
632
+
import { DurableStream } from "@durable-streams/client"
633
+
634
+
const chatDb = createStreamDB({
635
+
streamOptions: { url: "https://server.com/v1/stream/chat" },
636
+
state: {
637
+
users: {
638
+
schema: userSchema,
639
+
type: "user"
640
+
},
641
+
messages: {
642
+
schema: messageSchema,
643
+
type: "message"
644
+
},
645
+
reactions: {
646
+
schema: reactionSchema,
647
+
type: "reaction"
648
+
}
649
+
}
650
+
})
651
+
652
+
// Get online users
653
+
const onlineUsers = useLiveQuery((q) =>
654
+
q.from({ users: chatDb.collections.users })
655
+
.where(({ users }) => eq(users.status, "online"))
656
+
)
657
+
658
+
// Send message
659
+
async function sendMessage(text: string) {
660
+
const stream = await DurableStream.create({
661
+
url: "https://server.com/v1/stream/chat"
662
+
})
663
+
664
+
const msg = {
665
+
type: "message",
666
+
key: `msg:${Date.now()}`,
667
+
value: { userId, text, timestamp: Date.now() },
668
+
headers: { operation: "insert" }
669
+
}
670
+
671
+
await stream.append(msg)
672
+
}
673
+
```
674
+
675
+
## Example 3: Presence Tracking
676
+
677
+
Multi-device online status:
678
+
679
+
```typescript
680
+
// Presence state change
681
+
{
682
+
"type": "presence",
683
+
"key": "device:abc123",
684
+
"value": {
685
+
status: "online",
686
+
lastSeen: Date.now(),
687
+
metadata: { browser: "Chrome", os: "macOS" }
688
+
},
689
+
"headers": {
690
+
"operation": "update",
691
+
"txid": "session-123"
692
+
}
693
+
}
694
+
```
695
+
696
+
## Example 4: Background Jobs with Progress
697
+
698
+
From the examples in `.test-agent/durable-streams/examples/state/background-jobs/`:
699
+
700
+
```typescript
701
+
// Job update event
702
+
{
703
+
"type": "job",
704
+
"key": "job:456",
705
+
"value": {
706
+
status: "running",
707
+
progress: 45,
708
+
result: null
709
+
},
710
+
"old_value": {
711
+
status: "pending",
712
+
progress: 0,
713
+
result: null
714
+
},
715
+
"headers": {
716
+
"operation": "update",
717
+
"txid": "job-batch-789"
718
+
}
719
+
}
720
+
721
+
// Job completion
722
+
{
723
+
"type": "job",
724
+
"key": "job:456",
725
+
"value": {
726
+
status: "complete",
727
+
progress: 100,
728
+
result: { outputUrl: "/results/456.pdf" }
729
+
},
730
+
"old_value": { status: "running", ... },
731
+
"headers": {
732
+
"operation": "update",
733
+
"txid": "job-batch-789"
734
+
}
735
+
}
736
+
```
737
+
738
+
---
739
+
740
+
# journal - Features and Capabilities
741
+
742
+
## Key Features
743
+
744
+
| Feature | Description |
745
+
|----------|-------------|
746
+
| **Type Discrimination** | Route events to different handlers based on `type` field |
747
+
| **Composite Keys** | `{type}:{key}` pattern for multi-entity streams |
748
+
| **CRUD Operations** | Insert, update, delete with clear semantics |
749
+
| **Transaction Support** | Optional `txid` for grouping related changes |
750
+
| **Old Value Tracking** | `old_value` field for conflict detection and audit |
751
+
| **Snapshot Boundaries** | Control messages for extracting coherent state snapshots |
752
+
| **Reset Capability** | Clear state and restart from any offset |
753
+
| **Timestamps** | RFC3339 timestamps for ordering and debugging |
754
+
| **Schema Validation** | Standard Schema integration for type safety |
755
+
756
+
## Transport Agnosticism
757
+
758
+
State Protocol works over any transport that supports Durable Streams:
759
+
760
+
1. **HTTP GET (catch-up)** - Batch reads of historical events
761
+
2. **HTTP GET (long-poll)** - Low-latency waiting for new data
762
+
3. **HTTP GET (SSE)** - Real-time server push to browsers
763
+
4. **WebSocket** - Full-duplex bidirectional communication
764
+
5. **Future transports** - Any future Durable Streams transport (HTTP/2, QUIC, etc.)
765
+
766
+
## Content Types
767
+
768
+
State Protocol **REQUIRES** `Content-Type: application/json` for the underlying stream.
769
+
770
+
This requirement ensures:
771
+
- All clients speak same language
772
+
- JSON message boundaries are preserved
773
+
- Schema validation works consistently
774
+
775
+
---
776
+
777
+
# journal - Security Considerations
778
+
779
+
## Message Validation
780
+
781
+
Implementations **MUST**:
782
+
- Validate JSON structure before materialization
783
+
- Reject malformed messages
784
+
- Prevent injection attacks
785
+
786
+
**Required validations:**
787
+
788
+
| Field | Validation Rule |
789
+
|--------|---------------|
790
+
| `type` | Must be non-empty string |
791
+
| `key` | Must be non-empty string |
792
+
| `headers.operation` | Must be one of: `insert`, `update`, `delete` |
793
+
| `value` | Must be valid JSON (for insert/update) |
794
+
| `old_value` | If present, must be valid JSON |
795
+
796
+
## Type and Key Validation
797
+
798
+
Implementations **SHOULD**:
799
+
- Validate `type` values against allowed entity types
800
+
- Validate `key` format to prevent injection
801
+
- Use allowlists for entity types
802
+
803
+
## Untrusted Content
804
+
805
+
Since messages arrive over HTTP:
806
+
- Treat all content as untrusted
807
+
- Sanitize before applying to application state
808
+
- Consider resource limits and quotas
809
+
810
+
## Authentication and Authorization
811
+
812
+
State Protocol defers to Durable Streams Protocol:
813
+
814
+
- Use HTTP authentication (Bearer tokens, API keys, mTLS)
815
+
- Apply authorization at stream level or per-message
816
+
- Stream-level ACLs control who can read/write
817
+
818
+
---
819
+
820
+
# journal - Options and Alternatives
821
+
822
+
## Comparison with Other Protocols
823
+
824
+
| Aspect | State Protocol | GraphQL Subscriptions | Server-Sent Events | Firebase Realtime Database |
825
+
|----------|----------------|----------------------|---------------------|-------------------------|
826
+
| **Message Format** | JSON with defined schema | GraphQL response | Text/event-stream | Firebase Realtime Database SDK |
827
+
| **Operations** | Insert, update, delete | Queries, mutations | Push events | Set/Update/Delete |
828
+
| **Ordering** | Per stream | Per subscription | Per connection | Per operation |
829
+
| **Persistence** | Depends on backend | Depends on backend | Server-only | Built-in |
830
+
| **Offline Support** | Depends on implementation | Depends on implementation | None | Limited (cache) |
831
+
| **Conflict Resolution** | Application-defined | Application-defined | N/A | Last-write-wins |
832
+
| **Schema Validation** | Optional (Standard Schema) | GraphQL schema | N/A | Database rules |
833
+
| **CDN-Friendly** | Yes (via Durable Streams) | No | Yes | Limited |
834
+
835
+
## When to Use State Protocol
836
+
837
+
**Use State Protocol when:**
838
+
- ✅ You need database-style CRUD semantics on top of streaming
839
+
- ✅ Real-time state synchronization across clients
840
+
- ✅ Offline-first applications with conflict resolution
841
+
- ✅ Multi-entity type systems (users, messages, reactions, etc.)
842
+
- ✅ Complex queries with filtering and joins (via TanStack DB)
843
+
- ✅ Snapshot and replay capabilities
844
+
845
+
**Use direct Durable Streams when:**
846
+
- ✅ You only need ordered byte delivery
847
+
- ✅ AI token streaming (no insert/update/delete semantics needed)
848
+
- ✅ Simple log/event streaming (append-only)
849
+
850
+
**Use WebSocket/SSE when:**
851
+
- ✅ You need bidirectional real-time communication without persistence
852
+
- ✅ Simple notifications without sync requirements
853
+
- ✅ Browser-native push with minimal overhead
854
+
855
+
## Alternatives to State Protocol
856
+
857
+
| Alternative | Description | Trade-offs |
858
+
|-------------|-------------|-------------|
859
+
| **Direct WebSockets** | Lowest latency, full-duplex | No persistence, no replay, must implement sync yourself |
860
+
| **GraphQL Subscriptions** | Type-safe queries, schema-driven | Requires GraphQL server, more complex, less event-sourced |
861
+
| **Firebase Realtime Database** | Managed service, mobile SDKs | Vendor lock-in, proprietary, limited query capabilities |
862
+
| **Server-Sent Events** | Browser-native, simple | No persistence, server-only, limited replay |
863
+
| **Postgres Logical Replication** | Full SQL capabilities | Complex setup, not CDN-friendly, requires database servers |
864
+
| **CRDT libraries** (Yjs, Automerge) | Conflict-free offline editing | Complex, application-specific, learning curve |
865
+
| **Event sourcing frameworks** (EventStoreDB, EventStore) | Mature tooling | Framework-specific, may be overkill for simple cases |
866
+
867
+
---
868
+
869
+
# journal - Decision Points
870
+
871
+
## Choosing State Materialization Strategy
872
+
873
+
| Strategy | Complexity | Performance | Use Case |
874
+
|-----------|------------|------------|-----------|
875
+
| **MaterializedState** | Low | High | Simple configs, in-memory state, prototypes |
876
+
| **TanStack DB** | Medium | Medium-High | Real-time UIs, complex queries, reactive updates |
877
+
| **Custom IndexedDB** | High | Low-Medium | Offline-first web apps, browser persistence |
878
+
| **SQLite** | Medium | Medium | Mobile apps, local databases, ACID transactions |
879
+
| **Custom Backend** | High | High | Production apps, server-side rendering, database integration |
880
+
881
+
**Recommendation:** Start with MaterializedState for prototypes, upgrade to TanStack DB for production applications needing reactivity.
882
+
883
+
## Choosing Transport Layer
884
+
885
+
| Transport | Latency | Duplex | Browser Support | CDN-Friendly |
886
+
|-----------|----------|--------|----------------|--------------|
887
+
| **HTTP SSE** | Low | Server→Client | Universal | Yes |
888
+
| **WebSocket** | Very Low | Full | Universal | No |
889
+
| **Long-poll** | Medium | Half (request/response) | Universal | Yes |
890
+
| **HTTP/2** | Very Low | Full | Growing | Yes |
891
+
892
+
**Recommendation:** Use SSE for read-heavy workloads (leverages CDN), use WebSocket for interactive applications.
893
+
894
+
## Schema Validation Strategy
895
+
896
+
| Approach | Benefits | Costs |
897
+
|----------|---------|--------|
898
+
| **No validation** | Fastest, most flexible | Runtime type errors, potential invalid state |
899
+
| **Client-side validation** | Fast feedback, validation errors at client | Need to ship validation library to client |
900
+
| **Server-side validation** | Protects all clients, consistent errors | Server complexity, potential latency |
901
+
| **Both** | Best of both worlds | Maximum complexity, highest reliability |
902
+
903
+
**Recommendation:** Enable server-side validation for production, optionally add client-side validation for UX.
904
+
905
+
---
906
+
907
+
# journal - Implementation Notes
908
+
909
+
## Testing with Conformance Suite
910
+
911
+
The Durable Streams project includes conformance tests:
912
+
913
+
```bash
914
+
# Server conformance tests (124 tests)
915
+
npm install @durable-streams/server-conformance-tests
916
+
917
+
# Client conformance tests (221 tests)
918
+
npm install @durable-streams/client-conformance-tests
919
+
```
920
+
921
+
Running conformance tests ensures:
922
+
- Protocol compliance
923
+
- Correct HTTP status codes
924
+
- Proper header handling
925
+
- Message ordering guarantees
926
+
- Offset semantics
927
+
928
+
## Production Deployment Considerations
929
+
930
+
### Monitoring
931
+
932
+
Key metrics to track:
933
+
- Stream append latency
934
+
- Message processing throughput
935
+
- Error rates by type
936
+
- Client connection counts
937
+
- Backlog/lag metrics
938
+
939
+
### Scaling Considerations
940
+
941
+
- **Read scaling**: Leverage CDN caching via offset-based URLs
942
+
- **Write scaling**: Load balance across multiple backend instances
943
+
- **State materialization**: Scale horizontally across workers/instances
944
+
- **Connection pooling**: Reuse HTTP/2 connections, implement keep-alive
945
+
946
+
### Failure Handling
947
+
948
+
- **Graceful degradation**: Serve cached data when backends are unavailable
949
+
- **Circuit breakers**: Fail fast when error rates exceed thresholds
950
+
- **Retry with exponential backoff**: For transient failures
951
+
- **Dead letter queues**: Separate stream for processing failed messages
952
+
953
+
---
954
+
955
+
# Discussion Questions
956
+
957
+
1. **Should State Protocol be part of Durable Streams spec or remain separate?**
958
+
- Current approach (separate spec) enables clear separation of concerns
959
+
- Pro: Composable, can evolve independently
960
+
- Con: Two specs to maintain, potential for confusion
961
+
- **Verdict**: Keep separate but clarify relationship in docs
962
+
963
+
2. **Should control messages include optional data fields?**
964
+
- Current spec has minimal control messages (just `control` and optional `offset`)
965
+
- Could add: `reason`, `user_id`, `metadata` for richer semantics
966
+
- **Trade-off**: More informative vs increased complexity
967
+
- **Use case**: Reset with reason codes helps debugging
968
+
969
+
3. **Should State Protocol support conflict resolution strategies beyond last-write-wins?**
970
+
- Current spec: Application-defined
971
+
- Options: OT/CRDT, operational transforms, application-specific merging
972
+
- **Constraint**: Standardizing conflict resolution limits flexibility
973
+
- **Consideration**: Provide best practices, multiple example strategies
974
+
975
+
4. **Should `old_value` be required for all operations?**
976
+
- Current: Optional (MAY)
977
+
- Benefits: Conflict detection, audit trails, undo/redo
978
+
- Costs: Larger message size, storage overhead
979
+
- **Recommendation**: Make it optional but strongly recommended for production
980
+
981
+
5. **Should State Protocol support batch operations?**
982
+
- Current: Individual messages only
983
+
- Could add: `batch` operation with array of changes
984
+
- Benefits: Reduced network overhead, atomic multi-entity updates
985
+
- Challenges: Ordering semantics across batch, partial failure handling
986
+
- **Consideration**: Keep simple for v1.0, add to future version if needed
987
+
988
+
6. **How should schema validation errors be handled?**
989
+
- Options: Reject entire message, skip invalid fields, apply defaults
990
+
- Trade-offs: Data loss vs consistency vs graceful degradation
991
+
- **Recommendation**: Reject entire message for strict validation, provide opt-out for lenient mode
992
+
993
+
---
994
+
995
+
# Decision Points Summary
996
+
997
+
## Architecture Decisions
998
+
999
+
1. **Use State Protocol** when you need semantic state changes (insert/update/delete)
1000
+
2. **Use TanStack DB** for reactive UIs and complex queries
1001
+
3. **Use direct Durable Streams** for simple byte streaming without CRUD semantics
1002
+
4. **Enable schema validation** for production applications
1003
+
5. **Choose SSE** for read-heavy, CDN-leveraged workloads
1004
+
6. **Choose WebSocket** for interactive, bidirectional communication
1005
+
1006
+
## Implementation Decisions
1007
+
1008
+
1. **Implement server-side validation** for type safety
1009
+
2. **Use MaterializedState** for prototypes and simple apps
1010
+
3. **Track `old_value`** for audit trails and conflict detection
1011
+
4. **Use snapshot boundaries** for state export and replay
1012
+
5. **Implement monitoring** for production deployments
1013
+
6. **Design for horizontal scaling** with CDN-friendly offset-based reads
1014
+
1015
+
---
1016
+
1017
+
# References
1018
+
1019
+
## Official Documentation
1020
+
1021
+
- **Durable Streams Website**: https://electric-sql.com/products/durable-streams
1022
+
- **State Protocol Spec**: https://github.com/durable-streams/durable-streams/blob/main/packages/state/STATE-PROTOCOL.md
1023
+
- **Durable Streams Protocol (PROTOCOL)**: https://github.com/durable-streams/durable-streams/blob/main/PROTOCOL.md
1024
+
- **NPM Package (@durable-streams/state)**: https://www.npmjs.com/package/@durable-streams/state
1025
+
- **GitHub Repository**: https://github.com/durable-streams/durable-streams
1026
+
1027
+
## Related Packages
1028
+
1029
+
- **@durable-streams/client**: TypeScript client library
1030
+
- **@tanstack/db**: Reactive database with State Protocol integration
1031
+
- **@durable-streams/server-conformance-tests**: Server conformance test suite
1032
+
- **@durable-streams/client-conformance-tests**: Client conformance test suite
1033
+
- **@durable-streams/cli**: Development and testing CLI
1034
+
1035
+
## Example Applications
1036
+
1037
+
- **background-jobs**: Background job processing with progress tracking
1038
+
- **wikipedia-events**: Real-time Wikipedia edits dashboard
1039
+
- **wikipedia-worker**: Wikipedia EventStream consumption and transformation
1040
+
- **test-ui**: Visual testing interface for Durable Streams
1041
+
1042
+
## External Resources
1043
+
1044
+
- **Standard Schema**: https://github.com/standard-schema/spec
1045
+
- **TanStack DB Documentation**: https://tanstack.com/db
1046
+
- **Electric SQL Blog**: https://electric-sql.com/blog/
+907
doc/discovery/toon.md
+907
doc/discovery/toon.md
···
1
+
# TOON (Token-Oriented Object Notation) - Deep Dive
2
+
3
+
A comprehensive exploration of the TOON data format and Rust ecosystem.
4
+
5
+
---
6
+
7
+
# journal - Initial Investigation
8
+
9
+
I started by researching what TOON is and what Rust crates are available for it. From the official website [toonformat.dev](https://toonformat.dev), I learned that:
10
+
11
+
**TOON** (Token-Oriented Object Notation) is a compact, human-readable serialization format designed specifically for passing structured data to Large Language Models with significantly reduced token usage (30–60% compared to JSON).
12
+
13
+
Key technical characteristics:
14
+
- JSON data model compatibility
15
+
- Line-oriented, indentation-based syntax
16
+
- Minimal quoting (strings only quoted when necessary)
17
+
- Explicit array lengths for validation
18
+
- Tabular arrays for uniform object collections
19
+
- Three delimiter options: comma (default), tab, or pipe
20
+
21
+
The format was created by [Johann Schopplich](https://github.com/johannschopplich) and has an official specification maintained at [toon-format/spec](https://github.com/toon-format/spec).
22
+
23
+
## Tech Stack Overview
24
+
25
+
From reviewing the spec and official implementations, the TOON ecosystem involves:
26
+
27
+
| Component | Purpose |
28
+
|-----------|-----------|
29
+
| **Specification** | Normative document defining syntax, encoding rules, and conformance requirements |
30
+
| **Reference Implementation** | TypeScript/JavaScript implementation that serves as canonical reference |
31
+
| **Language Ports** | Community implementations in Rust, Python, Go, .NET, Swift, etc. |
32
+
| **CLI Tools** | Command-line utilities for JSON ↔ TOON conversion |
33
+
| **Validation Suite** | 345 conformance tests that implementations must pass |
34
+
35
+
## Available Rust Crates
36
+
37
+
Through research, I identified several Rust crates for TOON:
38
+
39
+
| Crate | Crate Name | Version | Status | Notes |
40
+
|-------|-------------|----------|--------|--------|
41
+
| toon-rs | `toon` | 0.1 | Active | Most complete, v3.0 spec compliant (345/345 tests), WASM bindings, CLI included |
42
+
| toon-rust | `toon-rust` | 0.1.3 | Active | v1.4 spec support, SIMD optimizations, streaming API |
43
+
| serde_toon | `serde_toon` | 0.2.0 | Active | Serde-focused, includes `toon!` macro for dynamic values |
44
+
| rtoon | `rtoon` | 0.2.1 | Active | Lightweight, focuses on encode/decode functions |
45
+
| tru | `tru` | Latest (3 days old) | Active | Listed on lib.rs |
46
+
| rustapi_toon | `rustapi-toon` | N/A | Niche | RustAPI framework specific |
47
+
48
+
Reference links:
49
+
- [toon-rs GitHub](https://github.com/jimmystridh/toon-rs)
50
+
- [toon-rust GitHub](https://github.com/dedsecrattle/toon-rust)
51
+
- [serde_toon GitHub](https://github.com/hxphsts/serde_toon)
52
+
- [rtoon GitHub](https://github.com/shreyasbhat0/rtoon)
53
+
54
+
---
55
+
56
+
# journal - Format Structure Review
57
+
58
+
I reviewed the [official TOON specification v3.0](https://github.com/toon-format/spec/blob/main/SPEC.md) to understand the file structure. The format has several key structural elements:
59
+
60
+
## Data Model
61
+
62
+
TOON encodes the same data types as JSON:
63
+
- **Primitives**: strings, numbers, booleans, null
64
+
- **Objects**: mappings from string keys to values
65
+
- **Arrays**: ordered sequences of values
66
+
67
+
## Root Forms
68
+
69
+
A TOON document can represent different root structures:
70
+
71
+
1. **Root object** (most common) - Fields appear at depth 0 with no parent key
72
+
2. **Root array** - Begins with `[N]:` or `[N]{fields}:` at depth 0
73
+
3. **Root primitive** - A single primitive value (string, number, boolean, or null)
74
+
75
+
## Objects
76
+
77
+
Objects use indentation-based structure (like YAML):
78
+
79
+
```toon
80
+
# Simple object
81
+
id: 123
82
+
name: Ada
83
+
active: true
84
+
85
+
# Nested objects
86
+
user:
87
+
id: 123
88
+
name: Ada
89
+
```
90
+
91
+
Key observations:
92
+
- Indentation replaces braces (default: 2 spaces per level)
93
+
- One space follows the colon in `key: value` pairs
94
+
- Empty objects at root yield empty documents
95
+
96
+
## Arrays
97
+
98
+
Arrays always declare their length in brackets `[N]`, with optional delimiter specification.
99
+
100
+
### Primitive Arrays (Inline)
101
+
102
+
```toon
103
+
tags[3]: admin,ops,dev
104
+
```
105
+
106
+
### Tabular Arrays (Uniform Objects)
107
+
108
+
When all objects in an array share the same fields with primitive values:
109
+
110
+
```toon
111
+
items[2]{sku,qty,price}:
112
+
A1,2,9.99
113
+
B2,1,14.5
114
+
```
115
+
116
+
This is TOON's signature feature - CSV-like rendering with explicit schema.
117
+
118
+
### Mixed/Non-Uniform Arrays
119
+
120
+
When tabular requirements aren't met:
121
+
122
+
```toon
123
+
items[3]:
124
+
- 1
125
+
- a: 1
126
+
- text
127
+
```
128
+
129
+
## Header Syntax
130
+
131
+
Array headers follow the pattern:
132
+
133
+
```
134
+
key[N<delimiter?>]{field1<delimiter>field2<delimiter>...}:
135
+
```
136
+
137
+
Where:
138
+
- `N` is the non-negative integer length
139
+
- `delimiter` (optional) can be tab (HTAB) or pipe (`|`) - comma is default
140
+
- Field names are optional, only used for tabular arrays
141
+
142
+
Example with explicit tab delimiter:
143
+
```toon
144
+
items[2 ]{sku name qty price}:
145
+
A1 Widget 2 9.99
146
+
```
147
+
148
+
## Key Folding (Optional Feature)
149
+
150
+
Since v1.5, TOON supports collapsing chains of single-key objects into dotted paths:
151
+
152
+
```toon
153
+
# Standard nesting
154
+
data:
155
+
metadata:
156
+
items[2]: a,b
157
+
158
+
# With key folding enabled
159
+
data.metadata.items[2]: a,b
160
+
```
161
+
162
+
Folding requires:
163
+
- Each object in chain has exactly one key
164
+
- All segments match `^[A-Za-z_][A-Za-z0-9_]*$` (identifier-safe)
165
+
- No segment contains the path separator (`.`)
166
+
167
+
## Quoting Rules
168
+
169
+
Strings are quoted **only when necessary**:
170
+
- Empty strings (`""`)
171
+
- Leading or trailing whitespace
172
+
- Equals `true`, `false`, or `null` (case-sensitive)
173
+
- Looks like a number (e.g., `"42"`, `"-3.14"`, `"05"`)
174
+
- Contains special characters: colon, quote, backslash, brackets, braces
175
+
- Contains the active delimiter in scope
176
+
- Equals `"-"` or starts with `"-"`
177
+
178
+
Valid escape sequences (only 5): `\\`, `\"`, `\n`, `\r`, `\t`
179
+
180
+
## Type Normalization
181
+
182
+
Encoders normalize values to JSON data model:
183
+
- `NaN`, `Infinity`, `-Infinity` → `null`
184
+
- Numbers use canonical decimal form (no exponent, no trailing zeros, `-0` → `0`)
185
+
- Dates → ISO 8601 strings
186
+
187
+
---
188
+
189
+
# journal - API Surface Analysis
190
+
191
+
I examined the API surfaces of the available Rust crates to understand their capabilities.
192
+
193
+
## toon-rs (Most Complete)
194
+
195
+
**Installation:**
196
+
```toml
197
+
[dependencies]
198
+
toon = "0.1"
199
+
200
+
# Optional features
201
+
toon = { version = "0.1", features = ["chrono"] }
202
+
```
203
+
204
+
**Core API:**
205
+
```rust
206
+
use toon::Options;
207
+
208
+
// Encoding
209
+
pub fn encode_to_string<T: Serialize>(value: &T, options: &Options) -> Result<String>
210
+
pub fn encode_to_writer<W: Write, T: Serialize>(writer: W, value: &T, options: &Options) -> Result<()>
211
+
212
+
// Decoding
213
+
pub fn decode_from_str<T: DeserializeOwned + 'static>(s: &str, options: &Options) -> Result<T>
214
+
pub fn decode_from_reader<R: Read, T: DeserializeOwned + 'static>(reader: R, options: &Options) -> Result<T>
215
+
```
216
+
217
+
**Options:**
218
+
```rust
219
+
pub struct Options {
220
+
pub indent: usize, // Default: 2
221
+
pub delimiter: Delimiter, // Comma | Tab | Pipe
222
+
pub key_folding: KeyFolding, // Off | Safe
223
+
pub flatten_depth: Option<usize>, // Max segments to fold
224
+
}
225
+
```
226
+
227
+
**Features:**
228
+
- `de_direct` - Direct Deserializer (bypasses intermediate JSON)
229
+
- `perf_memchr` - Faster scanning via memchr crate
230
+
- `perf_smallvec` - Reduce small allocations
231
+
- `perf_lexical` - Faster numeric parsing via lexical-core
232
+
- `chrono` - DateTime support
233
+
234
+
## toon-rust (SIMD Optimized)
235
+
236
+
**Installation:**
237
+
```toml
238
+
[dependencies]
239
+
toon-rust = "0.1.0"
240
+
serde = { version = "1.0", features = ["derive"], optional = true }
241
+
serde_json = "1.0"
242
+
```
243
+
244
+
**Core API (Standalone):**
245
+
```rust
246
+
use toon_rust::{encode, decode};
247
+
248
+
pub fn encode(value: &Value, options: Option<&EncodeOptions>) -> Result<String>
249
+
pub fn decode(input: &str, options: Option<&DecodeOptions>) -> Result<Value>
250
+
```
251
+
252
+
**Streaming API:**
253
+
```rust
254
+
use toon_rust::{encode_stream, decode_stream};
255
+
256
+
pub fn encode_stream<W: Write>(value: &Value, writer: &mut W, options: Option<&EncodeOptions>) -> Result<(), Error>
257
+
pub fn decode_stream<R: Read>(reader: R, options: Option<&DecodeOptions>) -> Result<Value, Error>
258
+
```
259
+
260
+
**Serde API:**
261
+
```rust
262
+
use toon_rust::{to_string, from_str};
263
+
264
+
pub fn to_string<T: Serialize>(value: &T) -> Result<String, Error>
265
+
pub fn from_str<T: DeserializeOwned>(s: &str) -> Result<T, Error>
266
+
```
267
+
268
+
**EncodeOptions:**
269
+
```rust
270
+
pub struct EncodeOptions {
271
+
delimiter: Delimiter, // Comma, Tab, or Pipe
272
+
length_marker: char, // Default: '['
273
+
indent: usize, // Default: 2
274
+
}
275
+
```
276
+
277
+
**DecodeOptions:**
278
+
```rust
279
+
pub struct DecodeOptions {
280
+
indent: usize, // Expected indentation
281
+
strict: bool, // Enable strict validation
282
+
}
283
+
```
284
+
285
+
## serde_toon (Serde-Focused)
286
+
287
+
**Installation:**
288
+
```toml
289
+
[dependencies]
290
+
serde = { version = "1.0", features = ["derive"] }
291
+
serde_toon = "0.2"
292
+
```
293
+
294
+
**Core API:**
295
+
```rust
296
+
use serde_toon::{to_string, from_str, to_value, toon};
297
+
298
+
pub fn to_string<T: Serialize>(value: &T) -> Result<String>
299
+
pub fn to_string_with_options<T: Serialize>(value: &T, options: &ToonOptions) -> Result<String>
300
+
pub fn to_string_pretty<T: Serialize>(value: &T) -> Result<String>
301
+
302
+
pub fn to_value<T: Serialize>(value: &T) -> Result<Value>
303
+
304
+
pub fn from_str<T: DeserializeOwned>(s: &str) -> Result<T>
305
+
pub fn from_reader<R: Read, T: DeserializeOwned>(reader: R) -> Result<T>
306
+
pub fn from_slice<T: DeserializeOwned>(slice: &[u8]) -> Result<T>
307
+
```
308
+
309
+
**Dynamic Value Construction:**
310
+
```rust
311
+
use serde_toon::{toon, Value};
312
+
313
+
let data = toon!({
314
+
"name": "Alice",
315
+
"age": 30,
316
+
"tags": ["rust", "serde", "llm"]
317
+
});
318
+
```
319
+
320
+
## rtoon (Lightweight)
321
+
322
+
**Installation:**
323
+
```toml
324
+
[dependencies]
325
+
rtoon = "0.2.1"
326
+
```
327
+
328
+
**Core API:**
329
+
```rust
330
+
use rtoon::{encode, decode};
331
+
332
+
pub fn encode(value: impl Serialize) -> ToonResult<String>
333
+
pub fn encode_default(value: impl Serialize) -> ToonResult<String>
334
+
pub fn encode_array(value: impl Serialize) -> ToonResult<String>
335
+
pub fn encode_object(value: impl Serialize) -> ToonResult<String>
336
+
337
+
pub fn decode(toon: &str) -> ToonResult<serde_json::Value>
338
+
pub fn decode_default(toon: &str) -> ToonResult<serde_json::Value>
339
+
pub fn decode_strict(toon: &str) -> ToonResult<serde_json::Value>
340
+
pub fn decode_no_coerce(toon: &str) -> ToonResult<serde_json::Value>
341
+
```
342
+
343
+
---
344
+
345
+
# journal - Examples and Use Cases
346
+
347
+
I created several examples to understand practical usage:
348
+
349
+
## Basic Object Serialization
350
+
351
+
```rust
352
+
use serde::{Serialize, Deserialize};
353
+
use toon::encode_to_string;
354
+
355
+
#[derive(Serialize, Deserialize)]
356
+
struct User {
357
+
id: u32,
358
+
name: String,
359
+
active: bool,
360
+
}
361
+
362
+
let user = User {
363
+
id: 123,
364
+
name: "Alice".to_string(),
365
+
active: true,
366
+
};
367
+
368
+
let toon = encode_to_string(&user, &Options::default())?;
369
+
// Output:
370
+
// id: 123
371
+
// name: Alice
372
+
// active: true
373
+
```
374
+
375
+
## Tabular Arrays
376
+
377
+
```rust
378
+
#[derive(Serialize, Deserialize)]
379
+
struct Product {
380
+
sku: String,
381
+
qty: u32,
382
+
price: f64,
383
+
}
384
+
385
+
let products = vec![
386
+
Product { sku: "A1".to_string(), qty: 2, price: 9.99 },
387
+
Product { sku: "B2".to_string(), qty: 1, price: 14.5 },
388
+
];
389
+
390
+
let toon = encode_to_string(&products, &Options::default())?;
391
+
// Output:
392
+
// products[2]{sku,qty,price}:
393
+
// A1,2,9.99
394
+
// B2,1,14.5
395
+
```
396
+
397
+
## Custom Delimiters
398
+
399
+
```rust
400
+
use toon::options::Delimiter;
401
+
402
+
let opts = Options {
403
+
delimiter: Delimiter::Pipe,
404
+
..Options::default()
405
+
};
406
+
407
+
let toon = encode_to_string(&data, &opts)?;
408
+
// Output:
409
+
// tags[#3|]: reading|gaming|coding
410
+
```
411
+
412
+
## Key Folding
413
+
414
+
```rust
415
+
let opts = Options {
416
+
key_folding: KeyFolding::Safe,
417
+
flatten_depth: None, // Fold entire chains
418
+
..Options::default()
419
+
};
420
+
421
+
let data = json!({
422
+
"data": { "metadata": { "items": ["a", "b"] } }
423
+
});
424
+
425
+
let toon = encode_to_string(&data, &opts)?;
426
+
// Output:
427
+
// data.metadata.items[2]: a,b
428
+
```
429
+
430
+
## Streaming Large Datasets
431
+
432
+
```rust
433
+
use std::fs::File;
434
+
use std::io::BufWriter;
435
+
436
+
let large_data = generate_large_dataset();
437
+
let file = File::create("output.toon")?;
438
+
let mut writer = BufWriter::new(file);
439
+
440
+
// Write incrementally without loading full string into memory
441
+
encode_to_string(&large_data, &Options::default())?;
442
+
```
443
+
444
+
## CLI Usage (toon-rs)
445
+
446
+
```bash
447
+
# Convert JSON to TOON
448
+
toon-cli data.json > data.toon
449
+
450
+
# Convert TOON to JSON
451
+
toon-cli --decode data.toon > data.json
452
+
453
+
# Use pipe delimiter for better token efficiency
454
+
toon-cli --delimiter tab data.json
455
+
456
+
# Strict mode validation
457
+
toon-cli --strict --decode data.toon
458
+
```
459
+
460
+
---
461
+
462
+
# journal - Delete Operation Analysis
463
+
464
+
The user asked: **"Is there a way to signify a 'delete' in TOON?"**
465
+
466
+
## Finding
467
+
468
+
**No**, TOON does not have a built-in "delete" operation or marker.
469
+
470
+
## Why TOON Doesn't Have Delete Operations
471
+
472
+
TOON is a **serialization format**, not a **data manipulation format**. Like JSON, YAML, XML, TOML, and other serialization formats:
473
+
474
+
1. **Purpose**: TOON is designed to encode/decode structured data for transmission or storage
475
+
2. **Immutability**: TOON documents are static representations of data at a point in time
476
+
3. **Round-trip**: The goal is faithful round-trip: `decode(encode(x)) == x`
477
+
478
+
## How Other Formats Handle Deletion
479
+
480
+
| Format | How deletion is handled |
481
+
|---------|------------------------|
482
+
| **JSON Patch** | Uses operations object with `"op": "remove"` and `"path": "/field"` |
483
+
| **JSON Merge Patch** | Fields are omitted from the patch to indicate deletion |
484
+
| **RFC 6902 JSON Patch** | Explicit operation types: add, remove, replace, move, copy, test |
485
+
| **MongoDB** | Uses `$unset` operator |
486
+
| **SQL DELETE** | Separate DELETE statement, not part of data format |
487
+
| **Git** | Deletion tracked via version control, not in file format itself |
488
+
489
+
## Alternatives for Deletion Semantics
490
+
491
+
If you need deletion semantics with TOON, consider these approaches:
492
+
493
+
### 1. Use a Higher-Level Protocol
494
+
495
+
```rust
496
+
// Define a message wrapper with operation type
497
+
#[derive(Serialize, Deserialize)]
498
+
enum Operation {
499
+
Update(Value),
500
+
Delete,
501
+
}
502
+
503
+
#[derive(Serialize, Deserialize)]
504
+
struct Message {
505
+
operation: Operation,
506
+
data: Value,
507
+
}
508
+
509
+
// Example delete message
510
+
let delete_msg = json!({
511
+
"operation": "Delete",
512
+
"data": { "path": "users.123" }
513
+
});
514
+
515
+
let toon = encode_to_string(&delete_msg, &opts)?;
516
+
```
517
+
518
+
### 2. Use Null to Signal Deletion
519
+
520
+
```rust
521
+
// Convention: null means delete this field
522
+
let data = json!({
523
+
"users": {
524
+
"123": null, // Convention: delete user 123
525
+
"456": { "name": "Bob" }
526
+
}
527
+
});
528
+
```
529
+
530
+
### 3. Use Explicit Versioning
531
+
532
+
```rust
533
+
#[derive(Serialize, Deserialize)]
534
+
struct VersionedData {
535
+
version: u64,
536
+
deleted: Vec<String>, // List of deleted IDs
537
+
data: Value,
538
+
}
539
+
```
540
+
541
+
### 4. Use JSON Patch as Complement
542
+
543
+
```rust
544
+
// Use TOON for data, JSON Patch for modifications
545
+
#[derive(Serialize, Deserialize)]
546
+
struct PatchSet {
547
+
data: String, // TOON-encoded data
548
+
patches: Value, // JSON Patch operations
549
+
}
550
+
```
551
+
552
+
### 5. Application-Level Delete Handling
553
+
554
+
```rust
555
+
// Your application interprets deletion based on business logic
556
+
struct UserStore {
557
+
users: HashMap<u32, User>,
558
+
}
559
+
560
+
impl UserStore {
561
+
fn apply_toon_update(&mut self, toon: &str, deleted_ids: &[u32]) {
562
+
let updates: HashMap<u32, User> = decode_from_str(toon, &opts)?;
563
+
for id in deleted_ids {
564
+
self.users.remove(id);
565
+
}
566
+
for (id, user) in updates {
567
+
self.users.insert(id, user);
568
+
}
569
+
}
570
+
}
571
+
```
572
+
573
+
## Recommendation
574
+
575
+
For most use cases, **use application-level logic** to handle deletions. TOON should remain a pure serialization format for data representation. If you need operation semantics, build a protocol on top of TOON that includes operation types, or use TOON in combination with a format designed for patch operations (like JSON Patch).
576
+
577
+
---
578
+
579
+
# journal - Performance Considerations
580
+
581
+
## Token Efficiency
582
+
583
+
TOON achieves 30-60% token reduction compared to JSON for typical use cases:
584
+
585
+
```json
586
+
// JSON: ~45 tokens
587
+
{"users":[{"id":1,"name":"Alice","role":"admin"},{"id":2,"name":"Bob","role":"user"}]}
588
+
```
589
+
590
+
```toon
591
+
// TOON: ~28 tokens (38% reduction)
592
+
users[2]{id,name,role}:
593
+
1,Alice,admin
594
+
2,Bob,user
595
+
```
596
+
597
+
Factors contributing to efficiency:
598
+
1. No braces `{}` - indentation-based
599
+
2. No redundant quotes - minimal quoting rules
600
+
3. No repeating keys - tabular arrays declare fields once
601
+
4. No commas between objects - newlines separate rows
602
+
5. Short array syntax - `[N]` instead of `[...]`
603
+
604
+
## Parsing Performance
605
+
606
+
| Crate | Performance Features | Notes |
607
+
|--------|-------------------|--------|
608
+
| **toon-rs** | Zero-copy parsing, direct deserializer, SIMD via perf flags | ~1.5-2x faster decode for small docs, ~2-3x for tabular arrays |
609
+
| **toon-rust** | SIMD optimizations (SSE2 on x86_64) | 30-50% faster parsing of large tabular arrays |
610
+
| **serde_toon** | Safe Rust, pre-allocated buffers | O(n) single-pass parsing |
611
+
| **rtoon** | Lightweight, minimal dependencies | Basic optimizations |
612
+
613
+
### toon-rs Performance Features
614
+
615
+
- `de_direct` - Bypasses intermediate JSON Value, deserializes directly to target types
616
+
- `perf_memchr` - Uses `memchr` crate for optimized byte scanning
617
+
- `perf_smallvec` - Uses `SmallVec` to reduce heap allocations in hot paths
618
+
- `perf_lexical` - Uses `lexical-core` for fast numeric parsing
619
+
620
+
Benchmark results from toon-rs:
621
+
- Small documents: ~1.5–2.0x faster decode
622
+
- 1k-row tabular arrays: ~2–3x faster decode
623
+
624
+
### toon-rust SIMD Features
625
+
626
+
- Automatic SIMD on x86_64 with SSE2 support
627
+
- Graceful fallback to scalar code on other platforms
628
+
- Parallel byte comparison for delimiter detection
629
+
- Quote-aware parsing using parallel character matching
630
+
- Threshold: SIMD used for inputs ≥ 32 bytes
631
+
632
+
## Memory Efficiency
633
+
634
+
**Streaming APIs** allow processing datasets larger than available RAM:
635
+
636
+
```rust
637
+
use std::fs::File;
638
+
use std::io::{BufReader, BufWriter};
639
+
640
+
// Encode without loading full document into memory
641
+
let file = File::create("large_output.toon")?;
642
+
let mut writer = BufWriter::new(file);
643
+
encode_stream(&large_dataset, &mut writer, &options)?;
644
+
645
+
// Decode incrementally
646
+
let file = File::open("large_input.toon")?;
647
+
let reader = BufReader::new(file);
648
+
let decoded = decode_stream(reader, &options)?;
649
+
```
650
+
651
+
## Performance Recommendations
652
+
653
+
1. **Use tab delimiters** for datasets with few quoted strings - token efficiency
654
+
2. **Enable performance features** in toon-rs (`de_direct`, `perf_*`)
655
+
3. **Use streaming APIs** for files > few MB
656
+
4. **Leverage BufWriter/BufReader** for file I/O
657
+
5. **Batch operations** on large arrays are more efficient than individual operations
658
+
659
+
---
660
+
661
+
# journal - Decision Points
662
+
663
+
## Choosing a Rust Crate
664
+
665
+
### Choose **toon-rs** if:
666
+
- You need full spec conformance (v3.0, 345/345 tests passing)
667
+
- You want the most feature-complete implementation
668
+
- You need WASM bindings
669
+
- You want an included CLI tool
670
+
- You need performance features (SIMD, zero-copy, streaming)
671
+
- You're building a production application
672
+
673
+
### Choose **toon-rust** if:
674
+
- You prefer v1.4 spec stability
675
+
- SIMD optimizations are critical for your workload
676
+
- You want a simpler streaming API
677
+
- You're working with `serde_json::Value` extensively
678
+
679
+
### Choose **serde_toon** if:
680
+
- You want a `toon!` macro for dynamic value construction
681
+
- You need a simple, Serde-focused API
682
+
- You want to minimize dependencies
683
+
- You don't need advanced features like CLI or WASM
684
+
685
+
### Choose **rtoon** if:
686
+
- You want the lightest possible implementation
687
+
- You need basic encode/decode with minimal overhead
688
+
- You don't need advanced features
689
+
690
+
## Choosing Delimiters
691
+
692
+
| Delimiter | Best For | Token Efficiency | Trade-offs |
693
+
|-----------|-----------|------------------|-------------|
694
+
| **Comma** | General purpose, mixed data | Baseline | May require quoting for values with commas |
695
+
| **Tab** | Data with few quoted strings, tabular data | ~10-15% better than comma | Less human-readable in some contexts |
696
+
| **Pipe** | Data with commas and tabs, CSV-like data | Similar to comma | Less conventional, may confuse readers |
697
+
698
+
## Choosing Spec Version
699
+
700
+
| Version | Stability | Features | Crate Support |
701
+
|----------|------------|-----------|---------------|
702
+
| **v3.0** | Working Draft | Latest: key folding, path expansion | toon-rs |
703
+
| **v1.4** | Stable | Core features | toon-rust, serde_toon, rtoon |
704
+
705
+
**Recommendation**: Use v3.0 (toon-rs) for new projects unless you need absolute stability from a mature spec.
706
+
707
+
## Strict vs Lenient Parsing
708
+
709
+
**Strict Mode** (default in most implementations):
710
+
- Enforces array length matches `[N]` declaration
711
+
- Validates indentation is multiple of indent size
712
+
- Rejects invalid escape sequences
713
+
- Errors on blank lines inside arrays
714
+
- Catches truncation and malformed data
715
+
716
+
**Lenient Mode**:
717
+
- Tolerates minor formatting issues
718
+
- Allows non-standard indentation
719
+
- Skips blank lines where possible
720
+
- Useful for debugging or human-edited files
721
+
722
+
**Recommendation**: Use strict mode in production, lenient mode during development.
723
+
724
+
---
725
+
726
+
# journal - Options and Alternatives
727
+
728
+
## Alternative Formats
729
+
730
+
| Format | Token Efficiency | Use Cases | Compared to TOON |
731
+
|---------|------------------|------------|-------------------|
732
+
| **JSON** | Baseline | APIs, web services, general data exchange | More verbose, but more widely supported |
733
+
| **JSONC** | Slightly worse | Config files with comments | Has comments, but no TOON's tabular format |
734
+
| **YAML** | Similar | Config files, human-readable data | More flexible but ambiguous, no explicit lengths |
735
+
| **TOML** | Better for config | Configuration files | Designed for config, not general data |
736
+
| **MessagePack** | Binary, not comparable | Binary protocols | Binary format, not human-readable |
737
+
| **CBOR** | Binary, not comparable | IoT, embedded systems | Binary format, concise but not text |
738
+
| **S-expressions** | Worse | Lisp-like data, code as data | More parentheses, less token-efficient |
739
+
| **EDN** | Similar | Clojure data, configuration | Lisp-like syntax, similar efficiency |
740
+
| **KDL** | Better | Configuration documents | Document-oriented, excellent for config |
741
+
742
+
## When to Use TOON
743
+
744
+
**Ideal for:**
745
+
- LLM prompt engineering
746
+
- Data passed between systems and AI/LLM components
747
+
- Log/event streaming where token cost matters
748
+
- Tabular datasets that benefit from compact representation
749
+
- Human-readable config that needs explicit structure
750
+
- Git-tracked data (line-oriented diffs)
751
+
752
+
**Not ideal for:**
753
+
- General-purpose APIs (use JSON)
754
+
- Binary protocols (use MessagePack, CBOR)
755
+
- Configuration with many comments (use TOML, YAML)
756
+
- Deeply nested, non-uniform structures (JSON may be more efficient)
757
+
- Maximum human readability for non-technical users (YAML)
758
+
759
+
## TOON vs JSON
760
+
761
+
```json
762
+
// JSON: Verbose, but universal
763
+
{
764
+
"users": [
765
+
{ "id": 1, "name": "Alice", "role": "admin" },
766
+
{ "id": 2, "name": "Bob", "role": "user" }
767
+
],
768
+
"count": 2
769
+
}
770
+
```
771
+
772
+
```toon
773
+
// TOON: Compact, with explicit structure
774
+
users[2]{id,name,role}:
775
+
1,Alice,admin
776
+
2,Bob,user
777
+
count: 2
778
+
```
779
+
780
+
**Advantages of TOON:**
781
+
- 30-60% fewer tokens
782
+
- Tabular format for uniform objects
783
+
- Explicit array lengths for validation
784
+
- Git-friendly (line-oriented)
785
+
786
+
**Advantages of JSON:**
787
+
- Universal support
788
+
- Better for non-uniform data
789
+
- More mature tooling ecosystem
790
+
- Well-understood by all developers
791
+
792
+
---
793
+
794
+
# Discussion Questions
795
+
796
+
1. **Should TOON adopt operation semantics (add/remove/replace) like JSON Patch?**
797
+
- Pro: More expressive for incremental updates
798
+
- Con: Adds complexity, moves away from pure serialization
799
+
- Verdict: Better as a layer on top of TOON, not part of the spec
800
+
801
+
2. **Is the current spec stable enough for production use?**
802
+
- v3.0 is a "Working Draft" but has extensive test coverage
803
+
- v1.4 is more stable but lacks key folding
804
+
- Verdict: v3.0 with toon-rs is production-ready for most use cases
805
+
806
+
3. **Should TOON support comments like YAML?**
807
+
- Pro: Better for config files
808
+
- Con: More complex parsing, LLM-focused format doesn't need it
809
+
- Verdict: Keep TOON focused; use JSONC or TOML for configs that need comments
810
+
811
+
4. **Should tab delimiter be the default instead of comma?**
812
+
- Pro: Better token efficiency for many datasets
813
+
- Con: Less conventional, confusing for users expecting CSV-like format
814
+
- Verdict: Keep comma as default, encourage tab for LLM-focused use cases
815
+
816
+
5. **Should array length declarations be optional?**
817
+
- Pro: More like other formats
818
+
- Con: Loses validation benefits, a key TOON feature
819
+
- Verdict: Keep mandatory for strict mode, optional for lenient
820
+
821
+
---
822
+
823
+
# Decision Points
824
+
825
+
## Should You Use TOON?
826
+
827
+
Use TOON if:
828
+
- ✅ You're passing data to/from LLMs
829
+
- ✅ Token costs are significant
830
+
- ✅ You have tabular or semi-tabular data
831
+
- ✅ You want human-readable but compact format
832
+
- ✅ Git diff friendliness matters
833
+
834
+
Don't use TOON if:
835
+
- ❌ You need universal language support (use JSON)
836
+
- ❌ You need maximum human readability (use YAML)
837
+
- ❌ You need comments in data (use JSONC, TOML)
838
+
- ❌ Your data is deeply nested and non-uniform
839
+
- ❌ You need binary format (use MessagePack, CBOR)
840
+
841
+
## Which Crate to Use?
842
+
843
+
| Use Case | Recommended Crate |
844
+
|-----------|------------------|
845
+
| General-purpose, feature-complete | **toon-rs** |
846
+
| Performance-critical tabular parsing | **toon-rust** (SIMD) |
847
+
| Simple Serde integration | **serde_toon** |
848
+
| Minimal dependencies | **rtoon** |
849
+
850
+
## Configuration Recommendations
851
+
852
+
```rust
853
+
// For LLM-focused use (maximum token efficiency)
854
+
Options {
855
+
delimiter: Delimiter::Tab,
856
+
key_folding: KeyFolding::Safe,
857
+
flatten_depth: None,
858
+
..Options::default()
859
+
}
860
+
861
+
// For human readability
862
+
Options {
863
+
delimiter: Delimiter::Comma,
864
+
key_folding: KeyFolding::Off,
865
+
indent: 2,
866
+
..Options::default()
867
+
}
868
+
869
+
// For strict validation in production
870
+
DecodeOptions {
871
+
strict: true,
872
+
..DecodeOptions::default()
873
+
}
874
+
```
875
+
876
+
---
877
+
878
+
# References
879
+
880
+
## Official Resources
881
+
882
+
- **TOON Format Website**: https://toonformat.dev
883
+
- **Specification**: https://github.com/toon-format/spec
884
+
- **Reference Implementation**: https://github.com/toon-format/toon
885
+
886
+
## Rust Crates
887
+
888
+
| Crate | Link | Docs |
889
+
|-------|-------|------|
890
+
| toon-rs | https://github.com/jimmystridh/toon-rs | https://docs.rs/toon |
891
+
| toon-rust | https://github.com/dedsecrattle/toon-rust | https://docs.rs/toon-rust |
892
+
| serde_toon | https://github.com/hxphsts/serde_toon | https://docs.rs/serde_toon |
893
+
| rtoon | https://github.com/shreyasbhat0/rtoon | https://docs.rs/rtoon |
894
+
895
+
## Test Fixtures
896
+
897
+
The test fixtures are available in `.test-agent/`:
898
+
- `toon-rust/` - Complete Rust implementation with tests
899
+
- `toon-rs/` - Feature-complete implementation with spec conformance
900
+
901
+
## Additional Reading
902
+
903
+
- [TOON Format Overview](https://toonformat.dev/guide/format-overview.html)
904
+
- [TOON for LLM Prompts](https://toonformat.dev/guide/llm-prompts.html)
905
+
- [Benchmarks](https://toonformat.dev/guide/benchmarks.html)
906
+
- [Syntax Cheatsheet](https://toonformat.dev/reference/syntax-cheatsheet.html)
907
+
- [Efficiency Formalization](https://toonformat.dev/reference/efficiency-formalization.html)
+1188
proto/state-protocol-toon.glm.md
+1188
proto/state-protocol-toon.glm.md
···
1
+
# The Durable Streams State Protocol (TOON Variant)
2
+
3
+
**Document:** Durable Streams State Protocol - TOON Serialization
4
+
**Base Version:** 1.0 (extends STATE-PROTOCOL.md)
5
+
**Date:** 2025-02-10
6
+
**Author:** Extension specification for duratoon project
7
+
**Status:** Alternative serialization for State Protocol
8
+
9
+
---
10
+
11
+
## Abstract
12
+
13
+
This document specifies a TOON-based serialization variant of the Durable Streams State Protocol. While maintaining identical semantics to the JSON-based specification, this variant uses TOON (Token-Oriented Object Notation) to reduce payload size by 30-60%. The protocol provides a shared vocabulary for state synchronization that works across different transport layers, storage backends, and application patterns, enabling database-style sync semantics over durable streams with reduced bandwidth consumption.
14
+
15
+
### Key Optimization: Tabular Array Batching
16
+
17
+
The primary optimization in this variant is the use of **TOON's tabular array format** to batch multiple change records together. Tabular arrays:
18
+
19
+
1. **Declare fields once** at the array header instead of repeating keys in every record
20
+
2. **Eliminate quotes** for most values (minimal quoting rules)
21
+
3. **Eliminate braces and commas** between objects (line-based structure)
22
+
4. **Declare length explicitly** for validation and streaming
23
+
24
+
This is both human-readable (CSV-like) and lightweight, making it ideal for state synchronization scenarios.
25
+
26
+
---
27
+
28
+
## Copyright Notice
29
+
30
+
Copyright (c) 2025 ElectricSQL
31
+
This specification is an extension of the original STATE-PROTOCOL.md, adapted for TOON serialization.
32
+
33
+
---
34
+
35
+
## Table of Contents
36
+
37
+
1. [Introduction](#1-introduction)
38
+
2. [Terminology](#2-terminology)
39
+
3. [Protocol Overview](#3-protocol-overview)
40
+
4. [Message Types](#4-message-types)
41
+
- 4.1. [Change Messages](#41-change-messages)
42
+
- 4.2. [Control Messages](#42-control-messages)
43
+
5. [Message Format](#5-message-format)
44
+
- 5.1. [Change Message Structure](#51-change-message-structure)
45
+
- 5.2. [Control Message Structure](#52-control-message-structure)
46
+
6. [State Materialization](#6-state-materialization)
47
+
7. [Schema Validation](#7-schema-validation)
48
+
8. [Use Cases](#8-use-cases)
49
+
9. [Security Considerations](#9-security-considerations)
50
+
10. [IANA Considerations](#10-iana-considerations)
51
+
11. [References](#11-references)
52
+
53
+
---
54
+
55
+
## 1. Introduction
56
+
57
+
### TOON vs JSON: Why This Variant Exists
58
+
59
+
The Durable Streams State Protocol extends the Durable Streams Protocol [PROTOCOL] by defining a standard message format for state synchronization. While the base specification uses `application/json`, this variant defines an equivalent `application/toon+json` format that reduces payload size while maintaining identical semantics.
60
+
61
+
**Payload Reduction Characteristics:**
62
+
63
+
| Aspect | JSON | TOON | Reduction |
64
+
|--------|------|------|-----------|
65
+
| Object keys | Repeated every time | Declared once in tabular header | 60-80% on repeated keys |
66
+
| String quotes | Required for all strings | Only when necessary | 40-60% on string values |
67
+
| Object delimiters | `{` `}` braces required | Indentation-based | 100% elimination |
68
+
| Array delimiters | `[` `]` brackets, `,` commas | `[N]` length, no commas | 30-50% on arrays |
69
+
| Structure overhead | Verbose | Minimal | Overall: 30-60% |
70
+
71
+
**Example: Inserting 5 Users**
72
+
73
+
JSON (288 bytes):
74
+
```json
75
+
[
76
+
{"type":"user","key":"user:1","value":{"name":"Alice","role":"admin"},"headers":{"operation":"insert"}},
77
+
{"type":"user","key":"user:2","value":{"name":"Bob","role":"user"},"headers":{"operation":"insert"}},
78
+
{"type":"user","key":"user:3","value":{"name":"Charlie","role":"user"},"headers":{"operation":"insert"}},
79
+
{"type":"user","key":"user:4","value":{"name":"Diana","role":"mod"},"headers":{"operation":"insert"}},
80
+
{"type":"user","key":"user:5","value":{"name":"Eve","role":"user"},"headers":{"operation":"insert"}}
81
+
]
82
+
```
83
+
84
+
TOON (162 bytes, 44% reduction):
85
+
```toon
86
+
changes[5]{type,key,value,headers.operation}:
87
+
user,user:1,"{name:Alice,role:admin}",insert
88
+
user,user:2,"{name:Bob,role:user}",insert
89
+
user,user:3,"{name:Charlie,role:user}",insert
90
+
user,user:4,"{name:Diana,role:mod}",insert
91
+
user,user:5,"{name:Eve,role:user}",insert
92
+
```
93
+
94
+
*Note: In practice, the value objects would also be structured using TOON, further reducing size. See Section 4.1.3 for full optimization.*
95
+
96
+
### Tabular Array Batching Pattern
97
+
98
+
The TOON variant enables a **batching pattern** not present in the original specification:
99
+
100
+
1. **Batch by operation type**: Group all `insert` operations together, all `update` operations together, etc.
101
+
2. **Batch by entity type**: Group changes for the same entity type together
102
+
3. **Explicit array length**: Enables streaming validation and memory-efficient parsing
103
+
104
+
This batching is transparent to the protocol semantics—clients materialize state identically regardless of serialization format.
105
+
106
+
### Design Goals
107
+
108
+
The protocol is designed to be:
109
+
110
+
- **Composable**: A building block that works with any transport layer (durable streams, WebSockets, Server-Sent Events)
111
+
- **Type-safe**: Supports multi-type streams with discriminated unions
112
+
- **Decoupled**: Separates event processing from persistence, allowing flexible storage backends
113
+
- **Bandwidth-efficient**: Reduces payload size by 30-60% through TOON serialization
114
+
- **Human-readable**: Tabular format remains readable and debuggable
115
+
- **Semantic equivalence**: Identical to JSON variant when deserialized
116
+
117
+
---
118
+
119
+
## 2. Terminology
120
+
121
+
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
122
+
123
+
**Change Message**: A message representing a state mutation (insert, update, or delete operation) on an entity identified by type and key.
124
+
125
+
**Control Message**: A message for stream management (snapshot boundaries, resets) separate from data changes.
126
+
127
+
**Entity Type**: A discriminator field in change messages that routes events to the correct collection or handler. Enables multi-type streams where different entity types coexist.
128
+
129
+
**Entity Key**: A unique identifier for an entity within a given type. Together with type, forms a composite key.
130
+
131
+
**Materialized State**: An in-memory or persistent view of state constructed by applying change events sequentially.
132
+
133
+
**Operation**: The type of change being applied: `insert` (create), `update` (modify), or `delete` (remove).
134
+
135
+
**Standard Schema**: A vendor-neutral schema format [STANDARD-SCHEMA] that enables validation with multiple schema libraries (Zod, Valibot, ArkType).
136
+
137
+
**TOON**: Token-Oriented Object Notation, a compact serialization format designed for reduced token usage while maintaining human readability.
138
+
139
+
**Tabular Array**: TOON array format where all objects share the same fields, allowing field names to be declared once in the header and values listed in CSV-like rows.
140
+
141
+
---
142
+
143
+
## 3. Protocol Overview
144
+
145
+
### Content-Type
146
+
147
+
The State Protocol TOON variant operates on streams created with `Content-Type: application/toon+json` as an alternative to `application/json` in the base Durable Streams Protocol [PROTOCOL].
148
+
149
+
**Content-Type Negotiation:**
150
+
- Clients **MAY** request `application/toon+json` via `Accept` header
151
+
- Servers **MAY** respond with TOON if requested and supported
152
+
- Fallback to `application/json` **MUST** be supported for compatibility
153
+
154
+
### Message Format
155
+
156
+
Messages are TOON objects that conform to one of two message types:
157
+
158
+
1. **Change Messages**: Represent state mutations (insert/update/delete)
159
+
2. **Control Messages**: Provide stream management signals
160
+
161
+
### Batch Delivery
162
+
163
+
When reading from streams, clients receive TOON-encoded message arrays (per Section 7.1 of [PROTOCOL]). The TOON variant encourages—but does not require—batching related change messages into tabular arrays for efficiency.
164
+
165
+
**Recommended Batching Strategies:**
166
+
167
+
1. **By Operation Type**: All `insert` messages in one array, all `update` messages in another
168
+
2. **By Entity Type**: Group messages for the same `type` together
169
+
3. **By Transaction**: Group messages with the same `txid` together
170
+
4. **Time-based**: Batch messages from the same time window
171
+
172
+
### Semantic Equivalence
173
+
174
+
Clients apply messages sequentially to materialize state, exactly as in the JSON variant. The serialization format is transparent to the materialization logic—after decoding TOON to JSON (or directly to application types), the process is identical.
175
+
176
+
### Implementation Considerations
177
+
178
+
The protocol does not prescribe:
179
+
- How state is persisted (in-memory, IndexedDB, SQLite, etc.)
180
+
- How queries are executed (direct map lookups, database queries, etc.)
181
+
- How conflicts are resolved (last-write-wins, CRDTs, etc.)
182
+
183
+
These decisions are left to implementations, enabling flexibility while providing a common event format.
184
+
185
+
---
186
+
187
+
## 4. Message Types
188
+
189
+
### 4.1. Change Messages
190
+
191
+
#### JSON vs TOON Structure
192
+
193
+
**JSON Format (from original spec):**
194
+
```json
195
+
{
196
+
"type": "user",
197
+
"key": "user:123",
198
+
"value": {"name": "Alice", "email": "alice@example.com"},
199
+
"headers": {"operation": "insert", "timestamp": "2025-01-15T10:30:00Z"}
200
+
}
201
+
```
202
+
203
+
**TOON Format (equivalent):**
204
+
```toon
205
+
type: user
206
+
key: user:123
207
+
value:
208
+
name: Alice
209
+
email: alice@example.com
210
+
headers:
211
+
operation: insert
212
+
timestamp: 2025-01-15T10:30:00Z
213
+
```
214
+
215
+
**Savings:**
216
+
- No quotes on key names: `"type":` → `type:`
217
+
- No quotes on unambiguous strings: `"user"` → `user`
218
+
- No braces `{}` at top level (indentation)
219
+
- No commas between fields
220
+
- **Reduction: ~45% for this example**
221
+
222
+
#### 4.1.1. Tabular Batching of Change Messages
223
+
224
+
The major optimization is batching multiple change messages into a tabular array:
225
+
226
+
```toon
227
+
changes[3]{type,key,value.name,value.email,headers.operation}:
228
+
user,user:1,Alice,alice@example.com,insert
229
+
user,user:2,Bob,bob@example.com,insert
230
+
user,user:3,Charlie,charlie@example.com,update
231
+
```
232
+
233
+
**Key observations:**
234
+
- Field names declared once in header
235
+
- Each row is one complete change message
236
+
- Length `[3]` enables streaming validation
237
+
- Values without special characters are unquoted
238
+
- **Savings vs JSON array: ~55-60%**
239
+
240
+
#### Single Message Requirements
241
+
242
+
When change messages are not batched (single message delivery), they **MUST** contain:
243
+
244
+
- `type` (string): Entity type discriminator
245
+
- `key` (string): Entity identifier
246
+
- `headers` (object): Operation metadata
247
+
248
+
For `insert` and `update` operations, change messages **MUST** also contain:
249
+
250
+
- `value` (any JSON/TOON value): The new value for the entity
251
+
252
+
For `delete` operations, change messages **MAY** contain:
253
+
254
+
- `value` (any JSON/TOON value): Typically `null` or omitted
255
+
256
+
Change messages **MAY** include:
257
+
258
+
- `old_value` (any JSON/TOON value): Previous value, useful for conflict detection or audit logging
259
+
260
+
The `headers` object **MUST** contain:
261
+
262
+
- `operation` (string): One of `"insert"`, `"update"`, or `"delete"`
263
+
264
+
The `headers` object **MAY** contain:
265
+
266
+
- `txid` (string): Transaction identifier for grouping related changes
267
+
- `timestamp` (string): RFC 3339 timestamp indicating when the change occurred
268
+
269
+
#### 4.1.2. Insert Operation
270
+
271
+
Insert operations create new entities. The `value` field **MUST** be present and contain the entity data. The `old_value` field **SHOULD NOT** be present for insert operations.
272
+
273
+
**Single Message Example:**
274
+
275
+
```toon
276
+
type: user
277
+
key: user:123
278
+
value:
279
+
name: Alice
280
+
email: alice@example.com
281
+
headers:
282
+
operation: insert
283
+
timestamp: 2025-01-15T10:30:00Z
284
+
```
285
+
286
+
**Tabular Batch Example (5 new users):**
287
+
288
+
```toon
289
+
users[5]{type,key,value.name,value.email,headers.operation,headers.timestamp}:
290
+
user,user:1,Alice,alice1@example.com,insert,2025-01-15T10:30:00Z
291
+
user,user:2,Bob,bob2@example.com,insert,2025-01-15T10:31:00Z
292
+
user,user:3,Charlie,charlie3@example.com,insert,2025-01-15T10:32:00Z
293
+
user,user:4,Diana,diana4@example.com,insert,2025-01-15T10:33:00Z
294
+
user,user:5,Eve,eve5@example.com,insert,2025-01-15T10:34:00Z
295
+
```
296
+
297
+
**Payload Comparison:**
298
+
299
+
| Format | Bytes | Reduction |
300
+
|--------|-------|-----------|
301
+
| JSON (5 objects) | ~480 | Baseline |
302
+
| TOON (tabular) | ~215 | **55%** |
303
+
304
+
#### 4.1.3. Update Operation
305
+
306
+
Update operations modify existing entities. The `value` field **MUST** be present and contain the new entity data. The `old_value` field **MAY** be present to enable conflict detection.
307
+
308
+
**Single Message Example:**
309
+
310
+
```toon
311
+
type: user
312
+
key: user:123
313
+
value:
314
+
name: Alice
315
+
email: alice.new@example.com
316
+
old_value:
317
+
name: Alice
318
+
email: alice@example.com
319
+
headers:
320
+
operation: update
321
+
timestamp: 2025-01-15T10:35:00Z
322
+
```
323
+
324
+
**Tabular Batch Example (3 updates with conflict detection):**
325
+
326
+
```toon
327
+
updates[3]{type,key,value.email,old_value.email,headers.operation,headers.timestamp}:
328
+
user,user:1,alice.new@example.com,alice@example.com,update,2025-01-15T10:35:00Z
329
+
user,user:2,bob.updated@example.com,bob@example.com,update,2025-01-15T10:36:00Z
330
+
user,user:3,charlie.new@example.com,charlie@example.com,update,2025-01-15T10:37:00Z
331
+
```
332
+
333
+
**Optimization Note:** When batching updates, if only certain fields change, those are the only fields needed in the `value` column. Fields not changing can be omitted or set to `-` placeholder in the tabular format.
334
+
335
+
#### 4.1.4. Delete Operation
336
+
337
+
Delete operations remove entities. The `value` field **MAY** be present (typically `null`) or **MAY** be omitted entirely. The `old_value` field **MAY** be present to preserve the deleted entity's data.
338
+
339
+
**Single Message Example (value omitted):**
340
+
341
+
```toon
342
+
type: user
343
+
key: user:123
344
+
old_value:
345
+
name: Alice
346
+
email: alice.new@example.com
347
+
headers:
348
+
operation: delete
349
+
timestamp: 2025-01-15T10:40:00Z
350
+
```
351
+
352
+
**Tabular Batch Example (soft delete batch):**
353
+
354
+
```toon
355
+
deletes[3]{type,key,old_value.name,old_value.email,headers.operation,headers.timestamp}:
356
+
user,user:1,Alice,alice@example.com,delete,2025-01-15T10:40:00Z
357
+
user,user:2,Bob,bob@example.com,delete,2025-01-15T10:41:00Z
358
+
user,user:3,Charlie,charlie@example.com,delete,2025-01-15T10:42:00Z
359
+
```
360
+
361
+
**Tabular Batch Example (hard delete batch, no old_value):**
362
+
363
+
```toon
364
+
deletes[3]{type,key,headers.operation,headers.timestamp}:
365
+
user,user:1,delete,2025-01-15T10:40:00Z
366
+
user,user:2,delete,2025-01-15T10:41:00Z
367
+
user,user:3,delete,2025-01-15T10:42:00Z
368
+
```
369
+
370
+
### 4.2. Control Messages
371
+
372
+
Control messages provide stream management signals separate from data changes. They **MUST** contain:
373
+
374
+
- `headers` (object): Control metadata
375
+
376
+
The `headers` object **MUST** contain:
377
+
378
+
- `control` (string): One of `"snapshot-start"`, `"snapshot-end"`, or `"reset"`
379
+
380
+
The `headers` object **MAY** contain:
381
+
382
+
- `offset` (string): Stream offset associated with the control event
383
+
384
+
#### 4.2.1. Snapshot Boundaries
385
+
386
+
The `snapshot-start` and `snapshot-end` control messages delimit snapshot boundaries. Servers **MAY** emit these to indicate that a sequence of change messages represents a complete snapshot of state at a point in time.
387
+
388
+
**Single Message Example:**
389
+
390
+
```toon
391
+
headers:
392
+
control: snapshot-start
393
+
offset: 123456_000
394
+
```
395
+
396
+
**Tabular Batch Example (multiple snapshot boundaries):**
397
+
398
+
While snapshot boundaries are typically emitted as single messages, they can be batched:
399
+
400
+
```toon
401
+
controls[2]{headers.control,headers.offset}:
402
+
snapshot-start,123456_000
403
+
snapshot-end,123456_789
404
+
```
405
+
406
+
#### 4.2.2. Reset Control
407
+
408
+
The `reset` control message signals that clients **SHOULD** clear their materialized state and restart from the indicated offset. This enables servers to signal state resets or schema migrations.
409
+
410
+
**Single Message Example:**
411
+
412
+
```toon
413
+
headers:
414
+
control: reset
415
+
offset: 123456_000
416
+
```
417
+
418
+
**Tabular Batch Example (multiple reset signals):**
419
+
420
+
```toon
421
+
resets[2]{headers.control,headers.offset}:
422
+
reset,100_000
423
+
reset,200_000
424
+
```
425
+
426
+
---
427
+
428
+
## 5. Message Format
429
+
430
+
### 5.1. Change Message Structure
431
+
432
+
#### Single Message Format
433
+
434
+
Change messages **MUST** be valid TOON objects with the following structure:
435
+
436
+
```toon
437
+
type: <entity-type>
438
+
key: <entity-key>
439
+
value: <any-toon-value>
440
+
old_value: <any-toon-value>
441
+
headers:
442
+
operation: insert | update | delete
443
+
txid: <transaction-id>
444
+
timestamp: <rfc3339-timestamp>
445
+
```
446
+
447
+
**Field Requirements (same as JSON variant):**
448
+
449
+
- `type`: **MUST** be a non-empty string
450
+
- `key`: **MUST** be a non-empty string
451
+
- `value`: **MUST** be a valid TOON value (string, number, boolean, null, array, or object)
452
+
- `old_value`: **MAY** be present; if present, **MUST** be a valid TOON value
453
+
- `headers.operation`: **MUST** be one of `"insert"`, `"update"`, or `"delete"`
454
+
- `headers.txid`: **MAY** be present; if present, **MUST** be a non-empty string
455
+
- `headers.timestamp`: **MAY** be present; if present, **MUST** be a valid RFC 3339 timestamp
456
+
457
+
#### Tabular Batch Format
458
+
459
+
Change messages **MAY** be batched into a tabular array with the following structure:
460
+
461
+
```toon
462
+
<array-name>[N]{<field-paths>}:
463
+
<row-1-values>
464
+
<row-2-values>
465
+
...
466
+
<row-N-values>
467
+
```
468
+
469
+
Where:
470
+
- `<array-name>`: An identifier for the batch (e.g., `changes`, `inserts`, `updates`, `deletes`)
471
+
- `[N]`: Explicit array length (non-negative integer)
472
+
- `{<field-paths>}`: Comma-separated list of field paths (e.g., `type,key,value.name,value.email`)
473
+
- `<row-values>`: Comma-separated values for each field path
474
+
475
+
**Field Path Syntax:**
476
+
477
+
Field paths use dot notation to access nested fields:
478
+
- `type` → root field
479
+
- `headers.operation` → nested field
480
+
- `value.name` → value object's `name` field
481
+
- `value.email` → value object's `email` field
482
+
483
+
**Example: Complete Tabular Batch**
484
+
485
+
```toon
486
+
changes[4]{type,key,value.name,value.role,headers.operation,headers.txid}:
487
+
user,user:1,Alice,admin,insert,tx_001
488
+
user,user:2,Bob,user,insert,tx_001
489
+
user,user:1,Alice Smith,admin,update,tx_002
490
+
user,user:2,Bob,mod,update,tx_002
491
+
```
492
+
493
+
This represents four change messages:
494
+
1. Insert user:1 as Alice/admin
495
+
2. Insert user:2 as Bob/user
496
+
3. Update user:1 name to "Alice Smith"
497
+
4. Update user:2 role to "mod"
498
+
499
+
**Payload Savings Analysis:**
500
+
501
+
| Format | Representation | Bytes |
502
+
|--------|---------------|-------|
503
+
| JSON | 4 separate objects | ~620 |
504
+
| TOON single | 4 separate TOON objects | ~340 |
505
+
| TOON tabular | 1 tabular array | ~175 |
506
+
507
+
**Tabular is 72% smaller than JSON, 49% smaller than single TOON messages.**
508
+
509
+
### 5.2. Control Message Structure
510
+
511
+
#### Single Message Format
512
+
513
+
Control messages **MUST** be valid TOON objects with the following structure:
514
+
515
+
```toon
516
+
headers:
517
+
control: snapshot-start | snapshot-end | reset
518
+
offset: <stream-offset>
519
+
```
520
+
521
+
**Field Requirements (same as JSON variant):**
522
+
523
+
- `headers.control`: **MUST** be one of `"snapshot-start"`, `"snapshot-end"`, or `"reset"`
524
+
- `headers.offset`: **MAY** be present; if present, **MUST** be a valid stream offset string
525
+
526
+
#### Tabular Batch Format
527
+
528
+
Control messages **MAY** be batched into a tabular array:
529
+
530
+
```toon
531
+
controls[N]{headers.control,headers.offset}:
532
+
<control-1>,<offset-1>
533
+
<control-2>,<offset-2>
534
+
...
535
+
<control-N>,<offset-N>
536
+
```
537
+
538
+
---
539
+
540
+
## 6. State Materialization
541
+
542
+
Clients materialize state by applying change messages sequentially in stream order. The materialization process **MUST**:
543
+
544
+
1. Process messages in the order they appear in the stream
545
+
2. For change messages:
546
+
- Apply `insert` operations by storing the entity at `type`/`key`
547
+
- Apply `update` operations by replacing the entity at `type`/`key`
548
+
- Apply `delete` operations by removing the entity at `type`/`key`
549
+
3. For control messages:
550
+
- Handle control signals according to application logic (e.g., clear state on `reset`)
551
+
552
+
### Materialization Process for Tabular Batches
553
+
554
+
When messages arrive in tabular batch format:
555
+
556
+
1. **Decode the tabular array** into individual change messages
557
+
2. **Process each row sequentially** in array order
558
+
3. **Apply each change** to the materialized state
559
+
560
+
The materialization logic is **identical** to the JSON variant—the only difference is the encoding/decoding step.
561
+
562
+
### Example Materialization
563
+
564
+
**Tabular Input:**
565
+
566
+
```toon
567
+
changes[3]{type,key,value.name,headers.operation}:
568
+
user,user:1,Alice,insert
569
+
user,user:2,Bob,insert
570
+
user,user:1,Alice Smith,update
571
+
```
572
+
573
+
**Decoded Messages:**
574
+
575
+
After decoding from TOON tabular format, the client processes:
576
+
577
+
```toon
578
+
# Message 1
579
+
type: user
580
+
key: user:1
581
+
value:
582
+
name: Alice
583
+
headers:
584
+
operation: insert
585
+
586
+
# Message 2
587
+
type: user
588
+
key: user:2
589
+
value:
590
+
name: Bob
591
+
headers:
592
+
operation: insert
593
+
594
+
# Message 3
595
+
type: user
596
+
key: user:1
597
+
value:
598
+
name: Alice Smith
599
+
headers:
600
+
operation: update
601
+
```
602
+
603
+
**Materialized State:**
604
+
605
+
```
606
+
type: "user"
607
+
key: "user:1" -> { "name": "Alice Smith" }
608
+
key: "user:2" -> { "name": "Bob" }
609
+
```
610
+
611
+
This state is **identical** to what would be produced by the JSON variant.
612
+
613
+
### Storage Considerations
614
+
615
+
The protocol does not prescribe how state is stored. Implementations **MAY** use:
616
+
617
+
- In-memory maps (for simple cases)
618
+
- IndexedDB (for browser persistence)
619
+
- SQLite (for local databases)
620
+
- TanStack DB collections (for query interfaces)
621
+
- Custom storage backends
622
+
623
+
The choice of serialization format (JSON vs TOON) for the protocol **does not affect** storage decisions—storage remains in whatever format the implementation chooses.
624
+
625
+
---
626
+
627
+
## 7. Schema Validation
628
+
629
+
Implementations **MAY** validate change message values using Standard Schema [STANDARD-SCHEMA]. Standard Schema provides a vendor-neutral format that works with multiple schema libraries (Zod, Valibot, ArkType).
630
+
631
+
### Validation Process
632
+
633
+
When schema validation is enabled:
634
+
635
+
- Change messages **SHOULD** be validated against the schema for their entity type before materialization
636
+
- Invalid messages **MAY** be rejected or logged according to implementation policy
637
+
- Schema validation **SHOULD NOT** block stream processing for other entity types
638
+
639
+
### TOON-Specific Validation Considerations
640
+
641
+
**Schema validation operates on the decoded values**, not the TOON encoding itself. The TOON format includes validation features that complement Standard Schema:
642
+
643
+
1. **Array length validation**: TOON's `[N]` syntax enables early detection of truncated arrays
644
+
2. **Type preservation**: TOON preserves JSON data model exactly, so schema validation is equivalent
645
+
3. **String quoting rules**: TOON's minimal quoting does not affect semantics—decoded strings are identical to JSON
646
+
647
+
### Validation Example
648
+
649
+
**Tabular TOON Input:**
650
+
651
+
```toon
652
+
users[3]{type,key,value.name,value.age,headers.operation}:
653
+
user,user:1,Alice,30,insert
654
+
user,user:2,Bob,25,insert
655
+
user,user:3,Charlie,35,insert
656
+
```
657
+
658
+
**After decoding**, validation proceeds as with JSON:
659
+
660
+
```javascript
661
+
const userSchema = z.object({
662
+
name: z.string(),
663
+
age: z.number().int().positive()
664
+
})
665
+
666
+
// Validate each row's value object
667
+
userSchema.parse({ name: "Alice", age: 30 }) // ✓ Valid
668
+
userSchema.parse({ name: "Bob", age: 25 }) // ✓ Valid
669
+
userSchema.parse({ name: "Charlie", age: 35 }) // ✓ Valid
670
+
```
671
+
672
+
The protocol does not require schema validation, but implementations **SHOULD** provide validation capabilities for production use.
673
+
674
+
---
675
+
676
+
## 8. Use Cases
677
+
678
+
The State Protocol with TOON serialization enables several common patterns with reduced bandwidth consumption.
679
+
680
+
### 8.1. Key/Value Store
681
+
682
+
Simple synced configuration with optimistic updates.
683
+
684
+
**JSON (65 bytes):**
685
+
```json
686
+
{
687
+
"type": "config",
688
+
"key": "theme",
689
+
"value": "dark",
690
+
"headers": {"operation": "insert"}
691
+
}
692
+
```
693
+
694
+
**TOON (38 bytes, 42% reduction):**
695
+
```toon
696
+
type: config
697
+
key: theme
698
+
value: dark
699
+
headers:
700
+
operation: insert
701
+
```
702
+
703
+
### 8.2. Presence Tracking
704
+
705
+
Real-time online status with heartbeat semantics.
706
+
707
+
**Tabular batch of presence updates (5 users):**
708
+
709
+
```toon
710
+
presence[5]{type,key,value.status,value.lastSeen,headers.operation}:
711
+
presence,user:1,online,1705312200000,update
712
+
presence,user:2,offline,1705312190000,update
713
+
presence,user:3,online,1705312210000,update
714
+
presence,user:4,away,1705312180000,update
715
+
presence,user:5,online,1705312220000,update
716
+
```
717
+
718
+
**Payload Comparison:**
719
+
720
+
| Format | Bytes | Reduction |
721
+
|--------|-------|-----------|
722
+
| JSON (5 objects) | ~520 | Baseline |
723
+
| TOON single (5 objects) | ~300 | 42% |
724
+
| TOON tabular | ~165 | **68%** |
725
+
726
+
### 8.3. Multi-Type Streams
727
+
728
+
Chat rooms with users, messages, reactions, and receipts.
729
+
730
+
**Tabular batch by type:**
731
+
732
+
```toon
733
+
users[3]{type,key,value.name,headers.operation}:
734
+
user,user:123,Alice,insert
735
+
user,user:456,Bob,insert
736
+
user,user:789,Charlie,insert
737
+
738
+
messages[5]{type,key,value.userId,value.text,headers.operation,headers.timestamp}:
739
+
message,msg:1,user:123,Hello!,insert,2025-01-15T10:00:00Z
740
+
message,msg:2,user:456,Hi Alice!,insert,2025-01-15T10:01:00Z
741
+
message,msg:3,user:123,How are you?,insert,2025-01-15T10:02:00Z
742
+
message,msg:4,user:789,Hey everyone!,insert,2025-01-15T10:03:00Z
743
+
message,msg:5,user:456,Good,insert,2025-01-15T10:04:00Z
744
+
745
+
reactions[3]{type,key,value.messageId,value.emoji,headers.operation}:
746
+
reaction,reaction:1,msg:1,👍,insert
747
+
reaction,reaction:2,msg:2,❤️,insert
748
+
reaction,reaction:3,msg:3,😂,insert
749
+
```
750
+
751
+
**Payload Comparison:**
752
+
753
+
| Format | Bytes | Reduction |
754
+
|--------|-------|-----------|
755
+
| JSON (11 objects) | ~1,200 | Baseline |
756
+
| TOON tabular (3 arrays) | ~520 | **57%** |
757
+
758
+
### 8.4. Feature Flags
759
+
760
+
Real-time configuration propagation.
761
+
762
+
**Tabular batch of flag updates:**
763
+
764
+
```toon
765
+
flags[3]{type,key,value.enabled,value.rollout.type,value.rollout.value,headers.operation}:
766
+
flag,new-editor,true,percentage,50,update
767
+
flag,dark-mode,true,percentage,100,update
768
+
flag,beta-api,false,percentage,0,update
769
+
```
770
+
771
+
**JSON Equivalent (3 objects):**
772
+
773
+
```json
774
+
[
775
+
{
776
+
"type": "flag",
777
+
"key": "new-editor",
778
+
"value": {"enabled": true, "rollout": {"type": "percentage", "value": 50}},
779
+
"headers": {"operation": "update"}
780
+
},
781
+
{
782
+
"type": "flag",
783
+
"key": "dark-mode",
784
+
"value": {"enabled": true, "rollout": {"type": "percentage", "value": 100}},
785
+
"headers": {"operation": "update"}
786
+
},
787
+
{
788
+
"type": "flag",
789
+
"key": "beta-api",
790
+
"value": {"enabled": false, "rollout": {"type": "percentage", "value": 0}},
791
+
"headers": {"operation": "update"}
792
+
}
793
+
]
794
+
```
795
+
796
+
**Payload Comparison:**
797
+
798
+
| Format | Bytes | Reduction |
799
+
|--------|-------|-----------|
800
+
| JSON | ~540 | Baseline |
801
+
| TOON tabular | ~240 | **56%** |
802
+
803
+
### 8.5. Snapshot Export
804
+
805
+
Snapshot boundaries wrapping a complete state export.
806
+
807
+
```toon
808
+
headers:
809
+
control: snapshot-start
810
+
offset: 123456_000
811
+
812
+
# Snapshot data (tabular batches for each entity type)
813
+
users[100]{type,key,value.name,value.email,value.role,headers.operation}:
814
+
user,user:1,Alice,alice@example.com,admin,insert
815
+
# ... 99 more users ...
816
+
817
+
messages[500]{type,key,value.userId,value.text,headers.operation,headers.timestamp}:
818
+
message,msg:1,user:1,First message!,insert,2025-01-15T09:00:00Z
819
+
# ... 499 more messages ...
820
+
821
+
headers:
822
+
control: snapshot-end
823
+
offset: 123456_789
824
+
```
825
+
826
+
---
827
+
828
+
## 9. Security Considerations
829
+
830
+
### 9.1. Message Validation
831
+
832
+
Clients **MUST** validate that received messages conform to the message format specified in this document. Malformed messages **SHOULD** be rejected to prevent injection attacks.
833
+
834
+
**TOON-Specific Considerations:**
835
+
836
+
- **Array length validation**: Verify that the number of rows matches `[N]` declaration
837
+
- **Delimiter validation**: Validate that row values are correctly separated by the expected delimiter
838
+
- **Field count validation**: Verify each row has the correct number of values for the declared fields
839
+
840
+
### 9.2. Schema Validation
841
+
842
+
When schema validation is enabled, implementations **MUST** validate change message values before materialization. Invalid values **SHOULD** be rejected to prevent type confusion attacks.
843
+
844
+
**Schema validation is performed after TOON decoding**, so the same validation logic as JSON applies.
845
+
846
+
### 9.3. Untrusted Content
847
+
848
+
As specified in base protocol [PROTOCOL], clients **MUST** treat stream contents as untrusted input. This applies to both the message structure and the values within change messages.
849
+
850
+
**TOON Decoding Safety:**
851
+
852
+
- Use well-tested TOON parser libraries with security audits
853
+
- Validate array bounds before allocation
854
+
- Sanitize decoded values before materialization
855
+
- Reject documents exceeding size limits
856
+
857
+
### 9.4. Type and Key Validation
858
+
859
+
Implementations **SHOULD** validate that `type` and `key` fields contain only expected values to prevent injection of unauthorized entity types or keys.
860
+
861
+
This validation is performed after decoding, regardless of TOON vs JSON encoding.
862
+
863
+
### 9.5. Transaction Identifiers
864
+
865
+
The `txid` field is opaque to clients. Servers **MAY** use transaction identifiers for grouping related changes, but clients **MUST NOT** rely on transaction semantics unless explicitly documented by the server.
866
+
867
+
### 9.6. Resource Limits
868
+
869
+
When parsing tabular arrays:
870
+
871
+
- **Enforce maximum array size**: Reject `[N]` values exceeding limits
872
+
- **Enforce maximum row length**: Reject rows that could cause buffer overflows
873
+
- **Use streaming parsing**: Avoid loading entire arrays into memory when possible
874
+
875
+
---
876
+
877
+
## 10. IANA Considerations
878
+
879
+
This document recommends the following media type registration:
880
+
881
+
**Media Type:** `application/toon+json`
882
+
883
+
**Type name:** application
884
+
885
+
**Subtype name:** toon+json
886
+
887
+
**Required parameters:** None
888
+
889
+
**Optional parameters:**
890
+
- `charset`: Same as application/json (default UTF-8)
891
+
892
+
**Encoding considerations:** Same as application/json
893
+
894
+
**Security considerations:** See Section 9 (Security Considerations)
895
+
896
+
**Interoperability considerations:** This format is semantically equivalent to application/json when decoded. Implementations supporting application/toon+json should also support application/json as a fallback.
897
+
898
+
**Published specification:** This document and the TOON specification [TOON-SPEC]
899
+
900
+
**Applications that use this media type:** Real-time state synchronization protocols, durable streams applications, systems requiring reduced bandwidth while maintaining JSON semantics.
901
+
902
+
**Additional information:**
903
+
Magic number(s): none
904
+
File extension(s): .toon
905
+
Macintosh file type code(s): TEXT
906
+
907
+
**Person & email address to contact for further information:**
908
+
TOON Format maintainers: https://github.com/toon-format/spec
909
+
910
+
**Intended usage:** COMMON
911
+
912
+
**Restrictions on usage:** none
913
+
914
+
**Author:** See Authors section of TOON specification
915
+
916
+
**Change controller:** TOON Format working group
917
+
918
+
---
919
+
920
+
## 11. References
921
+
922
+
### 11.1. Normative References
923
+
924
+
**[PROTOCOL]**
925
+
Durable Streams Protocol. ElectricSQL, 2025.
926
+
<https://github.com/electric-sql/durable-streams/blob/main/PROTOCOL.md>
927
+
928
+
**[STATE-PROTOCOL]**
929
+
Durable Streams State Protocol. ElectricSQL, 2025.
930
+
<https://github.com/durable-streams/durable-streams/blob/main/packages/state/STATE-PROTOCOL.md>
931
+
932
+
**[TOON-SPEC]**
933
+
TOON Format Specification. Johann Schopplich, 2025.
934
+
<https://github.com/toon-format/spec>
935
+
936
+
**[RFC2119]**
937
+
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
938
+
939
+
**[RFC3339]**
940
+
Klyne, G. and C. Newman, "Date and Time on Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002, <https://www.rfc-editor.org/info/rfc3339>.
941
+
942
+
**[RFC8174]**
943
+
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
944
+
945
+
**[STANDARD-SCHEMA]**
946
+
Standard Schema Specification.
947
+
<https://github.com/standard-schema/spec>
948
+
949
+
### 11.2. Informative References
950
+
951
+
**[JSON-SCHEMA]**
952
+
Wright, A., Andrews, H., and B. Hutton, "JSON Schema: A Media Type for Describing JSON Documents", draft-wright-json-schema-00 (work in progress).
953
+
954
+
**[TOON-WEBSITE]**
955
+
TOON Format Official Website.
956
+
<https://toonformat.dev>
957
+
958
+
**[TOON-RUST]**
959
+
toon-rs: Rust implementation of TOON format. Jimmy Stridh, 2025.
960
+
<https://github.com/jimmystridh/toon-rs>
961
+
962
+
---
963
+
964
+
## Appendix A: Payload Reduction Analysis
965
+
966
+
This appendix provides detailed payload reduction analysis comparing JSON and TOON for common State Protocol patterns.
967
+
968
+
### A.1. Single Change Message
969
+
970
+
| Field | JSON | TOON | Savings |
971
+
|-------|------|------|---------|
972
+
| `type` field | `"type":"user",` | `type: user` | 4 chars (36%) |
973
+
| `key` field | `"key":"user:123",` | `key: user:123` | 6 chars (38%) |
974
+
| `value` object | `"value":{` | `value:` | 7 chars (100%) |
975
+
| `headers` object | `"headers":{` | `headers:` | 9 chars (100%) |
976
+
| `operation` field | `"operation":"insert"` | `operation: insert` | 2 chars (18%) |
977
+
| Closing braces | `}}` | (none) | 2 chars (100%) |
978
+
| **Total** | **~145 bytes** | **~80 bytes** | **~45%** |
979
+
980
+
### A.2. Tabular Batch of 10 Change Messages
981
+
982
+
| Component | JSON | TOON | Savings |
983
+
|-----------|------|------|---------|
984
+
| Field declarations (per object) | 10 × 60 = 600 chars | 1 × 60 = 60 chars | 540 chars (90%) |
985
+
| Quotes on keys | 10 × 4 × 2 = 80 chars | 0 chars | 80 chars (100%) |
986
+
| Object braces | 10 × 2 = 20 chars | 0 chars | 20 chars (100%) |
987
+
| Array brackets | 2 chars | 2 chars | 0 chars |
988
+
| Comma separators | 19 chars | 0 chars (newlines) | 19 chars (100%) |
989
+
| **Total** | **~1,450 bytes** | **~500 bytes** | **~65%** |
990
+
991
+
### A.3. Snapshot Export (100 users, 500 messages)
992
+
993
+
| Entity Type | JSON | TOON | Reduction |
994
+
|-------------|------|------|-----------|
995
+
| Users (100 records) | ~30 KB | ~11 KB | **63%** |
996
+
| Messages (500 records) | ~150 KB | ~52 KB | **65%** |
997
+
| Snapshot metadata | ~200 bytes | ~120 bytes | **40%** |
998
+
| **Total** | **~180 KB** | **~63 KB** | **65%** |
999
+
1000
+
### A.4. Bandwidth Impact on Real-World Applications
1001
+
1002
+
| Application | Messages/sec | JSON Bandwidth | TOON Bandwidth | Monthly Savings (1TB cap) |
1003
+
|-------------|--------------|----------------|----------------|---------------------------|
1004
+
| Chat app (1M users) | 10 msg/sec avg | ~2 GB/hr | ~0.7 GB/hr | ~1 GB/day, ~30 GB/month |
1005
+
| Presence tracking | 5 updates/sec avg | ~1 GB/hr | ~0.4 GB/hr | ~0.6 GB/day, ~18 GB/month |
1006
+
| Feature flags | 1 update/min | ~10 MB/day | ~4 MB/day | ~6 MB/day, ~180 MB/month |
1007
+
1008
+
---
1009
+
1010
+
## Appendix B: Implementation Guidelines
1011
+
1012
+
### B.1. Encoding Recommendations
1013
+
1014
+
**When to use single messages:**
1015
+
- Low-frequency updates (less than 1 per second)
1016
+
- Messages with diverse field structures
1017
+
- Scenarios where batching adds latency
1018
+
- Debugging/development (easier to read individual messages)
1019
+
1020
+
**When to use tabular batching:**
1021
+
- High-frequency updates (more than 1 per second)
1022
+
- Messages with uniform field structure
1023
+
- Batch operations (bulk imports, initial sync)
1024
+
- Production environments where bandwidth matters
1025
+
1026
+
**Delimiter selection:**
1027
+
1028
+
| Delimiter | Best For | Quoting Impact |
1029
+
|-----------|-----------|----------------|
1030
+
| Comma (default) | General purpose | May require quotes for values containing commas |
1031
+
| Tab | Data with few quoted strings | ~10-15% better efficiency |
1032
+
| Pipe | Data with commas and tabs | Similar to comma |
1033
+
1034
+
### B.2. Decoding Guidelines
1035
+
1036
+
**Streaming tabular arrays:**
1037
+
1038
+
1. Read array header to get `[N]` and field declarations
1039
+
2. Pre-allocate buffers for N rows
1040
+
3. Parse rows sequentially, validating field count
1041
+
4. Decode each row into change message
1042
+
5. Apply messages immediately (don't wait for full array)
1043
+
1044
+
**Error handling:**
1045
+
1046
+
- If row count doesn't match `[N]`: Reject entire batch, request replay from last known good offset
1047
+
- If row has wrong field count: Skip row, log error, continue
1048
+
- If value cannot be decoded: Skip row, log error, continue
1049
+
1050
+
### B.3. Rust Implementation Recommendations
1051
+
1052
+
Based on the Rust TOON crates analysis (see `doc/discovery/toon.md`):
1053
+
1054
+
**For production use:**
1055
+
```toml
1056
+
[dependencies]
1057
+
toon = { version = "0.1", features = ["de_direct", "perf_memchr", "perf_smallvec"] }
1058
+
```
1059
+
1060
+
**For tabular arrays:**
1061
+
```rust
1062
+
use toon::{Options, Delimiter};
1063
+
1064
+
let opts = Options {
1065
+
delimiter: Delimiter::Comma, // or Tab, Pipe
1066
+
..Options::default()
1067
+
};
1068
+
1069
+
// Encode
1070
+
let toon = toon::encode_to_string(&changes_batch, &opts)?;
1071
+
1072
+
// Decode
1073
+
let changes: Vec<ChangeMessage> = toon::decode_from_str(&toon, &opts)?;
1074
+
```
1075
+
1076
+
### B.4. TypeScript Implementation Recommendations
1077
+
1078
+
For TypeScript/JavaScript clients, use the reference TOON implementation:
1079
+
1080
+
```bash
1081
+
npm install toon
1082
+
```
1083
+
1084
+
```typescript
1085
+
import { parse, stringify } from 'toon'
1086
+
1087
+
// Encode
1088
+
const toon = stringify(changesBatch, { delimiter: ',' })
1089
+
1090
+
// Decode
1091
+
const changes = parse(toon)
1092
+
```
1093
+
1094
+
---
1095
+
1096
+
## Appendix C: Migration Guide
1097
+
1098
+
### C.1. From JSON to TOON
1099
+
1100
+
**Server-side changes:**
1101
+
1102
+
1. Add TOON encoding capability:
1103
+
```typescript
1104
+
import { stringify } from 'toon'
1105
+
1106
+
function encodeMessages(messages: ChangeMessage[], format: 'json' | 'toon'): string {
1107
+
if (format === 'toon') {
1108
+
return stringify(messages, { delimiter: ',' })
1109
+
}
1110
+
return JSON.stringify(messages)
1111
+
}
1112
+
```
1113
+
1114
+
2. Support content negotiation:
1115
+
```typescript
1116
+
app.get('/stream/:id', (req, res) => {
1117
+
const format = req.accepts('application/toon+json', 'application/json')
1118
+
res.format({
1119
+
'application/toon+json': () => {
1120
+
res.type('application/toon+json')
1121
+
res.send(encodeToTOON(messages))
1122
+
},
1123
+
'application/json': () => {
1124
+
res.send(JSON.stringify(messages))
1125
+
}
1126
+
})
1127
+
})
1128
+
```
1129
+
1130
+
**Client-side changes:**
1131
+
1132
+
1. Update content-type request:
1133
+
```typescript
1134
+
const stream = await DurableStream.create({
1135
+
url: 'https://server.com/v1/stream/my-app',
1136
+
contentType: 'application/toon+json' // New: TOON
1137
+
// contentType: 'application/json' // Old: JSON
1138
+
})
1139
+
1140
+
// Decode TOON messages
1141
+
res.subscribeText(async (batch) => {
1142
+
const messages = parse(batch) // TOON decoder
1143
+
for (const msg of messages) {
1144
+
state.apply(msg)
1145
+
}
1146
+
})
1147
+
```
1148
+
1149
+
### C.2. Backward Compatibility Strategy
1150
+
1151
+
**Phase 1: Add TOON support (no clients using it)**
1152
+
- Deploy TOON encoding capability
1153
+
- Maintain JSON as default
1154
+
- Test with internal TOON clients
1155
+
1156
+
**Phase 2: Roll out TOON clients**
1157
+
- Update clients to request TOON
1158
+
- Monitor error rates
1159
+
- Gradually increase percentage
1160
+
1161
+
**Phase 3: Make TOON default**
1162
+
- Change server default to TOON
1163
+
- Keep JSON as fallback for compatibility
1164
+
- Remove JSON support in future version after all clients migrated
1165
+
1166
+
### C.3. A/B Testing
1167
+
1168
+
**Metrics to compare:**
1169
+
- Bandwidth consumption (bytes per message)
1170
+
- Encoding/decoding latency (time per batch)
1171
+
- Error rates (decode failures)
1172
+
- Memory usage (peak allocation)
1173
+
1174
+
**Expected results:**
1175
+
- Bandwidth: 40-60% reduction
1176
+
- Latency: 10-20% increase in encoding, 5-10% decrease in decoding
1177
+
- Error rates: No change (if TOON implementation is correct)
1178
+
- Memory: 20-30% reduction (smaller payloads in memory)
1179
+
1180
+
---
1181
+
1182
+
**Full Copyright Statement**
1183
+
1184
+
Copyright (c) 2025 ElectricSQL
1185
+
1186
+
This document and information contained herein are provided on an "AS IS" basis. ElectricSQL disclaims all warranties, express or implied, including but not limited to any warranty that the use of the information herein will not infringe any rights or any implied warranties of merchantability or fitness for a particular purpose.
1187
+
1188
+
This is an extension specification. For the base specification, see STATE-PROTOCOL.md.
+879
proto/state-protocol-toon.opus.md
+879
proto/state-protocol-toon.opus.md
···
1
+
# Durable Streams State Protocol - TOON Encoding Extension
2
+
3
+
**Document:** State Protocol TOON Extension
4
+
**Version:** 0.1
5
+
**Date:** 2025-02-10
6
+
**Status:** Draft Extension of Durable Streams State Protocol
7
+
8
+
---
9
+
10
+
## Abstract
11
+
12
+
This document specifies a TOON (Token-Oriented Object Notation) encoding for the Durable Streams State Protocol [STATE-PROTOCOL]. By leveraging TOON's tabular arrays, minimal quoting, and line-oriented structure, this extension achieves 30-60% reduction in payload size compared to JSON encoding while enabling efficient batching of homogeneous change messages.
13
+
14
+
### Key Optimization: Tabular Array Batching
15
+
16
+
The primary optimization is **TOON's tabular array format** for batching multiple change records:
17
+
18
+
1. **Declare fields once** in the array header instead of repeating keys per record
19
+
2. **Eliminate quotes** for most values (minimal quoting rules)
20
+
3. **Eliminate braces** between objects (line-based structure)
21
+
4. **Explicit length `[N]`** for validation and streaming
22
+
23
+
This is both human-readable (CSV-like) and lightweight, ideal for state synchronization.
24
+
25
+
## Table of Contents
26
+
27
+
1. [Introduction](#1-introduction)
28
+
2. [Rationale](#2-rationale)
29
+
3. [Content-Type](#3-content-type)
30
+
4. [Message Format Mapping](#4-message-format-mapping)
31
+
- 4.1. [Change Messages](#41-change-messages)
32
+
- 4.2. [Tabular Change Batches](#42-tabular-change-batches)
33
+
- 4.3. [Control Messages](#43-control-messages)
34
+
5. [Encoding Guidelines](#5-encoding-guidelines)
35
+
6. [Decoding Requirements](#6-decoding-requirements)
36
+
7. [Examples](#7-examples)
37
+
8. [Security Considerations](#8-security-considerations)
38
+
9. [IANA Considerations](#9-iana-considerations)
39
+
10. [References](#10-references)
40
+
11. [Appendix A: Payload Reduction Analysis](#appendix-a-payload-reduction-analysis)
41
+
12. [Appendix B: Implementation Guidelines](#appendix-b-implementation-guidelines)
42
+
13. [Appendix C: Migration Guide](#appendix-c-migration-guide)
43
+
44
+
---
45
+
46
+
## 1. Introduction
47
+
48
+
The base State Protocol [STATE-PROTOCOL] requires `Content-Type: application/json`. This extension defines an alternative encoding using TOON [TOON-SPEC], a token-efficient serialization format designed for LLM workloads and bandwidth-constrained environments.
49
+
50
+
### 1.1. Key Benefits
51
+
52
+
| Aspect | JSON Encoding | TOON Encoding |
53
+
|--------|---------------|---------------|
54
+
| Payload size | Baseline | 30-60% smaller |
55
+
| Batched changes | Array of objects | Tabular array (CSV-like) |
56
+
| Human readability | Good | Good (indentation-based) |
57
+
| Quoting overhead | All strings quoted | Minimal quoting |
58
+
| Array validation | Implicit | Explicit length `[N]` |
59
+
60
+
---
61
+
62
+
## 2. Rationale
63
+
64
+
### 2.1. Original JSON Encoding
65
+
66
+
Section 4.1 of STATE-PROTOCOL defines change messages as JSON objects:
67
+
68
+
```json
69
+
{
70
+
"type": "user",
71
+
"key": "user:123",
72
+
"value": { "name": "Alice", "email": "alice@example.com" },
73
+
"headers": { "operation": "insert", "timestamp": "2025-01-15T10:30:00Z" }
74
+
}
75
+
```
76
+
77
+
**Overhead sources:**
78
+
- Repeated field names (`"type"`, `"key"`, `"value"`, `"headers"`, `"operation"`) per message
79
+
- Mandatory quoting of all string keys and values
80
+
- Brace/bracket delimiters `{ } [ ]`
81
+
- Colon and comma separators with quotes
82
+
83
+
### 2.2. TOON Advantages
84
+
85
+
TOON addresses these overheads through:
86
+
87
+
1. **Tabular Arrays** - When multiple objects share the same schema, TOON renders them as header + rows, eliminating repeated field names
88
+
2. **Minimal Quoting** - Strings are unquoted unless they contain special characters or resemble other types
89
+
3. **Indentation-Based Structure** - Replaces braces with whitespace
90
+
4. **Explicit Lengths** - `[N]` declarations enable validation and stream recovery
91
+
92
+
### 2.3. Tabular Batching Opportunity
93
+
94
+
The State Protocol frequently transmits batches of homogeneous change messages (e.g., multiple inserts of the same entity type). TOON's tabular format is ideal for this pattern:
95
+
96
+
**JSON (3 user inserts):**
97
+
```json
98
+
[
99
+
{"type":"user","key":"1","value":{"name":"Alice","role":"admin"},"headers":{"operation":"insert"}},
100
+
{"type":"user","key":"2","value":{"name":"Bob","role":"user"},"headers":{"operation":"insert"}},
101
+
{"type":"user","key":"3","value":{"name":"Carol","role":"user"},"headers":{"operation":"insert"}}
102
+
]
103
+
```
104
+
105
+
**TOON tabular equivalent:**
106
+
```toon
107
+
[3]{type,key,value.name,value.role,headers.operation}:
108
+
user,1,Alice,admin,insert
109
+
user,2,Bob,user,insert
110
+
user,3,Carol,user,insert
111
+
```
112
+
113
+
The TOON representation eliminates repeated field names entirely, achieving ~60% size reduction for uniform batches.
114
+
115
+
---
116
+
117
+
## 3. Content-Type
118
+
119
+
Streams using TOON encoding **MUST** use:
120
+
121
+
```
122
+
Content-Type: application/toon+json
123
+
```
124
+
125
+
or with charset:
126
+
127
+
```
128
+
Content-Type: application/toon+json; charset=utf-8
129
+
```
130
+
131
+
The `+json` suffix indicates semantic equivalence to JSON when decoded (per RFC 6839).
132
+
133
+
### Content Negotiation
134
+
135
+
Implementations **MUST** support both JSON (`application/json`) and TOON (`application/toon+json`) content types.
136
+
137
+
**Client request:**
138
+
```http
139
+
Accept: application/toon+json, application/json;q=0.9
140
+
```
141
+
142
+
**Server response (TOON preferred):**
143
+
```http
144
+
Content-Type: application/toon+json
145
+
```
146
+
147
+
Servers **MAY** default to JSON for backward compatibility when no `Accept` header is present.
148
+
149
+
---
150
+
151
+
## 4. Message Format Mapping
152
+
153
+
This section maps each STATE-PROTOCOL message structure to its TOON equivalent.
154
+
155
+
### 4.1. Change Messages
156
+
157
+
#### 4.1.1. Single Change Message
158
+
159
+
**STATE-PROTOCOL Section 5.1** defines change messages with fields: `type`, `key`, `value`, `old_value` (optional), and `headers`.
160
+
161
+
**JSON (from spec):**
162
+
```json
163
+
{
164
+
"type": "user",
165
+
"key": "user:123",
166
+
"value": {
167
+
"name": "Alice",
168
+
"email": "alice@example.com"
169
+
},
170
+
"headers": {
171
+
"operation": "insert",
172
+
"timestamp": "2025-01-15T10:30:00Z"
173
+
}
174
+
}
175
+
```
176
+
177
+
**TOON equivalent:**
178
+
```toon
179
+
type: user
180
+
key: user:123
181
+
value:
182
+
name: Alice
183
+
email: alice@example.com
184
+
headers:
185
+
operation: insert
186
+
timestamp: 2025-01-15T10:30:00Z
187
+
```
188
+
189
+
**Commentary:** Single messages benefit from eliminated quoting and brace removal. The indentation-based nesting is more readable and ~25% smaller.
190
+
191
+
#### 4.1.2. Update with old_value
192
+
193
+
**JSON:**
194
+
```json
195
+
{
196
+
"type": "user",
197
+
"key": "user:123",
198
+
"value": { "name": "Alice Smith", "email": "alice.new@example.com" },
199
+
"old_value": { "name": "Alice", "email": "alice@example.com" },
200
+
"headers": { "operation": "update", "timestamp": "2025-01-15T10:35:00Z" }
201
+
}
202
+
```
203
+
204
+
**TOON equivalent:**
205
+
```toon
206
+
type: user
207
+
key: user:123
208
+
value:
209
+
name: Alice Smith
210
+
email: alice.new@example.com
211
+
old_value:
212
+
name: Alice
213
+
email: alice@example.com
214
+
headers:
215
+
operation: update
216
+
timestamp: 2025-01-15T10:35:00Z
217
+
```
218
+
219
+
#### 4.1.3. Delete Operation
220
+
221
+
**JSON (value omitted):**
222
+
```json
223
+
{
224
+
"type": "user",
225
+
"key": "user:123",
226
+
"old_value": { "name": "Alice", "email": "alice@example.com" },
227
+
"headers": { "operation": "delete", "timestamp": "2025-01-15T10:40:00Z" }
228
+
}
229
+
```
230
+
231
+
**TOON equivalent:**
232
+
```toon
233
+
type: user
234
+
key: user:123
235
+
old_value:
236
+
name: Alice
237
+
email: alice@example.com
238
+
headers:
239
+
operation: delete
240
+
timestamp: 2025-01-15T10:40:00Z
241
+
```
242
+
243
+
**Commentary:** Delete messages with `value: null` in JSON can simply omit the field in TOON, as the absence is semantically equivalent.
244
+
245
+
### 4.2. Tabular Change Batches
246
+
247
+
This is the primary optimization of the TOON extension. When a batch contains multiple change messages with uniform structure, they **SHOULD** be encoded as a tabular array.
248
+
249
+
#### 4.2.1. Tabular Format Requirements
250
+
251
+
Tabular encoding applies when:
252
+
1. All messages in the batch are change messages (not control messages)
253
+
2. All messages share the same set of scalar fields in `value`
254
+
3. All messages use the same `headers` fields
255
+
256
+
#### 4.2.2. Field Flattening
257
+
258
+
TOON tabular arrays require flat column schemas. Nested objects **MUST** be flattened using dot notation in the header:
259
+
260
+
| Nested Path | Tabular Column |
261
+
|-------------|----------------|
262
+
| `value.name` | `value.name` |
263
+
| `value.email` | `value.email` |
264
+
| `headers.operation` | `headers.operation` |
265
+
| `headers.timestamp` | `headers.timestamp` |
266
+
| `headers.txid` | `headers.txid` |
267
+
268
+
#### 4.2.3. Named Tabular Arrays
269
+
270
+
Tabular arrays **SHOULD** use descriptive names that indicate their content:
271
+
272
+
| Array Name | Use Case |
273
+
|------------|----------|
274
+
| `changes` | Mixed operations |
275
+
| `inserts` | Insert-only batch |
276
+
| `updates` | Update-only batch |
277
+
| `deletes` | Delete-only batch |
278
+
| `users` | Entity-type-specific batch |
279
+
| `messages` | Entity-type-specific batch |
280
+
281
+
#### 4.2.4. Homogeneous Batch Example
282
+
283
+
**STATE-PROTOCOL Section 8.3** shows multi-type streams. For homogeneous batches within a type:
284
+
285
+
**JSON:**
286
+
```json
287
+
[
288
+
{"type":"user","key":"1","value":{"name":"Alice","role":"admin"},"headers":{"operation":"insert","txid":"tx-001"}},
289
+
{"type":"user","key":"2","value":{"name":"Bob","role":"user"},"headers":{"operation":"insert","txid":"tx-001"}},
290
+
{"type":"user","key":"3","value":{"name":"Carol","role":"user"},"headers":{"operation":"insert","txid":"tx-001"}}
291
+
]
292
+
```
293
+
294
+
**TOON tabular (named `users`):**
295
+
```toon
296
+
users[3]{type,key,value.name,value.role,headers.operation,headers.txid}:
297
+
user,1,Alice,admin,insert,tx-001
298
+
user,2,Bob,user,insert,tx-001
299
+
user,3,Carol,user,insert,tx-001
300
+
```
301
+
302
+
**Size comparison:**
303
+
- JSON: ~380 bytes
304
+
- TOON: ~185 bytes
305
+
- **Reduction: ~51%**
306
+
307
+
#### 4.2.5. Mixed Operation Batch
308
+
309
+
When a batch contains different operations but the same schema:
310
+
311
+
**TOON:**
312
+
```toon
313
+
[4]{type,key,value.name,value.status,headers.operation}:
314
+
user,1,Alice,active,insert
315
+
user,2,Bob,active,insert
316
+
user,1,Alice,inactive,update
317
+
user,2,,deleted,delete
318
+
```
319
+
320
+
**Commentary:** Empty values (like `value.name` for delete) render as empty cells between delimiters.
321
+
322
+
#### 4.2.6. Delimiter Selection
323
+
324
+
Per TOON spec, delimiters can be comma (default), tab, or pipe. For State Protocol payloads:
325
+
326
+
- **Comma** (default): Best for most cases
327
+
- **Tab**: Better when values may contain commas (e.g., addresses, descriptions)
328
+
- **Pipe**: Alternative when both comma and tab appear in data
329
+
330
+
**Tab-delimited example:**
331
+
```toon
332
+
[2 ]{type key value.name value.bio headers.operation}:
333
+
user 1 Alice Writer, editor insert
334
+
user 2 Bob Developer, speaker insert
335
+
```
336
+
337
+
### 4.3. Control Messages
338
+
339
+
#### 4.3.1. Snapshot Boundaries
340
+
341
+
**STATE-PROTOCOL Section 4.2.1** defines snapshot-start and snapshot-end controls.
342
+
343
+
**JSON:**
344
+
```json
345
+
{ "headers": { "control": "snapshot-start", "offset": "123456_000" } }
346
+
```
347
+
348
+
**TOON:**
349
+
```toon
350
+
headers:
351
+
control: snapshot-start
352
+
offset: 123456_000
353
+
```
354
+
355
+
#### 4.3.2. Reset Control
356
+
357
+
**JSON:**
358
+
```json
359
+
{ "headers": { "control": "reset", "offset": "123456_000" } }
360
+
```
361
+
362
+
**TOON:**
363
+
```toon
364
+
headers:
365
+
control: reset
366
+
offset: 123456_000
367
+
```
368
+
369
+
#### 4.3.3. Control Messages in Batches
370
+
371
+
Control messages **MUST NOT** be included in tabular arrays because they have different schemas than change messages. When a batch contains both change and control messages, use mixed array format:
372
+
373
+
**TOON mixed array:**
374
+
```toon
375
+
[3]:
376
+
- headers:
377
+
control: snapshot-start
378
+
offset: 100
379
+
- type: user
380
+
key: 1
381
+
value:
382
+
name: Alice
383
+
headers:
384
+
operation: insert
385
+
- headers:
386
+
control: snapshot-end
387
+
offset: 200
388
+
```
389
+
390
+
---
391
+
392
+
## 5. Encoding Guidelines
393
+
394
+
### 5.1. Batching Strategies
395
+
396
+
The TOON variant enables batching patterns not present in the original specification:
397
+
398
+
| Strategy | Description | Best For |
399
+
|----------|-------------|----------|
400
+
| **By Operation** | Group `insert`, `update`, `delete` separately | Bulk imports, cleanup jobs |
401
+
| **By Entity Type** | Group same `type` together | Multi-type streams |
402
+
| **By Transaction** | Group same `txid` together | Atomic operations |
403
+
| **Time-based** | Batch messages within time window | High-frequency updates |
404
+
405
+
### 5.2. Tabular Preference
406
+
407
+
Encoders **SHOULD**:
408
+
1. Group consecutive change messages by schema shape
409
+
2. Encode homogeneous groups as tabular arrays
410
+
3. Fall back to object format for heterogeneous messages or control messages
411
+
412
+
**When to use single messages:**
413
+
- Low-frequency updates (< 1 per second)
414
+
- Messages with diverse field structures
415
+
- Scenarios where batching adds unacceptable latency
416
+
- Debugging/development (easier to read)
417
+
418
+
**When to use tabular batching:**
419
+
- High-frequency updates (> 1 per second)
420
+
- Messages with uniform field structure
421
+
- Bulk operations (imports, initial sync, snapshots)
422
+
- Production environments where bandwidth matters
423
+
424
+
### 5.3. Field Ordering
425
+
426
+
For tabular arrays, field order in the header **SHOULD** follow:
427
+
1. `type`
428
+
2. `key`
429
+
3. `value.*` fields (alphabetically)
430
+
4. `old_value.*` fields (alphabetically, if present)
431
+
5. `headers.*` fields (alphabetically)
432
+
433
+
Consistent ordering enables efficient streaming parsers.
434
+
435
+
### 5.4. Key Folding
436
+
437
+
TOON v3.0 supports key folding for single-key chains. For change messages, key folding **MAY** be used:
438
+
439
+
**Without folding:**
440
+
```toon
441
+
headers:
442
+
operation: insert
443
+
timestamp: 2025-01-15T10:30:00Z
444
+
```
445
+
446
+
**With folding:**
447
+
```toon
448
+
headers.operation: insert
449
+
headers.timestamp: 2025-01-15T10:30:00Z
450
+
```
451
+
452
+
Key folding is **OPTIONAL** but can reduce indentation overhead.
453
+
454
+
### 5.5. Quoting Rules
455
+
456
+
Per TOON spec, strings require quotes only when:
457
+
- Empty string (`""`)
458
+
- Leading/trailing whitespace
459
+
- Equals `true`, `false`, or `null` literally
460
+
- Looks like a number
461
+
- Contains special characters (`:`, `"`, `\`, `[`, `]`, `{`, `}`)
462
+
- Contains the active delimiter
463
+
464
+
**Unquoted (valid):**
465
+
```toon
466
+
name: Alice
467
+
email: alice@example.com
468
+
key: user:123
469
+
```
470
+
471
+
**Quoted (required):**
472
+
```toon
473
+
name: "Alice "
474
+
key: "user:123"
475
+
status: "true"
476
+
count: "42"
477
+
```
478
+
479
+
**Commentary:** The `key: user:123` example shows that colons within values require quoting. Implementations **MUST** handle this correctly.
480
+
481
+
---
482
+
483
+
## 6. Decoding Requirements
484
+
485
+
### 6.1. Tabular Expansion
486
+
487
+
Decoders **MUST** expand tabular arrays into individual change message objects:
488
+
489
+
**TOON input:**
490
+
```toon
491
+
[2]{type,key,value.name,headers.operation}:
492
+
user,1,Alice,insert
493
+
user,2,Bob,insert
494
+
```
495
+
496
+
**Decoded objects:**
497
+
```json
498
+
[
499
+
{"type":"user","key":"1","value":{"name":"Alice"},"headers":{"operation":"insert"}},
500
+
{"type":"user","key":"2","value":{"name":"Bob"},"headers":{"operation":"insert"}}
501
+
]
502
+
```
503
+
504
+
### 6.2. Dot-Notation Unflattening
505
+
506
+
Dotted field names in tabular headers **MUST** be unflattened to nested objects:
507
+
508
+
| Column | Resulting Path |
509
+
|--------|----------------|
510
+
| `value.name` | `{"value":{"name":...}}` |
511
+
| `value.address.city` | `{"value":{"address":{"city":...}}}` |
512
+
| `headers.operation` | `{"headers":{"operation":...}}` |
513
+
514
+
### 6.3. Empty Cell Handling
515
+
516
+
Empty cells in tabular rows **SHOULD** be interpreted as:
517
+
- Omitted field (not present in decoded object)
518
+
- NOT as empty string or null
519
+
520
+
This matches delete operation semantics where `value` may be absent.
521
+
522
+
### 6.4. Length Validation
523
+
524
+
Per TOON spec, the `[N]` length declaration enables validation. Decoders **SHOULD** verify row count matches declared length. Mismatches indicate truncation or corruption.
525
+
526
+
---
527
+
528
+
## 7. Examples
529
+
530
+
### 7.1. Chat Application Batch
531
+
532
+
**Scenario:** Real-time chat with users, messages, and reactions (STATE-PROTOCOL Section 8.3).
533
+
534
+
**JSON (original):**
535
+
```json
536
+
[
537
+
{"type":"user","key":"user:123","value":{"name":"Alice"},"headers":{"operation":"insert"}},
538
+
{"type":"message","key":"msg:456","value":{"userId":"user:123","text":"Hello!"},"headers":{"operation":"insert"}},
539
+
{"type":"reaction","key":"reaction:789","value":{"messageId":"msg:456","emoji":"👍"},"headers":{"operation":"insert"}}
540
+
]
541
+
```
542
+
543
+
**TOON (mixed types, non-tabular):**
544
+
```toon
545
+
[3]:
546
+
- type: user
547
+
key: user:123
548
+
value:
549
+
name: Alice
550
+
headers:
551
+
operation: insert
552
+
- type: message
553
+
key: msg:456
554
+
value:
555
+
userId: user:123
556
+
text: Hello!
557
+
headers:
558
+
operation: insert
559
+
- type: reaction
560
+
key: reaction:789
561
+
value:
562
+
messageId: msg:456
563
+
emoji: 👍
564
+
headers:
565
+
operation: insert
566
+
```
567
+
568
+
**Commentary:** Heterogeneous types prevent tabular optimization. The line-oriented format still provides ~20% reduction through eliminated quoting.
569
+
570
+
### 7.2. Bulk User Import
571
+
572
+
**Scenario:** Importing 1000 users with uniform schema.
573
+
574
+
**JSON approach:** 1000 repeated `{"type":"user","key":...,"value":{...},"headers":{...}}` objects.
575
+
576
+
**TOON tabular:**
577
+
```toon
578
+
[1000]{type,key,value.name,value.email,value.role,headers.operation,headers.txid}:
579
+
user,1,Alice,alice@example.com,admin,insert,import-001
580
+
user,2,Bob,bob@example.com,user,insert,import-001
581
+
user,3,Carol,carol@example.com,user,insert,import-001
582
+
... (997 more rows)
583
+
```
584
+
585
+
**Estimated savings:**
586
+
- JSON: ~100 bytes/user × 1000 = 100KB
587
+
- TOON: 80 byte header + 50 bytes/row × 1000 = 50KB
588
+
- **Reduction: ~50%**
589
+
590
+
### 7.3. Presence Heartbeats
591
+
592
+
**Scenario:** High-frequency presence updates (STATE-PROTOCOL Section 8.2).
593
+
594
+
**TOON tabular (tab-delimited for readability):**
595
+
```toon
596
+
[5 ]{type key value.status value.lastSeen headers.operation}:
597
+
presence user:1 online 1705312200000 update
598
+
presence user:2 online 1705312200100 update
599
+
presence user:3 away 1705312199000 update
600
+
presence user:4 online 1705312200200 update
601
+
presence user:5 offline 1705312100000 update
602
+
```
603
+
604
+
### 7.4. Snapshot with Boundaries
605
+
606
+
**Scenario:** Full state snapshot with control messages.
607
+
608
+
**TOON:**
609
+
```toon
610
+
[5]:
611
+
- headers:
612
+
control: snapshot-start
613
+
offset: 1000
614
+
- type: user
615
+
key: 1
616
+
value:
617
+
name: Alice
618
+
active: true
619
+
headers:
620
+
operation: insert
621
+
- type: user
622
+
key: 2
623
+
value:
624
+
name: Bob
625
+
active: true
626
+
headers:
627
+
operation: insert
628
+
- type: config
629
+
key: theme
630
+
value: dark
631
+
headers:
632
+
operation: insert
633
+
- headers:
634
+
control: snapshot-end
635
+
offset: 1003
636
+
```
637
+
638
+
---
639
+
640
+
## 8. Security Considerations
641
+
642
+
### 8.1. Parser Security
643
+
644
+
TOON parsers **MUST** implement the same security validations as JSON parsers:
645
+
- Depth limits to prevent stack overflow
646
+
- Size limits to prevent memory exhaustion
647
+
- UTF-8 validation
648
+
649
+
### 8.2. Injection Prevention
650
+
651
+
Per STATE-PROTOCOL Section 9.4, `type` and `key` fields require validation. TOON's quoting rules do not alter this requirement. Implementations **MUST** validate field contents after decoding.
652
+
653
+
### 8.3. Length Declaration Attacks
654
+
655
+
Malicious payloads may declare large `[N]` lengths without providing rows. Decoders **SHOULD**:
656
+
- Stream process rows without pre-allocating based on declared length
657
+
- Timeout or abort on missing rows
658
+
- Validate actual row count matches declaration
659
+
660
+
### 8.4. Resource Limits
661
+
662
+
When parsing tabular arrays, implementations **MUST** enforce limits:
663
+
664
+
| Resource | Recommended Limit | Rationale |
665
+
|----------|-------------------|-----------|
666
+
| Array size `[N]` | 10,000 rows | Prevent memory exhaustion |
667
+
| Row length | 64 KB | Prevent buffer overflow |
668
+
| Nesting depth | 32 levels | Prevent stack overflow |
669
+
| Document size | 10 MB | Prevent memory exhaustion |
670
+
671
+
Implementations **SHOULD** use streaming parsing to avoid loading entire arrays into memory.
672
+
673
+
---
674
+
675
+
## 9. IANA Considerations
676
+
677
+
This document recommends the following media type registration:
678
+
679
+
**Media Type:** `application/toon+json`
680
+
681
+
| Field | Value |
682
+
|-------|-------|
683
+
| Type name | application |
684
+
| Subtype name | toon+json |
685
+
| Required parameters | None |
686
+
| Optional parameters | `charset` (default UTF-8) |
687
+
| Encoding considerations | Same as application/json |
688
+
| Security considerations | See Section 8 |
689
+
| Interoperability | Semantically equivalent to application/json when decoded |
690
+
| Published specification | This document and [TOON-SPEC] |
691
+
692
+
**Intended usage:** COMMON
693
+
694
+
**Applications:** Real-time state synchronization, durable streams, bandwidth-constrained systems requiring JSON semantics.
695
+
696
+
---
697
+
698
+
## 10. References
699
+
700
+
### 10.1. Normative References
701
+
702
+
**[STATE-PROTOCOL]**
703
+
Durable Streams State Protocol. ElectricSQL, 2025.
704
+
<https://github.com/durable-streams/durable-streams/blob/main/packages/state/STATE-PROTOCOL.md>
705
+
706
+
**[TOON-SPEC]**
707
+
TOON (Token-Oriented Object Notation) Specification v3.0.
708
+
<https://github.com/toon-format/spec>
709
+
710
+
**[PROTOCOL]**
711
+
Durable Streams Protocol. ElectricSQL, 2025.
712
+
<https://github.com/durable-streams/durable-streams/blob/main/PROTOCOL.md>
713
+
714
+
### 10.2. Informative References
715
+
716
+
**[TOON-FORMAT]**
717
+
TOON Format Website.
718
+
<https://toonformat.dev>
719
+
720
+
---
721
+
722
+
## Appendix A: Payload Reduction Analysis
723
+
724
+
### A.1. Single Change Message
725
+
726
+
| Field | JSON | TOON | Savings |
727
+
|-------|------|------|---------|
728
+
| `type` field | `"type":"user",` | `type: user` | 4 chars (36%) |
729
+
| `key` field | `"key":"user:123",` | `key: user:123` | 6 chars (38%) |
730
+
| `value` object | `"value":{` | `value:` | 7 chars (100%) |
731
+
| `headers` object | `"headers":{` | `headers:` | 9 chars (100%) |
732
+
| `operation` field | `"operation":"insert"` | `operation: insert` | 2 chars (18%) |
733
+
| Closing braces | `}}` | (none) | 2 chars (100%) |
734
+
| **Total** | **~145 bytes** | **~80 bytes** | **~45%** |
735
+
736
+
### A.2. Tabular Batch Scaling
737
+
738
+
| Batch Size | JSON | TOON | Reduction |
739
+
|------------|------|------|-----------|
740
+
| 1 message | 145 bytes | 80 bytes | 45% |
741
+
| 10 messages | 1,450 bytes | 500 bytes | 65% |
742
+
| 100 messages | 14,500 bytes | 4,200 bytes | 71% |
743
+
| 1,000 messages | 145 KB | 40 KB | 72% |
744
+
745
+
### A.3. Snapshot Export (100 users, 500 messages)
746
+
747
+
| Entity Type | JSON | TOON | Reduction |
748
+
|-------------|------|------|-----------|
749
+
| Users (100 records) | ~30 KB | ~11 KB | **63%** |
750
+
| Messages (500 records) | ~150 KB | ~52 KB | **65%** |
751
+
| Snapshot metadata | ~200 bytes | ~120 bytes | **40%** |
752
+
| **Total** | **~180 KB** | **~63 KB** | **65%** |
753
+
754
+
### A.4. Real-World Bandwidth Impact
755
+
756
+
| Application | Messages/sec | JSON Bandwidth | TOON Bandwidth | Monthly Savings |
757
+
|-------------|--------------|----------------|----------------|-----------------|
758
+
| Chat (1M users) | 10 msg/sec | ~2 GB/hr | ~0.7 GB/hr | ~30 GB/month |
759
+
| Presence tracking | 5 updates/sec | ~1 GB/hr | ~0.4 GB/hr | ~18 GB/month |
760
+
| Feature flags | 1 update/min | ~10 MB/day | ~4 MB/day | ~180 MB/month |
761
+
762
+
---
763
+
764
+
## Appendix B: Implementation Guidelines
765
+
766
+
### B.1. Rust Crate Recommendations
767
+
768
+
| Use Case | Recommended Crate | Features |
769
+
|----------|-------------------|----------|
770
+
| Full spec compliance | `toon` (toon-rs) | `de_direct`, `perf_memchr`, `perf_smallvec` |
771
+
| Performance-critical | `toon-rust` | SIMD optimizations |
772
+
| Serde integration | `serde_toon` | `toon!` macro |
773
+
774
+
**Example configuration:**
775
+
```toml
776
+
[dependencies]
777
+
toon = { version = "0.1", features = ["de_direct", "perf_memchr"] }
778
+
```
779
+
780
+
### B.2. Delimiter Selection
781
+
782
+
| Delimiter | Best For | Quoting Impact |
783
+
|-----------|----------|----------------|
784
+
| Comma (default) | General purpose | May require quotes for values with commas |
785
+
| Tab | Data with few quoted strings | ~10-15% better efficiency |
786
+
| Pipe | Data with commas and tabs | Similar to comma |
787
+
788
+
### B.3. Streaming Tabular Arrays
789
+
790
+
1. Read array header to get `[N]` and field declarations
791
+
2. Pre-allocate buffers for N rows (with limits)
792
+
3. Parse rows sequentially, validating field count
793
+
4. Decode each row into change message
794
+
5. Apply messages immediately (don't wait for full array)
795
+
796
+
### B.4. Error Handling
797
+
798
+
| Error Condition | Recommended Action |
799
+
|-----------------|-------------------|
800
+
| Row count ≠ `[N]` | Reject batch, request replay from last good offset |
801
+
| Wrong field count | Skip row, log error, continue |
802
+
| Decode failure | Skip row, log error, continue |
803
+
804
+
### B.5. Hybrid Encoding Strategy
805
+
806
+
For maximum efficiency, implementations **MAY**:
807
+
1. Accumulate change messages by type
808
+
2. Encode each type batch as tabular array
809
+
3. Interleave control messages as individual objects
810
+
4. Concatenate into single TOON document
811
+
812
+
---
813
+
814
+
## Appendix C: Migration Guide
815
+
816
+
### C.1. Server-Side Changes
817
+
818
+
**Add TOON encoding capability:**
819
+
```typescript
820
+
import { stringify } from 'toon'
821
+
822
+
function encodeMessages(messages: ChangeMessage[], format: 'json' | 'toon'): string {
823
+
if (format === 'toon') {
824
+
return stringify(messages, { delimiter: ',' })
825
+
}
826
+
return JSON.stringify(messages)
827
+
}
828
+
```
829
+
830
+
**Support content negotiation:**
831
+
```typescript
832
+
app.get('/stream/:id', (req, res) => {
833
+
const format = req.accepts('application/toon+json', 'application/json')
834
+
res.format({
835
+
'application/toon+json': () => {
836
+
res.type('application/toon+json')
837
+
res.send(encodeToTOON(messages))
838
+
},
839
+
'application/json': () => {
840
+
res.send(JSON.stringify(messages))
841
+
}
842
+
})
843
+
})
844
+
```
845
+
846
+
### C.2. Client-Side Changes
847
+
848
+
```typescript
849
+
const stream = await DurableStream.create({
850
+
url: 'https://server.com/v1/stream/my-app',
851
+
contentType: 'application/toon+json' // Request TOON
852
+
})
853
+
854
+
res.subscribeText(async (batch) => {
855
+
const messages = parse(batch) // TOON decoder
856
+
for (const msg of messages) {
857
+
state.apply(msg)
858
+
}
859
+
})
860
+
```
861
+
862
+
### C.3. Phased Rollout
863
+
864
+
| Phase | Description | Risk |
865
+
|-------|-------------|------|
866
+
| **Phase 1** | Add TOON support, keep JSON default | Low |
867
+
| **Phase 2** | Roll out TOON clients gradually | Medium |
868
+
| **Phase 3** | Make TOON default, JSON fallback | Low |
869
+
| **Phase 4** | Deprecate JSON (future) | High |
870
+
871
+
### C.4. A/B Testing Metrics
872
+
873
+
| Metric | Expected Change |
874
+
|--------|-----------------|
875
+
| Bandwidth | 40-60% reduction |
876
+
| Encoding latency | 10-20% increase |
877
+
| Decoding latency | 5-10% decrease |
878
+
| Error rates | No change |
879
+
| Memory usage | 20-30% reduction |