# binary encoding

patterns for encoding/decoding binary wire formats (CBOR, CAR, protocol frames). distinct from JSON - you're working with raw bytes and need to handle endianness, varints, and content addressing.

## anytype writer for encoders

the core pattern: an encoder function that accepts any writer via `anytype`. this lets the same encoder write to fixed buffers, ArrayLists, or any other writer:

```zig
pub fn encode(allocator: Allocator, writer: anytype, value: Value) !void {
    switch (value) {
        .unsigned => |v| try writeArgument(writer, 0, v),
        .text => |t| {
            try writeArgument(writer, 3, t.len);
            try writer.writeAll(t);
        },
        .map => |entries| {
            // sort keys (DAG-CBOR determinism), needs allocator
            const sorted = try allocator.dupe(MapEntry, entries);
            defer allocator.free(sorted);
            std.mem.sort(MapEntry, sorted, {}, keyLessThan);
            // ...
        },
        // ...
    }
}
```

the allocator parameter is separate from the writer - needed for temporary allocations during encoding (sorting map keys, building intermediate buffers), not for the output itself.

usage with different writers:

```zig
// fixed buffer (no allocation for output)
var buf: [1024]u8 = undefined;
var stream = std.io.fixedBufferStream(&buf);
try encode(alloc, stream.writer(), value);
const result = stream.getWritten();

// growable buffer
var list: std.ArrayList(u8) = .{};
defer list.deinit(alloc);
try encode(alloc, list.writer(alloc), value);
```

**note**: `std.io.fixedBufferStream` is deprecated in 0.15 — the stdlib says to use `std.Io.Writer.fixed` / `std.Io.Reader.fixed` instead. the old API still compiles (zat uses it in 3 files) but new code should prefer the non-deprecated form. the `anytype` writer pattern itself is fine either way — the encoder doesn't care which writer type backs it.

see: [zat/cbor.zig](https://tangled.sh/@zzstoatzz.io/zat/tree/main/src/internal/cbor.zig)

## encodeAlloc convenience

wrap the growable-buffer pattern into a helper:

```zig
pub fn encodeAlloc(allocator: Allocator, value: Value) ![]u8 {
    var list: std.ArrayList(u8) = .{};
    errdefer list.deinit(allocator);
    try encode(allocator, list.writer(allocator), value);
    return try list.toOwnedSlice(allocator);
}
```

caller owns the returned slice. `errdefer` ensures cleanup if encoding fails partway through.

## big-endian integers without writeInt

when writing fixed-width big-endian integers to an `anytype` writer, build the bytes manually rather than depending on `writeInt` (which may not be available on all writer types):

```zig
fn writeArgument(writer: anytype, major: u3, val: u64) !void {
    const prefix: u8 = @as(u8, major) << 5;
    if (val <= 0xffff) {
        try writer.writeByte(prefix | 25);
        const v: u16 = @intCast(val);
        try writer.writeAll(&[2]u8{ @truncate(v >> 8), @truncate(v) });
    }
    // ...
}
```

`@truncate` on shifted values is the idiomatic way to extract individual bytes.

## unsigned varint (LEB128)

used by CID, CAR, and other IPLD formats for variable-length integers:

```zig
// write
pub fn writeUvarint(writer: anytype, val: u64) !void {
    var v = val;
    while (v >= 0x80) {
        try writer.writeByte(@as(u8, @truncate(v)) | 0x80);
        v >>= 7;
    }
    try writer.writeByte(@as(u8, @truncate(v)));
}

// read
fn readUvarint(data: []const u8, pos: *usize) ?u64 {
    var result: u64 = 0;
    var shift: u6 = 0;
    while (pos.* < data.len) {
        const byte = data[pos.*];
        pos.* += 1;
        result |= @as(u64, byte & 0x7f) << shift;
        if (byte & 0x80 == 0) return result;
        shift +|= 7;
        if (shift >= 64) return null;
    }
    return null;
}
```

note `+|=` (saturating add) prevents overflow on the shift counter.

## arena per message

for streaming protocols, create an arena per incoming message. all decoding allocations go into it, then free everything at once:

```zig
pub fn serverMessage(self: *Self, data: []const u8) !void {
    var arena = std.heap.ArenaAllocator.init(self.allocator);
    defer arena.deinit();

    const event = decodeFrame(arena.allocator(), data) catch |err| {
        log.debug("decode error: {s}", .{@errorName(err)});
        return;
    };

    self.handler.onEvent(event);
    // arena freed here — all decoded data is gone
}
```

this means the handler's `onEvent` must not hold references to event data past the call. if it needs to, it must copy into its own allocator.

see: [zat/firehose.zig](https://tangled.sh/@zzstoatzz.io/zat/tree/main/src/internal/firehose.zig), [zat/jetstream.zig](https://tangled.sh/@zzstoatzz.io/zat/tree/main/src/internal/jetstream.zig)

## specialized decoders

when generic decoding is too expensive, write a purpose-built parser for a known schema. the generic path builds `Value` unions, `MapEntry` arrays, and handles every CBOR type. if you know the exact shape, skip all that.

example: MST nodes are always `map(2) { "e": array[entries...], "l": CID|null }`. instead of `cbor.decodeAll()` → extract fields from Value unions, parse the CBOR bytes directly:

```zig
pub fn decodeMstNode(allocator: Allocator, data: []const u8) MstDecodeError!MstNodeData {
    // expect map(2), key "e", array(n) — known byte sequence
    // parse entries inline, zero-copy slicing into input buffer
    // only allocation: the entries array itself
}

pub const MstNodeData = struct {
    left: ?[]const u8,       // raw CID bytes (borrowed from input)
    entries: []MstEntryData, // heap-allocated array
};

pub const MstEntryData = struct {
    prefix_len: usize,
    key_suffix: []const u8,  // borrowed from input
    value_cid: []const u8,   // borrowed from input
    tree: ?[]const u8,       // borrowed from input
};
```

the result: MST walk went from 45.5ms (generic decode per node) to 39.3ms (specialized decode) on 243k blocks. the bigger win was avoiding the full tree rebuild (218ms → 39ms total) by verifying structure during the walk.

when to use this pattern:
- you decode the same schema thousands of times (MST nodes, CBOR blocks)
- the schema is stable and well-known
- profiling shows decode time dominates

when NOT to use it:
- the schema varies or is user-defined
- you only decode a handful of times
- generic decode is fast enough

see: [zat/mst.zig decodeMstNode](https://tangled.sh/@zzstoatzz.io/zat/tree/main/src/internal/repo/mst.zig)

## deterministic encoding

DAG-CBOR requires deterministic output (same value → same bytes). the main rules:

- **shortest integer encoding**: 0-23 inline, 24-255 in 1 byte, etc.
- **map keys sorted**: by byte length first, then lexicographically
- **no floats, no indefinite lengths**

sorting map keys during encoding:

```zig
fn dagCborKeyLessThan(_: void, a: MapEntry, b: MapEntry) bool {
    if (a.key.len != b.key.len) return a.key.len < b.key.len;
    return std.mem.order(u8, a.key, b.key) == .lt;
}

// in encoder:
const sorted = try allocator.dupe(MapEntry, entries);
defer allocator.free(sorted);
std.mem.sort(MapEntry, sorted, {}, dagCborKeyLessThan);
```

the dupe + sort pattern avoids mutating the input — the caller's `entries` slice stays unchanged.