commits
Sweep ocaml-scc, ocaml-scitt, ocaml-sdls, ocaml-sdnv, ocaml-sgp4,
ocaml-sle, ocaml-spake2, ocaml-sqlite, ocaml-srp to declare every
test-scope dependency that is actually used at test time.
The block had a missing newline before [Sqlite.Table.put] (running
the [create] line into the next call) and free [fs]. Wrap as
[run ~fs = ...], assert the [Sqlite.find] roundtrip, and add
[eio.core] to the mdx libraries.
Remove libraries declared in '(libraries ...)' clauses but unreferenced
by any module in the same source tree, as flagged by 'monopam lint'
after the new Dead_lib detection landed. Touches 131 dune files across
~80 packages.
A few stanzas needed a positive correction instead of a pure removal:
- ocaml-git/bin/diag: depended on eio_main + bytesrw-eio for an
Eio_posix.run call site; the umbrella was overkill, switch to the
precise eio_posix package.
- ocaml-scaleway/lib, ocaml-s3/lib: scaleway.mli / s3.mli reference
Eio_unix.Stdenv.base; eio.unix is required and was missing.
- merlint/lib: pulled bytesrw + nox-opam.bytesrw to surface
Opam_bytesrw, used by rule e915 and lint helpers.
Stanzas where Dead_lib was a false positive (transitive dep needed
for module visibility, virtual-library impls) are left untouched —
e.g. helix.jx.jsoo for ocaml-globe/demo retains its (libraries ...)
entry because it provides the impl of the helix.jx virtual lib.
The READMEs all share the standard install/overlay snippet, but the
sh blocks lacked the "<!-- $MDX skip -->" directive. `dune test`
would shell out to `opam install` against the live switch, which
either prompts interactively or fails with a package conflict —
either way diffing as a test failure.
Bulk-add skip directives in front of every install/overlay block.
Also collapse the doubled "non-deterministic + skip" stack on three
READMEs (memtrace, ocaml-dpop, ocaml-pid1, ocaml-yaml, merlint) where
`skip` already implies the runtime is bypassed.
Twelve docs that no longer track real work or describe the current
fork. Removed:
ocaml-tls/{attacks,sni,design}.md — pre-fork architecture notes from
mirleft/ocaml-tls that reference 2014 CVEs, polarssl, Lwt/Async
flows; nothing in the current Eio-based fork's design.
memtrace/CONTRIBUTING.md — Jane Street upstream guide pointing at
github.com PRs, doesn't apply on tangled.
ocaml-requests/HTTP2_{IMPLEMENTATION_PLAN,CONPOOL_INTEGRATION}.md —
1500 lines combined for an HTTP/2 rollout that hasn't moved in 6+
weeks; revive from git history if/when the work restarts.
merlint/TODO.md, merlint/todo/{new_rules,codes,duplication}.md,
ocaml-sqlite/TODO.md, prune/TODO.md — stale TODO/proposal lists,
last touched 6+ weeks ago. Active TODOs (monopam, irmin, jwt,
cookie, chor, requests SPEC-TODO, claude ARCHITECTURE) all kept.
Also drop the now-empty merlint/todo/ directory and roll in dune fmt
fallout for irmin/test/bench/dune.
Renames 35 packages to make blacksun forks distinguishable from their
opam-repository upstreams. Module names (Git.x, Tls.x, ...) stay bare;
opam package names and dune (public_name) findlib references move to
nox-X. After this commit, zero local package names overlap with
opam-repository.
Renamed:
- nox-git, nox-irmin
- nox-crypto, nox-crypto-pk, nox-crypto-rng, nox-crypto-ec
- nox-tls, nox-tls-eio, nox-tar, nox-tar-eio, nox-tty, nox-tty-eio
- nox-arp, nox-ca-certs, nox-cbor, nox-cookie, nox-crc, nox-csv
- nox-gpt, nox-hkdf, nox-http, nox-jwt, nox-kdf, nox-loc
- nox-memtrace, nox-pds, nox-sexp, nox-slack, nox-toml
- nox-websocket, nox-x509, nox-xdge, nox-yaml
Also drops orphan tar-mirage and tar-unix opam templates that had no
matching package stanza.
Pure formatting changes from `dune fmt`: doc comment placement moves
from above the binding to below it for `type`s, multi-line `match`
expressions collapse onto one line where they fit, and infix operator
applications pick up spaces (`Soup.($?)` -> `Soup.( $? )`). No
semantic changes.
[lib/dune] has a [(menhir ...)] stanza and [dune-project] declares
[(using menhir 3.0)], but [sqlite.opam] did not list [menhir] so
[dune build -p sqlite] failed with "Program menhir not found". Same
story for the README [(mdx)] stanza.
Also correct a stale comment: the SQL lexer uses [ocamllex] (see
[lib/lexer.mll]), not sedlex.
Object combinators: [Object.mem] -> [Object.member], [Object.opt_mem]
-> [Object.opt_member], [Object.case_mem] -> [Object.case_member]. The
sibling submodules [Object.Mem] / [Object.Mems] become
[Object.Member] / [Object.Members]. RFC 8259 §4 calls these
"name/value pairs, referred to as the members", so mirror the spec
name rather than the shortened [mem].
[Object.finish] -> [Object.seal]. "Seal" reads as "close the map, no
more members added", which is what the operation does.
Value constructors/queries: [Value.mem] (function) -> [Value.member];
[Value.mem_find] -> [Value.member_key]; [Value.mem_names] ->
[Value.member_names]; [Value.mem_keys] -> [Value.member_keys].
[type mem = ...] -> [type member = ...]; [type object'] still points
at [member list].
Downstream (~80 files across slack, sbom, stripe, sigstore, requests,
claude, irmin, freebox) updated via perl-pie. dune build clean,
dune test ocaml-json clean.
Follow up to the module rename: update the remaining callers that
still referenced [Err] (library [claude.ml{,i}], [client.ml], the test
driver [test.ml]), and fix one stray [^ e] string concatenation in
hermest's CLI that needed [Json.Error.to_string e] now that
[Json.of_string] yields a structured error.
Warning 69 (unused-field, mutable-never-assigned). Four independent
record fields were flagged as mutable but the code only mutates their
referents in place, never rebinds the record slot itself:
- ocaml-wal/lib/wal.ml: [t.file] (the Eio file resource; methods call
Eio.File.pwrite_all etc., the slot is set once at open time).
- ocaml-block/lib/block.ml: [Memory.state.data] (the backing bytes,
written via Bytes.blit_string; [Bytes.t] is already mutable).
- ocaml-sse/lib/sse.ml: [Parser.t.data_buf] (a Buffer.t, written via
Buffer.add_*; the slot never changes).
- ocaml-zephyr/lib/zephyr.ml: drop [mode : Read | Write] entirely —
set at open-time, read nowhere. The open_read / open_write
constructors already distinguish the two call shapes, so mode
tracking was redundant.
read_master: extract a small classify_master_row helper that returns a
[Mtable | Mautoindex | Mextra | Mskip] variant. The big match/for-loop
shrinks to a flat tag dispatch.
create_table: stop substring-scanning the SQL for the table name. The
lexer/parser already produces an Ast.create_table value, so use
[ct.tbl_name] directly. Drops ~50 lines of fragile string fiddling.
Test cleanups: rename 9 tests with 5+ underscores to shorter forms.
Narrow the [with _ -> ...] handlers in test_sqlite.ml to
[Sqlite.Unique_violation _ | Failure _], and the inlined hostile suite
adds Invalid_argument / Sys_error / End_of_file via a [safe_exn]
predicate so unexpected exceptions still surface.
Inline test_hostile.ml content into test_sqlite.ml as flat helpers /
test functions plus a [hostile_cases] list appended to the main suite.
Drops the "t" suffix and follows the value/codec/toml/core pattern
(jsont.json_base style). The internal raw TOML module moves from
[Toml] to [Value] (file: lib/value.ml, was lib/toml.ml) to make room
for the top-level Toml facade (file: lib/toml.ml, was lib/tomlt.ml).
External callers now reach the raw AST through [Toml.Value.X] instead
of [Tomlt.Toml.X]. Every downstream reference updated in lockstep.
The CREATE TABLE lexer matched on ASCII characters only — punctuation,
keywords, identifiers, numbers, and quoted strings whose body bytes
pass through verbatim. None of sedlex's Unicode classes were used, so
sedlex.ppx was paying for nothing while merlint flagged every
PPX-expanded [__sedlex_state_*] / [__sedlex_partition_*] identifier as
double-underscore module access (~68 spurious E335 hits).
Convert lib/lexer.ml -> lib/lexer.mll. The grammar is unchanged: same
keyword table, same quoted-identifier and single-quoted string readers
(now ocamllex sub-rules), same NUMBER fallback for operators / passthrough
bytes (UTF-8 round-trips because ocamllex's [_] matches any byte).
Drop sedlex, sedlex.ppx, and menhirLib from the library deps; menhir's
traditional API plugs into [Parser.create_table token lexbuf] directly.
Generate .opam.template files with x-quality-* fields based on
detected package features:
- x-quality-build: has lib/ with .ml files
- x-quality-test: has test/ with .ml files
- x-quality-fuzz: has fuzz/ with .ml files
- x-quality-interop: has test/interop/ directory
- x-quality-cram: has test/*.t/ directories
These fields are picked up by dune's opam generation and will be
checked by merlint E910 for consistency.
Also: add fmt dep to ocaml-sse/lib/dune (Fmt.pf used without dep).
monopam quality — scans packages for quality features, caches by
git commit hash. 166 packages: build=163, test=162, fuzz=94,
interop=39, doc=42.
Standard vocabulary based on crates.io categories, erratique/opam
conventions, and monorepo domain coverage:
Org: org:blacksun
Domain: aerospace, codec, crypto, network, storage, git, merkle
Purpose: cli, test, bench, format, log, system
Protocol: ccsds, uslp, cop1, sdls, sle, atproto, tls, http, json, binary
Cross-cutting: eio, simulation, math, compression
Tags placed in dune-project (package ...) stanzas via (tags ...).
Propagated to .opam files by dune's opam generation.
- Update .ocamlformat to 0.29.0 across all 591 files
- csvt: reuse single Buffer.t for field reads (no alloc per field)
- sexpt: Obj members decoded from stream into Dict, typed Variant GADT
- Reformat all source files for 0.29.0
- ocaml-cookeio: wrap test_clear in Eio_main.run to provide Eio context
- ocaml-pds: remove explicit Pds.close calls that double-closed the DB
when Eio.Switch.on_release already handles cleanup
- ocaml-sqlite: make sw field mutable; clear it in close so sync takes
the direct pager-write path instead of opening a new WAL on a dying
switch (fixes irmin's 17 failures and pds's "Switch finished!" errors)
- ocaml-spacedata: improve fetch_json timeout error message with URL
(2 live tests still fail due to CelesTrak being unreachable)
160 new tests exercising security-critical code paths identified by
mapping known CVEs from C/reference implementations to our OCaml code:
- ocaml-sqlite (9): cyclic pages, oversized varints, record overflow,
wrong page kind, truncated WAL, out-of-bounds root, garbage files
- ocaml-cbort (12): deep nesting (CVE-2025-24302), indefinite-length
DoS, integer overflow in lengths, truncated input, invalid types
- ocaml-tar (10): path traversal (CVE-2021-32803), symlink escape
(CVE-2025-45582), oversized octal, truncated headers, checksum
- ocaml-http (14): CRLF header injection (CWE-113), null bytes,
Content-Length overflow, empty/duplicate headers
Also hardens validate_header_name_str to reject null bytes/empty names
- ocaml-jsonwt (21): "none" algorithm bypass (CVE-2015-9235) case
variations, algorithm confusion (CVE-2016-10555), malformed headers,
empty segments, extra dots, large payloads
- ocaml-cose (8): algorithm substitution, missing algorithm header,
malformed CBOR, wrong types, label overlap (RFC 9052)
- ocaml-git (18): tree path traversal, null bytes, symlink mode,
malformed tree data, pack delta attacks, pack format validation
- ocaml-tomlt (25): duplicate keys, integer overflow, malformed dates
(invalid month/day/hour/minute), deep nesting, long strings
- ocaml-squashfs (20): symlink traversal edge cases, fragment table
bounds, inode self-reference, compression bomb limits, bad superblock
- ocaml-cpio (23): symlink target validation, null bytes in filenames,
oversized filesize, truncated archives, invalid magic numbers
- PRIMARY KEY on non-INTEGER columns now enforces UNIQUE (tests 91-93).
Added has_primary_key field to Ast.column_def; parse_unique_constraints
now treats both column-level and table-level PRIMARY KEY as UNIQUE.
- create_table rejects duplicate table names instead of silently creating
a second entry (test 97).
- Table.create ~name:"kv" reuses the existing default kv table instead of
creating a duplicate (test 98).
- Quoted table names ("my table", [my table], `my table`) are now properly
unquoted during CREATE TABLE parsing (test 99).
- Add memtrace instrumentation to test runner.
monotonic_now: increment changed from 1μs to 1ms, drift capped at 300s.
Rejects registration if clock drift exceeds cap instead of silently
issuing synthetic timestamps indefinitely.
verify/verify_receipt_only now return receipt_summary with total,
verified, failed, skipped counts so callers can detect partial
verification (e.g. 2 of 3 receipts tampered).
bottle_checksums split on the last "-" to extract the binary name,
but date-hash versions (20260401-d14d5fc) have a "-" in them. This
produced names like "cascade-20260401" instead of "cascade".
Now finds the first "-" followed by a digit (the version's YYYYMMDD
prefix), correctly splitting "cascade-20260401-d14d5fc..." into
name="cascade" and version="20260401-d14d5fc...".
Convert remaining bare Mutex.lock/unlock patterns to
Eio.Mutex.use_rw/use_ro in cookeio, requests, oci, and space.
Fix unattached doc comment in vds.ml Node_cache.
Read .git/HEAD and follow the ref to get the commit hash. No
subprocess, no dependency on git being in PATH. Uses Bos.OS.File.read
which is already a dependency.
git pull --ff-only fails when the remote tap repo has no commits
(empty repo, no main branch). Check rev-parse origin/main first;
if it fails, skip the pull.
Table operations now read the B-tree directly instead of a hash table
cache populated once at open time. This fixes multi-handle safety:
a second handle can no longer bypass duplicate-key checks or collide
on rowids.
Compact.append and Compact.root used asr (arithmetic shift right)
which preserves the sign bit. Values are non-negative by construction,
but lsr matches the unsigned division semantics of the algorithm and
is consistent with verify_inclusion in proof.ml.
E331 now checks ctx.config.allowed_words before flagging a name,
matching the pattern used by E300. This lets projects exempt specific
names like create_table via .merlint:
allowed_words: [create_table]
Updated the man page to document allowed_words and acronyms.
Per alcobar's README, corpus generation uses fuzz.exe --gen-corpus,
not a separate gen_corpus.ml. Updated E718 and E724 rules to check
for --gen-corpus in the dune fuzz rule instead of gen_corpus.ml/.exe.
Removed gen_corpus from the valid fuzz filename list.
Also: add .merlint to ocaml-sqlite excluding E331 for create_table,
and doc comments on test suite mli files (E405).
import used List.find_opt on raw map pairs, so a duplicate
"version" or "entries" key would shadow later occurrences.
Now checks for byte-equal duplicate keys before field extraction,
consistent with the receipt parser's cbor_check_unique_keys.
- Remove underscore prefix from used binding in lexer.ml (E335)
- Add test_ast.ml and test_lexer.ml for schema parsing coverage (E605)
- Replace hand-written gen_corpus.ml with alcobar --gen-corpus rule
in ocaml-sqlite, ocaml-scitt, and ocaml-cdm fuzz dirs (E718)
The append sequence (push_hash → compact.append → put_entry) writes
3 keys to SQLite. A crash between them left the database inconsistent.
Now wrapped in Sqlite.with_transaction — all 3 writes commit or none
do. In-memory backend uses a no-op atomic wrapper.
Add Pager.snapshot/rollback for capturing and restoring dirty page
state. Expose Table.save_root/restore_root and Index.save_root/
restore_root for B-tree root page rollback after splits.
Build Sqlite.with_transaction on top: snapshots pager state, B-tree
roots, and in-memory KV caches before running the user function. On
exception, all state is rolled back and the exception re-raised.
Five new tests cover commit, KV rollback, unique index rollback,
and nested transaction failure.
Add missing opam dependencies across 17 packages (vec3, bytesrw, http,
cbort, fmt, etc.) and remove unused deps (requests from respond, menhir
from sqlite). Teach monopam lint to recognize (menhir ...) dune stanzas
as requiring the menhir package. Move bytesrw dep from scitt to
atp-lexicon-scitt where it actually belongs.
Ensure all 67 fuzz/dune files include gen_corpus.exe in the (alias fuzz)
rule deps for AFL corpus generation. Adds both missing runtest and fuzz
rules to ocaml-cose which had neither.
Renames: find_vds_info → vds_info, make_unique_indexes → unique_indexes,
create_new → new_db. Extract err_userinfo_http, err_userinfo_parse,
err_userinfo_empty_uid helpers for consistent error formatting (E340).
Extract exchange_code and find_or_create_user from the 87-line
callback_route (E005). Extract index_row helper to reduce nesting
depth in populate_unique_indexes from 5 to 4 (E010).
- parser.mly: conflict-free LR(1) grammar; handles structure (parens,
commas, table-vs-column dispatch)
- lexer.ml: sedlex tokenizer with case-preserving keyword tokens,
4 quoting styles, SQL comments
- ast.ml: column body classification (type affinity + constraints)
done in OCaml, not the grammar
Also fixes:
- CONSTRAINT name UNIQUE(...) now enforced (named table constraints)
- NULL columns skip UNIQUE checks per SQL standard (multiple NULLs
allowed under UNIQUE)
create_user now catches Unique_violation from the identity insert and
deletes the just-inserted user row before re-raising. Added delete_row
to ocaml-sqlite for generic table row deletion (with index cleanup).
Splits the 117-line open_ function into smaller pieces to satisfy
merlint E005 (long function threshold).
Indexes are now stored as Btree.Index B-trees in the database file and
registered in sqlite_master as type "index" (sqlite_autoindex_<tbl>_<n>),
matching real SQLite behavior. No more full table scan on reopen.
ocaml-sqlite: Parse and enforce UNIQUE constraints (column-level and
table-level) via in-memory hash indexes. Raises Unique_violation on
conflict. Indexes are rebuilt on database reopen.
ocaml-auth: Add UNIQUE(provider, provider_uid) to oauth_identities
schema. Callback now catches Unique_violation on concurrent creates
and falls back to the existing identity.
ocaml-oauth: Classify parse_token_response errors into Invalid_json,
Missing_access_token, and Invalid_token_format instead of collapsing
all failures to Invalid_json.
1. Sqlite: silent data loss on corrupt DB — Store.v caught all exceptions
and fell back to Sqlite.v (which truncates). Now only catches Eio.Io
(file not found). Remove Sqlite.v; add Sqlite.open_ ~create flag.
2. Sign-out: now revokes server-side session via Store.delete_session.
Previously only cleared the browser cookie — copied sid stayed valid.
Changed to POST /auth/signout with header-based session lookup.
3. OAuth: authorization URL now includes response_type=code per RFC 6749.
4. OAuth: token exchange/refresh now uses application/x-www-form-urlencoded
per RFC 6749, not JSON. Removed JSON-body functions, added form_encode
helper and exchange_form_body/refresh_form_body returning strings.
5. Auth: callback now uses provider.userinfo_url instead of hardcoded
GitHub API. Added userinfo_url to Oauth.provider type. Google, GitLab,
and custom providers can now complete login.
ocaml-kvn: CCSDS Key-Value Notation parser.
ocaml-ocm: CCSDS 505.0-B-2 Orbit Comprehensive Message parser.
Tested against TraCSS LANDSAT 5 OCM file.
Add generic write API: create_table parses CREATE TABLE SQL to register
new tables, insert writes rows with auto or explicit rowids (including
INTEGER PRIMARY KEY handling). 11 new tests cover all value types,
persistence, coexistence with KV API, and error cases.
Fix invalid odoc markup in 54 files: convert {\!Module} to {!Module}
in fuzz .mli files, replace inline {v ... v} with [...] code spans,
fix "paragraph should begin on its own line" warnings, escape bare
brackets, and resolve ambiguous docstring placement (warning 50).
- Remove vendored crowbar/ directory
- Replace all Crowbar references with Alcobar across 176 .ml files
- Update all fuzz dune files: crowbar → alcobar in libraries
- Remove 77 gen_corpus.ml files (alcobar handles corpus internally)
- Update dune-project files: crowbar → alcobar in dependencies
- Update merlint rules (e705, e726): Crowbar → Alcobar in checks,
docs, and examples
- Update merlint generated docs (index.html)
428 files changed, ~1200 lines removed net.
The 20-byte reserved-for-expansion region was not being zeroed,
causing non-zero bytes from the reused page buffer to leak into
the database header. Found by the spec header values test.
B-tree tests (56 total):
- Overflow thresholds: X=U-35, M=((U-12)*32/255)-23 for all page sizes
- Boundary: exact max_local (4061), X+1 (4062), multi-page overflow
- All valid page sizes (512..65536) with and without overflow
- Page type flags, header sizes, varint 9-byte encoding
- Record serial types (NULL, int8/16/24/32/48/64, float, text, blob)
- Stress: 50 cells of increasing size with overflow + splits
SQLite tests (56 total):
- Database header byte-level: magic string, page size, payload fractions
(64/32/32), schema format, text encoding (UTF-8), reserved bytes
- change_counter == version_valid_for validity check
- Page 1 B-tree header at offset 100 (type=0x0d, fragmented<=60)
- Overflow value roundtrip and persistence across close/reopen
- Rename find_matching_paren -> matching_paren, find_table -> table
- Extract add_paren_content and extract_named_kv_tables helpers
- Add pp for type t; add err_kv_pair helper in sql.ml
- Parsing functions (int_of_string, float_of_string, Int64.of_string):
catch Failure _ instead of _
- TLS inhibit/try_write_t: use _exn to name the caught exception
- sql open_or_create: catch Failure _ | Eio.Io _
- run_tests json_equal: catch Failure _ | Invalid_argument _
Extract helpers: pp_column, pp_schema, write_entry, delete_entry,
sum_int_values to bring nesting below threshold of 4.
- ocaml-sqlite: merge generic_suite into suite, single export in .mli
- ocaml-tcpcl: add missing test_tcpcl.mli
- ocaml-tls: rename test_rng to mock_rng (helper, not test module)
- ocaml-tls: restructure test_tls_eio/unix stubs with test.ml runners
- ocaml-tomlt: restructure test_tomlt_{bytesrw,eio,unix} stubs with
test.ml runners and .mli files
open_ no longer requires a 'kv' table — it parses CREATE TABLE SQL from
sqlite_master to build schemas for every table. New generic read API
(tables, iter_table, fold_table, read_table) lets callers read any SQLite
database. INTEGER PRIMARY KEY columns get rowid substitution, and trailing
NULLs are padded per the SQLite storage optimization.
CLI gains `sql tables DB` and `sql read DB -t TABLE` subcommands.
Remove libraries declared in '(libraries ...)' clauses but unreferenced
by any module in the same source tree, as flagged by 'monopam lint'
after the new Dead_lib detection landed. Touches 131 dune files across
~80 packages.
A few stanzas needed a positive correction instead of a pure removal:
- ocaml-git/bin/diag: depended on eio_main + bytesrw-eio for an
Eio_posix.run call site; the umbrella was overkill, switch to the
precise eio_posix package.
- ocaml-scaleway/lib, ocaml-s3/lib: scaleway.mli / s3.mli reference
Eio_unix.Stdenv.base; eio.unix is required and was missing.
- merlint/lib: pulled bytesrw + nox-opam.bytesrw to surface
Opam_bytesrw, used by rule e915 and lint helpers.
Stanzas where Dead_lib was a false positive (transitive dep needed
for module visibility, virtual-library impls) are left untouched —
e.g. helix.jx.jsoo for ocaml-globe/demo retains its (libraries ...)
entry because it provides the impl of the helix.jx virtual lib.
The READMEs all share the standard install/overlay snippet, but the
sh blocks lacked the "<!-- $MDX skip -->" directive. `dune test`
would shell out to `opam install` against the live switch, which
either prompts interactively or fails with a package conflict —
either way diffing as a test failure.
Bulk-add skip directives in front of every install/overlay block.
Also collapse the doubled "non-deterministic + skip" stack on three
READMEs (memtrace, ocaml-dpop, ocaml-pid1, ocaml-yaml, merlint) where
`skip` already implies the runtime is bypassed.
Twelve docs that no longer track real work or describe the current
fork. Removed:
ocaml-tls/{attacks,sni,design}.md — pre-fork architecture notes from
mirleft/ocaml-tls that reference 2014 CVEs, polarssl, Lwt/Async
flows; nothing in the current Eio-based fork's design.
memtrace/CONTRIBUTING.md — Jane Street upstream guide pointing at
github.com PRs, doesn't apply on tangled.
ocaml-requests/HTTP2_{IMPLEMENTATION_PLAN,CONPOOL_INTEGRATION}.md —
1500 lines combined for an HTTP/2 rollout that hasn't moved in 6+
weeks; revive from git history if/when the work restarts.
merlint/TODO.md, merlint/todo/{new_rules,codes,duplication}.md,
ocaml-sqlite/TODO.md, prune/TODO.md — stale TODO/proposal lists,
last touched 6+ weeks ago. Active TODOs (monopam, irmin, jwt,
cookie, chor, requests SPEC-TODO, claude ARCHITECTURE) all kept.
Also drop the now-empty merlint/todo/ directory and roll in dune fmt
fallout for irmin/test/bench/dune.
Renames 35 packages to make blacksun forks distinguishable from their
opam-repository upstreams. Module names (Git.x, Tls.x, ...) stay bare;
opam package names and dune (public_name) findlib references move to
nox-X. After this commit, zero local package names overlap with
opam-repository.
Renamed:
- nox-git, nox-irmin
- nox-crypto, nox-crypto-pk, nox-crypto-rng, nox-crypto-ec
- nox-tls, nox-tls-eio, nox-tar, nox-tar-eio, nox-tty, nox-tty-eio
- nox-arp, nox-ca-certs, nox-cbor, nox-cookie, nox-crc, nox-csv
- nox-gpt, nox-hkdf, nox-http, nox-jwt, nox-kdf, nox-loc
- nox-memtrace, nox-pds, nox-sexp, nox-slack, nox-toml
- nox-websocket, nox-x509, nox-xdge, nox-yaml
Also drops orphan tar-mirage and tar-unix opam templates that had no
matching package stanza.
[lib/dune] has a [(menhir ...)] stanza and [dune-project] declares
[(using menhir 3.0)], but [sqlite.opam] did not list [menhir] so
[dune build -p sqlite] failed with "Program menhir not found". Same
story for the README [(mdx)] stanza.
Also correct a stale comment: the SQL lexer uses [ocamllex] (see
[lib/lexer.mll]), not sedlex.
Object combinators: [Object.mem] -> [Object.member], [Object.opt_mem]
-> [Object.opt_member], [Object.case_mem] -> [Object.case_member]. The
sibling submodules [Object.Mem] / [Object.Mems] become
[Object.Member] / [Object.Members]. RFC 8259 §4 calls these
"name/value pairs, referred to as the members", so mirror the spec
name rather than the shortened [mem].
[Object.finish] -> [Object.seal]. "Seal" reads as "close the map, no
more members added", which is what the operation does.
Value constructors/queries: [Value.mem] (function) -> [Value.member];
[Value.mem_find] -> [Value.member_key]; [Value.mem_names] ->
[Value.member_names]; [Value.mem_keys] -> [Value.member_keys].
[type mem = ...] -> [type member = ...]; [type object'] still points
at [member list].
Downstream (~80 files across slack, sbom, stripe, sigstore, requests,
claude, irmin, freebox) updated via perl-pie. dune build clean,
dune test ocaml-json clean.
Follow up to the module rename: update the remaining callers that
still referenced [Err] (library [claude.ml{,i}], [client.ml], the test
driver [test.ml]), and fix one stray [^ e] string concatenation in
hermest's CLI that needed [Json.Error.to_string e] now that
[Json.of_string] yields a structured error.
Warning 69 (unused-field, mutable-never-assigned). Four independent
record fields were flagged as mutable but the code only mutates their
referents in place, never rebinds the record slot itself:
- ocaml-wal/lib/wal.ml: [t.file] (the Eio file resource; methods call
Eio.File.pwrite_all etc., the slot is set once at open time).
- ocaml-block/lib/block.ml: [Memory.state.data] (the backing bytes,
written via Bytes.blit_string; [Bytes.t] is already mutable).
- ocaml-sse/lib/sse.ml: [Parser.t.data_buf] (a Buffer.t, written via
Buffer.add_*; the slot never changes).
- ocaml-zephyr/lib/zephyr.ml: drop [mode : Read | Write] entirely —
set at open-time, read nowhere. The open_read / open_write
constructors already distinguish the two call shapes, so mode
tracking was redundant.
read_master: extract a small classify_master_row helper that returns a
[Mtable | Mautoindex | Mextra | Mskip] variant. The big match/for-loop
shrinks to a flat tag dispatch.
create_table: stop substring-scanning the SQL for the table name. The
lexer/parser already produces an Ast.create_table value, so use
[ct.tbl_name] directly. Drops ~50 lines of fragile string fiddling.
Test cleanups: rename 9 tests with 5+ underscores to shorter forms.
Narrow the [with _ -> ...] handlers in test_sqlite.ml to
[Sqlite.Unique_violation _ | Failure _], and the inlined hostile suite
adds Invalid_argument / Sys_error / End_of_file via a [safe_exn]
predicate so unexpected exceptions still surface.
Inline test_hostile.ml content into test_sqlite.ml as flat helpers /
test functions plus a [hostile_cases] list appended to the main suite.
Drops the "t" suffix and follows the value/codec/toml/core pattern
(jsont.json_base style). The internal raw TOML module moves from
[Toml] to [Value] (file: lib/value.ml, was lib/toml.ml) to make room
for the top-level Toml facade (file: lib/toml.ml, was lib/tomlt.ml).
External callers now reach the raw AST through [Toml.Value.X] instead
of [Tomlt.Toml.X]. Every downstream reference updated in lockstep.
The CREATE TABLE lexer matched on ASCII characters only — punctuation,
keywords, identifiers, numbers, and quoted strings whose body bytes
pass through verbatim. None of sedlex's Unicode classes were used, so
sedlex.ppx was paying for nothing while merlint flagged every
PPX-expanded [__sedlex_state_*] / [__sedlex_partition_*] identifier as
double-underscore module access (~68 spurious E335 hits).
Convert lib/lexer.ml -> lib/lexer.mll. The grammar is unchanged: same
keyword table, same quoted-identifier and single-quoted string readers
(now ocamllex sub-rules), same NUMBER fallback for operators / passthrough
bytes (UTF-8 round-trips because ocamllex's [_] matches any byte).
Drop sedlex, sedlex.ppx, and menhirLib from the library deps; menhir's
traditional API plugs into [Parser.create_table token lexbuf] directly.
Generate .opam.template files with x-quality-* fields based on
detected package features:
- x-quality-build: has lib/ with .ml files
- x-quality-test: has test/ with .ml files
- x-quality-fuzz: has fuzz/ with .ml files
- x-quality-interop: has test/interop/ directory
- x-quality-cram: has test/*.t/ directories
These fields are picked up by dune's opam generation and will be
checked by merlint E910 for consistency.
Also: add fmt dep to ocaml-sse/lib/dune (Fmt.pf used without dep).
Standard vocabulary based on crates.io categories, erratique/opam
conventions, and monorepo domain coverage:
Org: org:blacksun
Domain: aerospace, codec, crypto, network, storage, git, merkle
Purpose: cli, test, bench, format, log, system
Protocol: ccsds, uslp, cop1, sdls, sle, atproto, tls, http, json, binary
Cross-cutting: eio, simulation, math, compression
Tags placed in dune-project (package ...) stanzas via (tags ...).
Propagated to .opam files by dune's opam generation.
- ocaml-cookeio: wrap test_clear in Eio_main.run to provide Eio context
- ocaml-pds: remove explicit Pds.close calls that double-closed the DB
when Eio.Switch.on_release already handles cleanup
- ocaml-sqlite: make sw field mutable; clear it in close so sync takes
the direct pager-write path instead of opening a new WAL on a dying
switch (fixes irmin's 17 failures and pds's "Switch finished!" errors)
- ocaml-spacedata: improve fetch_json timeout error message with URL
(2 live tests still fail due to CelesTrak being unreachable)
160 new tests exercising security-critical code paths identified by
mapping known CVEs from C/reference implementations to our OCaml code:
- ocaml-sqlite (9): cyclic pages, oversized varints, record overflow,
wrong page kind, truncated WAL, out-of-bounds root, garbage files
- ocaml-cbort (12): deep nesting (CVE-2025-24302), indefinite-length
DoS, integer overflow in lengths, truncated input, invalid types
- ocaml-tar (10): path traversal (CVE-2021-32803), symlink escape
(CVE-2025-45582), oversized octal, truncated headers, checksum
- ocaml-http (14): CRLF header injection (CWE-113), null bytes,
Content-Length overflow, empty/duplicate headers
Also hardens validate_header_name_str to reject null bytes/empty names
- ocaml-jsonwt (21): "none" algorithm bypass (CVE-2015-9235) case
variations, algorithm confusion (CVE-2016-10555), malformed headers,
empty segments, extra dots, large payloads
- ocaml-cose (8): algorithm substitution, missing algorithm header,
malformed CBOR, wrong types, label overlap (RFC 9052)
- ocaml-git (18): tree path traversal, null bytes, symlink mode,
malformed tree data, pack delta attacks, pack format validation
- ocaml-tomlt (25): duplicate keys, integer overflow, malformed dates
(invalid month/day/hour/minute), deep nesting, long strings
- ocaml-squashfs (20): symlink traversal edge cases, fragment table
bounds, inode self-reference, compression bomb limits, bad superblock
- ocaml-cpio (23): symlink target validation, null bytes in filenames,
oversized filesize, truncated archives, invalid magic numbers
- PRIMARY KEY on non-INTEGER columns now enforces UNIQUE (tests 91-93).
Added has_primary_key field to Ast.column_def; parse_unique_constraints
now treats both column-level and table-level PRIMARY KEY as UNIQUE.
- create_table rejects duplicate table names instead of silently creating
a second entry (test 97).
- Table.create ~name:"kv" reuses the existing default kv table instead of
creating a duplicate (test 98).
- Quoted table names ("my table", [my table], `my table`) are now properly
unquoted during CREATE TABLE parsing (test 99).
- Add memtrace instrumentation to test runner.
monotonic_now: increment changed from 1μs to 1ms, drift capped at 300s.
Rejects registration if clock drift exceeds cap instead of silently
issuing synthetic timestamps indefinitely.
verify/verify_receipt_only now return receipt_summary with total,
verified, failed, skipped counts so callers can detect partial
verification (e.g. 2 of 3 receipts tampered).
bottle_checksums split on the last "-" to extract the binary name,
but date-hash versions (20260401-d14d5fc) have a "-" in them. This
produced names like "cascade-20260401" instead of "cascade".
Now finds the first "-" followed by a digit (the version's YYYYMMDD
prefix), correctly splitting "cascade-20260401-d14d5fc..." into
name="cascade" and version="20260401-d14d5fc...".
Per alcobar's README, corpus generation uses fuzz.exe --gen-corpus,
not a separate gen_corpus.ml. Updated E718 and E724 rules to check
for --gen-corpus in the dune fuzz rule instead of gen_corpus.ml/.exe.
Removed gen_corpus from the valid fuzz filename list.
Also: add .merlint to ocaml-sqlite excluding E331 for create_table,
and doc comments on test suite mli files (E405).
Add Pager.snapshot/rollback for capturing and restoring dirty page
state. Expose Table.save_root/restore_root and Index.save_root/
restore_root for B-tree root page rollback after splits.
Build Sqlite.with_transaction on top: snapshots pager state, B-tree
roots, and in-memory KV caches before running the user function. On
exception, all state is rolled back and the exception re-raised.
Five new tests cover commit, KV rollback, unique index rollback,
and nested transaction failure.
Add missing opam dependencies across 17 packages (vec3, bytesrw, http,
cbort, fmt, etc.) and remove unused deps (requests from respond, menhir
from sqlite). Teach monopam lint to recognize (menhir ...) dune stanzas
as requiring the menhir package. Move bytesrw dep from scitt to
atp-lexicon-scitt where it actually belongs.
- parser.mly: conflict-free LR(1) grammar; handles structure (parens,
commas, table-vs-column dispatch)
- lexer.ml: sedlex tokenizer with case-preserving keyword tokens,
4 quoting styles, SQL comments
- ast.ml: column body classification (type affinity + constraints)
done in OCaml, not the grammar
Also fixes:
- CONSTRAINT name UNIQUE(...) now enforced (named table constraints)
- NULL columns skip UNIQUE checks per SQL standard (multiple NULLs
allowed under UNIQUE)
ocaml-sqlite: Parse and enforce UNIQUE constraints (column-level and
table-level) via in-memory hash indexes. Raises Unique_violation on
conflict. Indexes are rebuilt on database reopen.
ocaml-auth: Add UNIQUE(provider, provider_uid) to oauth_identities
schema. Callback now catches Unique_violation on concurrent creates
and falls back to the existing identity.
ocaml-oauth: Classify parse_token_response errors into Invalid_json,
Missing_access_token, and Invalid_token_format instead of collapsing
all failures to Invalid_json.
1. Sqlite: silent data loss on corrupt DB — Store.v caught all exceptions
and fell back to Sqlite.v (which truncates). Now only catches Eio.Io
(file not found). Remove Sqlite.v; add Sqlite.open_ ~create flag.
2. Sign-out: now revokes server-side session via Store.delete_session.
Previously only cleared the browser cookie — copied sid stayed valid.
Changed to POST /auth/signout with header-based session lookup.
3. OAuth: authorization URL now includes response_type=code per RFC 6749.
4. OAuth: token exchange/refresh now uses application/x-www-form-urlencoded
per RFC 6749, not JSON. Removed JSON-body functions, added form_encode
helper and exchange_form_body/refresh_form_body returning strings.
5. Auth: callback now uses provider.userinfo_url instead of hardcoded
GitHub API. Added userinfo_url to Oauth.provider type. Google, GitLab,
and custom providers can now complete login.
- Remove vendored crowbar/ directory
- Replace all Crowbar references with Alcobar across 176 .ml files
- Update all fuzz dune files: crowbar → alcobar in libraries
- Remove 77 gen_corpus.ml files (alcobar handles corpus internally)
- Update dune-project files: crowbar → alcobar in dependencies
- Update merlint rules (e705, e726): Crowbar → Alcobar in checks,
docs, and examples
- Update merlint generated docs (index.html)
428 files changed, ~1200 lines removed net.
B-tree tests (56 total):
- Overflow thresholds: X=U-35, M=((U-12)*32/255)-23 for all page sizes
- Boundary: exact max_local (4061), X+1 (4062), multi-page overflow
- All valid page sizes (512..65536) with and without overflow
- Page type flags, header sizes, varint 9-byte encoding
- Record serial types (NULL, int8/16/24/32/48/64, float, text, blob)
- Stress: 50 cells of increasing size with overflow + splits
SQLite tests (56 total):
- Database header byte-level: magic string, page size, payload fractions
(64/32/32), schema format, text encoding (UTF-8), reserved bytes
- change_counter == version_valid_for validity check
- Page 1 B-tree header at offset 100 (type=0x0d, fragmented<=60)
- Overflow value roundtrip and persistence across close/reopen
- ocaml-sqlite: merge generic_suite into suite, single export in .mli
- ocaml-tcpcl: add missing test_tcpcl.mli
- ocaml-tls: rename test_rng to mock_rng (helper, not test module)
- ocaml-tls: restructure test_tls_eio/unix stubs with test.ml runners
- ocaml-tomlt: restructure test_tomlt_{bytesrw,eio,unix} stubs with
test.ml runners and .mli files
open_ no longer requires a 'kv' table — it parses CREATE TABLE SQL from
sqlite_master to build schemas for every table. New generic read API
(tables, iter_table, fold_table, read_table) lets callers read any SQLite
database. INTEGER PRIMARY KEY columns get rowid substitution, and trailing
NULLs are padded per the SQLite storage optimization.
CLI gains `sql tables DB` and `sql read DB -t TABLE` subcommands.