this repo has no description
more
+105
-3
+88
requests/CLAUDE.md
+88
requests/CLAUDE.md
···
1
+
## Requests Library Implementation
2
+
3
+
The `requests` library has been implemented as a standalone OCaml library
4
+
providing HTTP client functionality similar to Python's requests/urllib3. It
5
+
successfully abstracts TLS configuration and CA certificate complexity while
6
+
providing a clean Eio-style API.
7
+
8
+
### Key Implementation Details
9
+
10
+
1. **Module Organization**:
11
+
- Auth module must be defined before Config in both .mli and .ml files (Config references Auth.t)
12
+
- ConnectionPool uses a GADT to hide the network type parameter
13
+
- Session and PoolManager use internal type aliases to avoid naming conflicts
14
+
15
+
2. **Buffer Size Limitations**:
16
+
- Cannot use `Int.max_int` for Eio buffer sizes (causes Bigarray creation errors)
17
+
- Use reasonable sizes like `16 * 1024 * 1024` (16MB) for buffer creation
18
+
- When converting Cohttp_eio body to Buf_read, always specify max_size
19
+
20
+
3. **Type Constraints**:
21
+
- Network types use polymorphic variants: `[> \`Generic] Net.t`
22
+
- Cannot use wildcard `_` in type declarations
23
+
- Use GADTs or existential types when hiding type parameters in module signatures
24
+
25
+
4. **Build Commands**:
26
+
- Development build: `opam exec -- dune build @check`
27
+
- Release build (ignores warnings): `opam exec -- dune build @check --profile=release`
28
+
- Documentation: `opam exec -- dune build @doc`
29
+
30
+
### Implemented Features
31
+
32
+
#### Core (urllib3-inspired):
33
+
- ✅ Per-host connection pooling with configurable limits
34
+
- ✅ Automatic retries with exponential backoff (default 10 retries)
35
+
- ✅ Advanced timeout handling (connect, read, total)
36
+
- ✅ HTTP caching with Cache-Control support
37
+
- ✅ Progress tracking via OCaml Logs
38
+
- ✅ Cookie handling in sessions
39
+
40
+
#### Authentication (requests-inspired):
41
+
- ✅ Basic authentication
42
+
- ✅ Digest authentication (RFC 2617)
43
+
- ✅ Bearer token (OAuth 2.0)
44
+
- ✅ OAuth 1.0a support
45
+
- ✅ OAuth 2.0 support
46
+
- ✅ Custom authentication handlers
47
+
48
+
#### Streaming:
49
+
- ✅ Upload streaming with Eio Flow sources
50
+
- ✅ Download streaming with Eio Flow sinks
51
+
- ✅ Response iteration (chunked/line-based)
52
+
53
+
### TODO - Future Work
54
+
55
+
1. **High Priority** (Core functionality):
56
+
- [ ] Fix remaining build issues in lib/ruminant.ml (Github.Api.Https module)
57
+
- [ ] Fix bin/main.ml (data.issues field issue)
58
+
- [ ] Complete OAuth implementation (currently stubbed)
59
+
- [ ] Implement actual HTTP caching storage (memory/file backends partially done)
60
+
- [ ] Add proper connection pool lifecycle management
61
+
62
+
2. **Medium Priority** (Enhanced features from DESIGN.md):
63
+
- [ ] Implement hooks system for request/response middleware
64
+
- [ ] Add SOCKS proxy support (design in DESIGN.md)
65
+
- [ ] Implement character encoding detection and conversion
66
+
- [ ] Add compression support (gzip, deflate, brotli when available)
67
+
- [ ] Implement proper multipart form data encoding
68
+
69
+
3. **Low Priority** (Advanced features):
70
+
- [ ] WebSocket support
71
+
- [ ] HTTP/2 support (when cohttp-eio adds it)
72
+
- [ ] HTTP/3 support (future)
73
+
- [ ] Connection persistence and keep-alive optimization
74
+
- [ ] Advanced certificate pinning
75
+
76
+
4. **Testing & Documentation**:
77
+
- [ ] Add comprehensive test suite for requests library
78
+
- [ ] Create examples directory with common use cases
79
+
- [ ] Write API documentation and usage guide
80
+
- [ ] Add benchmarks comparing with Python requests
81
+
82
+
### Known Issues
83
+
84
+
2. **Type Shadowing**: Module-level type `t` can cause shadowing issues in nested modules. Use unique type names or GADTs to avoid conflicts.
85
+
86
+
3. **Module Dependencies**: Auth module must come before Config in the interface due to Config's create function taking an Auth.t parameter.
87
+
88
+
4. **Warnings**: Many unused variables in stub implementations need cleanup once features are fully implemented.
+1
-1
requests/lib/dune
+1
-1
requests/lib/dune
···
1
1
(library
2
2
(public_name requests)
3
3
(name requests)
4
-
(libraries eio cohttp-eio tls-eio ca-certs x509 uri yojson logs base64 unix digestif mirage-crypto-rng mirage-crypto-rng.unix domain-name))
4
+
(libraries eio cohttp-eio tls-eio ca-certs x509 uri yojson logs base64 unix digestif mirage-crypto-rng mirage-crypto-rng.unix domain-name xdg))
+11
-1
requests/lib/requests.ml
+11
-1
requests/lib/requests.ml
···
967
967
mutable misses : int;
968
968
}
969
969
970
-
let create ~cache_dir ~max_size () = {
970
+
let default_cache_dir () =
971
+
(* Use XDG cache directory for storing HTTP cache *)
972
+
let xdg = Xdg.create ~env:Sys.getenv_opt () in
973
+
let cache_home = Xdg.cache_dir xdg in
974
+
let cache_dir = Filename.concat cache_home "ocaml-requests" in
975
+
(* Ensure cache directory exists *)
976
+
if not (Sys.file_exists cache_dir) then
977
+
Unix.mkdir cache_dir 0o755;
978
+
cache_dir
979
+
980
+
let create ?(cache_dir = default_cache_dir ()) ~max_size () = {
971
981
cache_dir; max_size; size = 0L; hits = 0; misses = 0;
972
982
}
973
983
end
+5
-1
requests/lib/requests.mli
+5
-1
requests/lib/requests.mli
···
404
404
405
405
module File : sig
406
406
type storage
407
-
val create : cache_dir:string -> max_size:int64 -> unit -> storage
407
+
val default_cache_dir : unit -> string
408
+
(** Get the default cache directory using XDG standards *)
409
+
410
+
val create : ?cache_dir:string -> max_size:int64 -> unit -> storage
411
+
(** Create a file-based cache storage. Uses XDG cache directory by default *)
408
412
end
409
413
410
414
type storage = [