upstream: https://github.com/stedolan/crowbar
1# Crowbar
2
3**Crowbar** is a library for testing code, combining QuickCheck-style
4 property-based testing and the magical bug-finding powers of
5 [afl-fuzz](http://lcamtuf.coredump.cx/afl/).
6
7## TL;DR
8
9There are [some examples](./examples).
10
11Some brief hints:
12
131. Use an opam switch with AFL instrumentation enabled (e.g. `opam sw 4.04.0+afl`).
142. Run in AFL mode with `afl-fuzz -i in -o out -- ./_build/myprog.exe @@`.
153. If you run your executable without arguments, crowbar will perform some simple (non-AFL) testing instead.
164. Test binaries have a small amount of documentation, available with `--help`.
17
18## writing tests
19
20To test your software, come up with a property you'd like to test, then decide on the input you'd like for Crowbar to vary. A Crowbar test is some invocation of `Crowbar.check_eq` or `Crowbar.check`:
21
22```ocaml
23let identity x =
24 Crowbar.check_eq x x
25```
26
27and instructions for running the test with generated items with `Crowbar.add_test`:
28
29```ocaml
30let () =
31 Crowbar.(add_test ~name:"identity function" [int] (fun i -> identity i))
32```
33
34There are [more examples available](./examples), with varying levels complexity.
35
36## building tests
37
38Include `crowbar` in your list of dependencies via your favorite build system. The resulting executable is a Crowbar test. (Be sure to build a native-code executable, not bytecode.)
39
40To build tests that run under AFL, you'll need to build your tests with a compiler that has AFL instrumentation enabled. (You can also enable it specifically for your build, although this is not recommended if your code has any dependencies, including the OCaml standard library). OCaml compiler variants with AFL enabled by default are available in `opam` with the `+afl` tag. All versions published starting with 4.05.0 are available, along with a backported 4.04.0.
41
42```shell
43$ opam switch 4.06.0+afl
44$ eval `opam config env`
45$ ./build_my_rad_test.sh # or your relevant build runes
46```
47
48## running Tests
49
50Crowbar tests have two modes:
51
52* a simple quickcheck-like mode for testing propositions against totally random input
53* a mode using [afl-persistent](https://github.com/stedolan/ocaml-afl-persistent) to get good performance from `afl-fuzz` with OCaml's instrumentation enabled
54
55Crowbar tests can be directly invoked with `--help` for more documentation at runtime.
56
57### fully random test mode
58
59If you wish to use the quickcheck-like, fully random mode to run all tests distributed here, build the tests as above and then run the binary with no arguments.
60
61```
62$ ./my_rad_test.exe | head -5
63the first test: PASS
64
65the second test: PASS
66```
67
68### AFL mode requirements
69
70To run the tests in AFL mode, you'll need to install American Fuzzy Lop ([latest source tarball](http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz), although your distribution may also have a package available).
71
72Once `afl-fuzz` is available on your system, create an `input` directory with a non-empty file in it (or use `test/input`, conveniently provided in this repository), and an `output` directory for `afl-fuzz` to store its findings. Then, invoke your test binary:
73
74```
75afl-fuzz -i test/input -o output ./my_rad_test.exe @@
76```
77
78This will launch AFL, which will generate new test cases and track the exploration of the state space. When inputs are discovered which cause a property not to hold, they will be reported as crashes (along with actual crashes, although in the OCaml standard library these are rare). See the [afl-fuzz documentation](https://lcamtuf.coredump.cx/afl/status_screen.txt) for more on AFL's excellent interface.
79
80# What bugs have you found?
81
82[An open issue](https://github.com/stedolan/crowbar/issues/2) has a list of issues discovered by testing with Crowbar. If you use Crowbar to improve your software, please let us know!