Results of cram tests are cached but depend on environment #36

Closed

jonmsterling.com 1mo ago

Closed opened by

jonmsterling.com 1mo ago

I noticed something unfortunate when running the tests. Here's what I did:

I accidentally had another instance of Forester already running.
I ran the tests. They failed because they could not bind port 8080 (Fatal error: exception Unix.Unix_error(Unix.EADDRINUSE, "bind", "")).
I killed my other process, and ran the tests again. The results were cached rather than running again.
I tried running dune runtest --force and it had no impact. It still uses the cached results.

Basically, caching is not suitable for cram tests because they depend on the environment. We should find a way to ensure that the cram tests always run every time.

jonmsterling.com 1mo ago

It looks like dune clean will get rid of the cache. God, I hate the OCaml ecosystem so much. It's so crazy that dune runtest --force does not do what it says it does.

jonmsterling.com 1mo ago

I'm closing this because we can use dune clean, but let it be known that this was my last straw and I'm never creating a new project in OCaml again 😂

kento.tngl.sh 1mo ago

Yeah... I run the tests in the pre-commit hook. It copies the entire index into a temp directory and kills all running instances of forester, which works around this issue.