new-tree-polling test failure #53

Closed

jonmsterling.com 3w ago

Closed opened by

jonmsterling.com assigned to

kento.tngl.sh 3w ago

Not sure what's the deal with this, maybe some subtle different behaviour in neovim on different platforms?

--- a/_build/default/test/lsp/new-tree-polling.t
+++ b/_build/default/test/lsp/new-tree-polling.t.corrected
@@ -25,7 +25,6 @@ See new-tree-polling.lua

     \transclude{}

-  requesting completions in trees/completion.tree at position line 0, character 12
-  {
+  requesting completions in trees/completion.tree at position line 0, character 12{
     new_tree_completions = { "New (new-tree)" }
   }

kento.tngl.sh 3w ago

Yes, I updated neovim and it introduced a bunch of failures like this, so the current state corresponds to the behavior on my machine. Perhaps we should encode the version of neovim in some dune file, and only run the tests if the same version is available?

jonmsterling.com 3w ago

Maybe... I am worried however that the result will be that as soon as all developers upgrade their neovims, this test never gets run. Which is, I think, the definition of a test that is too flaky to exist.

kento.tngl.sh 3w ago

true...

jonmsterling.com 3w ago

It does seem like we do get mileage out of these tests as maintainers. I wonder if there is a way to just separate them out from the main dune test routine, so that we may run them deliberately. I think a practice of running the tests deliberately, and being responsible for keeping our neovim installation in sync, may be more resilient than forgetting they exist and having them stop being run because of an invisible condition.

kento.tngl.sh 3w ago

I'll try to set this up

kento.tngl.sh 3w ago

I removed the lsp tests from the default test alias, they can now be run with dune build @lsp-test. I also added a cram test that records the minor version of neovim that I used, but that can be removed if it is inconvenient.

jonmsterling.com 3w ago

Something unusual is happening: on the latest commit, these tests are still being run. Can you please double check what is going on ?

jonmsterling.com 3w ago

Did you try this yourself before committing?

kento.tngl.sh 3w ago

yes, but I think I had to delete the build folder? I just verified it by adding a failing test to lsp/, and it did not run under the default alias.

kento.tngl.sh 3w ago

It seems like dune does not invalidate the cache when the test alias changes, so it reused the old result.

jonmsterling.com 2w ago

I have deleted the build folder, though.

jonmsterling.com 2w ago

Just re-verified this on two separate machines; the output of rm -rf _build; dune test contains

-  requesting completions in trees/completion.tree at position line 0, character 12
-  {
+  requesting completions in trees/completion.tree at position line 0, character 12{
     new_tree_completions = { "New (new-tree)" }
   }

kento.tngl.sh 2w ago

what's your dune version? I'll try to reproduce it by using the same one as yours. Mine is 3.24.2, for the record

kento.tngl.sh 2w ago

I figured it out, just pushed a fix. The gist of it was that in the old dune file, all tests in the second stanza were not covered by runtest_alias false. My probe was not able to detect this, since it was covered by the first stanza.

jonmsterling.com 2w ago

Thanks for getting to the bottom of this! I felt like I was going nuts 😃