August 30th, 2019

2009

NUnit screw up

NUnit team tried to improve performance, but created very confusing and annoying bug in NUnit Adapter for Visual Studio.
https://github.com/nunit/nunit3-vs-adapter/issues/648#issuecomment-525762809
Siko91 commented 2 days ago:
We use NUnit 3.10 and NUnit3TestAdapter 3.15
With VS2017, when trying to run or debug a test, half of the attempts would end up with this error.
The other half of the attempts successfully start running/debugging the test.

It fails with this error on every second attempt.
Accurate like a clock.

first time running a test starts properly
second time - NUnit failed to load
third time starts properly
fourth time - failed to load
and so on...

I am guessing that the run that starts properly changes some persisting state (file?) and sets it in an invalid way, so that the next one can't start, but can reset the state... so that the next one can start and screw it up again.
Why NUnit team could not catch this obvious bug themselves?
My explanation is that they tested new NUnit Adapter code against most recent NUnit Framework code.
But most of developers update NUnit Framework version only occasionally.
We had NUnit Framework 3.9 (which is older than the newest NUnit Framework 3.12).
On another hand, Visual Studio automatically updates NUnit Adapter version.
Several days ago we automatically received new NUnit Adapter update (NUnit Adapter 3.15), which conflicts against our installed version of NUnit Framework.
https://github.com/nunit/nunit3-vs-adapter/issues/651
We have issues in some cases with pre-filtering
...
what is affected:
• Any version of NUnit below 3.11.0
• Any test assemblies using SetupFixture
• TestCaseData using SetName instead of SetArgDisplayNames.


Originally posted at: https://dennisgorelik.dreamwidth.org/175445.html