xUnit.net runs tests in random order
Tuesday, November 6, 2007 – 3:04 PMWhen I was testing the behavior of MSTest’s DeploymentItem attribute I relied on fact that MSTest executes tests in a predictable order to figure out how the attribute works. Typically it seems to execute test methods in the order they appear in the class. You should never ever rely on this and a good unit testing framework shouldn’t let you! Apparently NUnit has the similar ordering issues, it runs tests alphabetically by name..
It’s evil because allows the unwary to write tests with implicit dependencies between them. The first test sets up some state which the second test relies on to in order to pass. Next thing you know changes to the first test cause some of the other tests to fail because they are effected by it. Much head scratching will ensue. In other words, it’s a “gotcha”.
Jim Newkirk was talking about xUnit.net today at the p&p Summit and happened to mention that xUnit executes tests in a random order! Great to see this explicitly encoded in a testing framework.
There should be a prize for the smallest feature ever added to a product that showed that the developers clearly understood their users. If there was this xUnit.net feature might just win it!
3 Responses to “xUnit.net runs tests in random order”
I’d love it if testing frameworks had a “shake-things-up” option, which I’d always turn on. This would increase the chances of shaking out some hidden dependencies between tests.
By Jason Bock on Nov 13, 2007
xUnit.net is random
see http://xunit.codeplex.com/Thread/View.aspx?ThreadId=75896
Brad Wilson: “In every public release we’ve ever made, classes are run in reflection order, and tests within those classes are run in randomized order”.
By gerry lowry on Feb 12, 2010