#2782

xUnit.net runs tests in random order

When I was testing the behavior of MSTest’s DeploymentItem attribute I relied on fact that MSTest executes tests in a predictable order to figure out how the attribute works. Typically it seems to execute test methods in the order they appear in the class. You should never ever rely on this and a good unit testing framework shouldn’t let you! Apparently NUnit has the similar ordering issues, it runs tests alphabetically by name..

It’s evil because allows the unwary to write tests with implicit dependencies between them. The first test sets up some state which the second test relies on to in order to pass. Next thing you know changes to the first test cause some of the other tests to fail because they are effected by it. Much head scratching will ensue. In other words, it’s a “gotcha”.

Jim Newkirk was talking about xUnit.net today at the p&p Summit and happened to mention that xUnit executes tests in a random order! Great to see this explicitly encoded in a testing framework.

There should be a prize for the smallest feature ever added to a product that showed that the developers clearly understood their users. If there was this xUnit.net feature might just win it!