#2782

Gotchas: MSTest’s [DeploymentItem] attribute

So we’ve been struggling with our usage of the DeploymentItem attribute within our unit tests on the Service Factory for pretty much the entire project. We had some odd requirements like copying dynamically loaded assemblies into our test deployment root in order to make them available during testing.

Essentially it seemed far too easy to create a test that worked fine on the local developer’s machine and failed either on the CI server or in someone else’s development environment. In fact 25% of our unit test related build breaks were caused by this issue and some of them took a while to track down!

A number of things have come to light. In order of general nastiness:

1) If the file referred to by DeploymentItem does not exist or cannot be copied the test doesn’t fail automatically. Your test code may fail because it cannot find the file but DeploymentItem just writes a warning into the MSTest log which you’ll probably never see.

Workaround: Make DeploymentItem output visible in the build log. You can add the following to your build to parse the .TRX files and make DeploymentItem log entries into warnings or errors.

<CreateItem Include=“$(MSTestOutputDir)\*.trx“>
  <Output ItemName=“TestResultFiles“ TaskParameter=“Include“/>
</CreateItem>
<Xml.GetValue
Path=“%(TestResultFiles.Identity)”
  XPath=“//Tests/TestRun/result/runInfoList/_items/element/text“ >
  <Output ItemName=“TestDeployErrors” TaskParameter=“Results“ />
</Xml.GetValue>
<ErrorText=“MSTest: %(TestDeployErrors.Identity)“
  Condition=“ ‘@(TestDeployErrors)’ != ” “ />
…
<UsingTask TaskName=“Microsoft.Sdc.Tasks.Xml.GetValue”
  AssemblyFile=“$(ToolsBinDir)Microsoft.SDC.Tasks.dll“ />


The SDC tasks, for those who haven’t used them are available on CodePlex.

2) If a file already exists then DeploymentItem will not copy it again. So if Test1 has a deploymentItem(”A.txt”) attribute and Test2 has the same DeploymentItem attribute A.txt will only get copied once. This is fine unless Test1 runs before Test2 and changes A.txt. In which case Test2 may fail because A.txt will contain unexpected data.

Workaround: Use separate directories for each test. The way to get around this is always use OutputDirectory property on the attribute to give each test its own private working area. Like so:

[TestMethod]
[DeploymentItem(@"..\..\Data_2\Test.txt")]
[DeploymentItem(@"..\..\Data_1\Test.txt")]
public void WhichItemGetsWrittenTest()
{
  string filePath = Path.Combine(
    Directory.GetCurrentDirectory(), @”Test.txt”);
  string actual = File.ReadAllText(filePath);

  Assert.AreEqual(“1″, actual);
}

So this test fails. The “2″ version of Test.txt gets copied and the “1″ version does not. So the test reads “2″ from the file and fails. DeploymentItem does not error even though two attributes on the same test are trying to deploy the two files with the same name.

5) DeploymentItem is a method, not class attribute. If you want the same file present for all tests in a fixture then in principle you need to apply the attribute to every test method. MSTest seems to run test methods in the order they appear in the class you might be tempted to try and get away with adding the item to the first method. Relying on this is bad your tests are now order dependent! Do not do this.

How did I figure this out? Well I actually ended up writing a series of unit tests that used DeploymentItem. You can download them yourself from here.

I’m told much of this issue goes away in the upcoming Visual Studio release as MSTest doesn’t force you don’t have to “deploy” all your tests to a separate folder. If you do end up using DeploymentItem then do so with caution! My personal preference will be to move to storing data as resources and/or virtualizing the filesystem using a mock.