Gotchas: MSTest’s [DeploymentItem] attribute
Tuesday, October 2, 2007 – 3:15 PMSo we’ve been struggling with our usage of the DeploymentItem attribute within our unit tests on the Service Factory for pretty much the entire project. We had some odd requirements like copying dynamically loaded assemblies into our test deployment root in order to make them available during testing.
Essentially it seemed far too easy to create a test that worked fine on the local developer’s machine and failed either on the CI server or in someone else’s development environment. In fact 25% of our unit test related build breaks were caused by this issue and some of them took a while to track down!
A number of things have come to light. In order of general nastiness:
1) If the file referred to by DeploymentItem does not exist or cannot be copied the test doesn’t fail automatically. Your test code may fail because it cannot find the file but DeploymentItem just writes a warning into the MSTest log which you’ll probably never see.
Workaround: Make DeploymentItem output visible in the build log. You can add the following to your build to parse the .TRX files and make DeploymentItem log entries into warnings or errors.
<CreateItem Include="$(MSTestOutputDir)\*.trx">
<Output ItemName="TestResultFiles" TaskParameter="Include"/>
</CreateItem>
<Xml.GetValue
Path="%(TestResultFiles.Identity)"
XPath="//Tests/TestRun/result/runInfoList/_items/element/text" >
<Output ItemName="TestDeployErrors" TaskParameter="Results" />
</Xml.GetValue>
<ErrorText="MSTest: %(TestDeployErrors.Identity)"
Condition=" '@(TestDeployErrors)' != '' " />
...
<UsingTask TaskName="Microsoft.Sdc.Tasks.Xml.GetValue"
AssemblyFile="$(ToolsBinDir)Microsoft.SDC.Tasks.dll" />
The SDC tasks, for those who haven’t used them are available on CodePlex.
2) If a file already exists then DeploymentItem will not copy it again. So if Test1 has a deploymentItem(“A.txt”) attribute and Test2 has the same DeploymentItem attribute A.txt will only get copied once. This is fine unless Test1 runs before Test2 and changes A.txt. In which case Test2 may fail because A.txt will contain unexpected data.
Workaround: Use separate directories for each test. The way to get around this is always use OutputDirectory property on the attribute to give each test its own private working area. Like so:
[TestMethod]
[DeploymentItem(@”A.txt”, “Test1Files”)]
public void Test1()
{
//…
}
[TestMethod]
[DeploymentItem(@”A.txt”, “Test2Files”)]
public void Test2()
{
//…
}
3) If a file gets deleted by a test DeploymentItem will not re-copy it. Subsequent tests will fail because the file they expect to be there will not exist.
Workaround: You can use the same fix for issue #2 and give each test its own private working area.
4) The ordering of DeploymentItem attributes matters. The first DeploymentItem copies a file, others are ignored as per #2.
[TestMethod]
[DeploymentItem(@"..\..\Data_2\Test.txt")]
[DeploymentItem(@"..\..\Data_1\Test.txt")]
public void WhichItemGetsWrittenTest()
{
string filePath = Path.Combine(
Directory.GetCurrentDirectory(), @"Test.txt");
string actual = File.ReadAllText(filePath);
Assert.AreEqual("1", actual);
}
So this test fails. The “2” version of Test.txt gets copied and the “1” version does not. So the test reads “2” from the file and fails. DeploymentItem does not error even though two attributes on the same test are trying to deploy the two files with the same name.
5) DeploymentItem is a method, not class attribute. If you want the same file present for all tests in a fixture then in principle you need to apply the attribute to every test method. MSTest seems to run test methods in the order they appear in the class you might be tempted to try and get away with adding the item to the first method. Relying on this is bad your tests are now order dependent! Do not do this.
How did I figure this out? Well I actually ended up writing a series of unit tests that used DeploymentItem. You can download them yourself from here.
I’m told much of this issue goes away in the upcoming Visual Studio release as MSTest doesn’t force you don’t have to “deploy” all your tests to a separate folder. If you do end up using DeploymentItem then do so with caution! My personal preference will be to move to storing data as resources and/or virtualizing the filesystem using a mock.
15 Responses to “Gotchas: MSTest’s [DeploymentItem] attribute”
Google must have mis-interpreted the “Gotcha” and “Test” words, because I have advertisements for local STD testing centers.
By Brian on Oct 3, 2007
One other thing fixed in orcas — we have Class level deployment attributes, so you can put your dependancies there.
However, the list of files is still collapsed into a flat list prior to test execution.
One other — we dont execute tests in the order on the test class. While that maybe the behaviour your seeing, thats just luck… dont rely on it because it can change!
By Dominic Hopton on Oct 4, 2007
Where is the MSTest log file located? I can’t seem to find it within my solution.
I’m having a problem when I #include “afxwin.h”. The solution builds and then when it gets to running the tests, the first test will hang and no other tests will run. If I comment out the #include, then the tests will run fine.
My compiler options are /clr, /mTd and I’m compiling to an MFC shared dll.
By Jared on May 4, 2009
Jared,
MSTest creates a TRX file. This is unfortunately in a proprietary format although you can convert it with this tool.
The hang when including afxwin.h might be due to the not being able to load another dependent library. I’ve seen similar behavior with WinSxS and OpenMP libraries (see this post). I see you posted on stack overflow, which is probably a better place to get an answer for this. SxS load errors appear in the event log so that’s a good place to look to see what’s wrong also.
Ade
By Ade Miller on May 4, 2009
Hi Ade,
I’ve gotten past the issue of #including “afxwin.h” and I can get the unit tests to run fine through the IDE. When I run it using mstest.exe though, the tests hang while loading the unit test DLL. I believe this is because it is missing a DeploymentItem, although there are no external files that the tests rely upon.
This only starts to happen when I call AfxGetApp() and also AFX_MANAGE_STATE, of which I’m not sure which one is causing the issue.
My guess is that it is looking for an MFC DLL. Is there any way that you can have mstest or another tool tell you what items it is looking for?
I have tried Process Monitor but there are so many calls it gets overwhelming trying to find the missing DLL.
Thanks in advance,
Jared
By Jared on Jun 1, 2009
Jared,
Process monitor is very noisy but the filter feature will let you cut down the output. I’d try setting a filter on the Process Name to exclude everything but MSTEST.EXE.
Ade
By Ade Miller on Jun 4, 2009
Actually, it appears that test order is quite strange. Example, I’ve posted the following at stackoverflow:
was looking at a legacy MSTest project. The tests were always running in the same order. The order was not alphabetic and was bouncing between methods in two *.cs TestMethod files.
I did not change the physical order of the legacy code. I did for my convenience append “MSTest01” to the method name of the first test, “MSTest02” to the method name of the second test, et cetera.
To my surprise, the execution order of the TestMethod functions changed; #3 first, #6 second, #5 third, et cetera.
When I removed the “MSTestnn” strings from the TestMethod function names, their execution order changed back to the previous ordering, i.e., one test from the first .cs file, two tests from the second .cs file, five tests from the first .cs file, et cetera.
It seems that file location may not be a factor while TestMethod function name may be a factor.
http://stackoverflow.com/questions/2255284/how-does-mstest-determine-the-order-in-which-to-run-test-methods
[P.S.: I appreciate you sharing your MSTest tips. Have you looked at xUnit.net?]
By gerry lowry on Feb 12, 2010
Gerry,
I think the order is based on how reflection returns the methods and classes. I never investigated this as I never rely on this – it’s an anti-pattern to expect tests to run in the same order or rely on ordering.
Ade
By Ade Miller on Feb 12, 2010
Another one for the list: Don’t place special characters in your DeploymentItem. If you do, Visual Studio will tell you t hat you did not put the [TestClass] attribute on your class.
Example:
[TestClass]
[DeploymentItem(“MyFolder\MyTestFile.dat”)]
public class MyTest
.
.
.
Did you see it? The developer forgot the @ before the string! So that \M produces a special ASCII character!
If you do this, you will get the following error on every method in the test class:
UTA004: Illegal use of attribute on methodname. The TestMethodAttribute can be defined only inside a class marked with the TestClass attribute.
By William Garrison on Jul 26, 2010
William,
Thanks… Definitely another gotcha.
Ade
By Ade Miller on Jul 26, 2010
This was useful, especially the workaround in 2. Thanks!
By Johnny on Oct 26, 2011