UPDATE: I have included the Code Coverage exe in the download
I have seen lots of questions about how to manage code coverage. Some people use Custom Check In policies to stop people checking in code with low coverage. This is one way to do it. Traditionally these have been difficult to deploy and manage although it getting better. Another option is to fail the build if the code coverage drops below a certain threshold. This has the advantage of being a server side solution and therefore doesn’t rely on anything special installed on the client.
We can build on this approach that was used in Visual Studio 2005. The concept is the same. We will call out to an exe (using the trusty InvokeProcess activity from our build xaml file.) This executable will parse the binary data.coverage file that gets generated by MSTest. We can then sum up the Code Coverage to an overall percentage and compare it to a value in the build definition. If our coverage is too low we will fail the build.
Update the Code
The same I linked to above was based on VS2005. Things have changed a bit since then. Here is a good blog post on code coverage changes. I followed that post to ensure that the code will work with VS2010. I also changed the code to work out the total code coverage for the solution and output that to the console stream.
First lets create a new build definition.
We will then create a new Build Process Template
and open it. Add a new Argument to hold the code coverage limit.
Then click the Metadata argument add our percentage as a Process Parameter.
Click Add and enter Parameter Name and values.
This will be our code coverage limit that the build will accept. Anything less than this and we will fail the build. Now lets edit our Xaml file to use this value.
Add another variable to hold the total code coverage
We will need to add an InvokeProcess shape to our build process. I have added this inside the Run Tests sequence.
The total code coverage (as received from stdOutput) gets assigned to the CodeCovered variable.
The FileName is
obviously the CodeCoverageConverter.exe needs to be installed onto the Build Server in this path.
The arguments are set as follows:-
"/infile:" + TestResultsDirectory + "\CodeCoverageTests\In\" + System.Environment.MachineName + "\data.coverage /outfile:" + TestResultsDirectory + "\coverageresults.xml /exepath:" + TestResultsDirectory + "\CodeCoverageTests\Out\"
If you look at this you will see an interesting point. The default path for the TestResults In and out folder have been changed. These folders contains our .coverage file as well as the dlls used to execute the tests. The standard path includes the Date and Time that the test executed. In order for us to be able to find these files from the build we need to set the path.
Setting the Path
We can do this from our TestSettings file that the build uses. You can see from the Build process parameters here that my build is the local.testsettings file.
I change this as follows:-
This means that the TestResults will always be placed in that folder. However this introduces another issue. If we leave it like this I will end up with the following structure:-
and so on
So as part of my build I need to delete this folder to ensure that there will only be one copy. To do this I have used a DeleteDirectory shape.
The parameters are:-
This will ensure that only one CodeCoverageTests folder exists.
We now need to check if the overall code coverage is below our threshold. To do this I have used.
The IF shape checks if our coverage is lower than our threshold. If it is then it will update the Build Test Status to failed and write a message to the Build Log as a Build Error.
Testing the Build
I have set the Code Coverage threshold at 80%. I have also selected the “Fail build on test failure” option
As usual I have included all the code as a sample. Running a build with 100% code coverage gives us.
No surprises there. If we now comment out some tests to bring out Code Coverage down to 50% we see.
Success we have failed our build!
Code can be found here.