Fail a Build if Code Coverage is Low

4/14/2011

UPDATE: I have included the Code Coverage exe in the download

I have seen lots of questions about how to manage code coverage. Some people use Custom Check In policies to stop people checking in code with low coverage. This is one way to do it. Traditionally these have been difficult to deploy and manage although it getting better. Another option is to fail the build if the code coverage drops below a certain threshold. This has the advantage of being a server side solution and therefore doesn’t rely on anything special installed on the client.

Approach

We can build on this approach that was used in Visual Studio 2005. The concept is the same. We will call out to an exe (using the trusty InvokeProcess activity from our build xaml file.) This executable will parse the binary data.coverage file that gets generated by MSTest. We can then sum up the Code Coverage to an overall percentage and compare it to a value in the build definition. If our coverage is too low we will fail the build.

Update the Code

The same I linked to above was based on VS2005. Things have changed a bit since then. Here is a good blog post on code coverage changes. I followed that post to ensure that the code will work with VS2010. I also changed the code to work out the total code coverage for the solution and output that to the console stream.

The Build

First lets create a new build definition.

image

We will then create a new Build Process Template

image

and open it. Add a new Argument to hold the code coverage limit.

image

Then click the Metadata argument add our percentage as a Process Parameter.

image

Click Add and enter Parameter Name and values.

image

This will be our code coverage limit that the build will accept. Anything less than this and we will fail the build. Now lets edit our Xaml file to use this value.

Add another variable to hold the total code coverage

image

We will need to add an InvokeProcess shape to our build process. I have added this inside the Run Tests sequence.

image

The total code coverage (as received from stdOutput) gets assigned to the CodeCovered variable.

The FileName is

image

obviously the CodeCoverageConverter.exe needs to be installed onto the Build Server in this path.

The arguments are set as follows:-

"/infile:" + TestResultsDirectory + "\CodeCoverageTests\In\" + System.Environment.MachineName + "\data.coverage /outfile:" + TestResultsDirectory + "\coverageresults.xml /exepath:" + TestResultsDirectory + "\CodeCoverageTests\Out\"

If you look at this you will see an interesting point. The default path for the TestResults In and out folder have been changed. These folders contains our .coverage file as well as the dlls used to execute the tests. The standard path includes the Date and Time that the test executed. In order for us to be able to find these files from the build we need to set the path.

Setting the Path

We can do this from our TestSettings file that the build uses. You can see from the Build process parameters here that my build is the local.testsettings file.

image

I change this as follows:-

image

This means that the TestResults will always be placed in that folder. However this introduces another issue. If we leave it like this I will end up with the following structure:-

TestResults

CodeCoverageTests

CodeCoverageTests[1]

CodeCoverageTests[2]

and so on

So as part of my build I need to delete this folder to ensure that there will only be one copy. To do this I have used a DeleteDirectory shape.

image

The parameters are:-

image

This will ensure that only one CodeCoverageTests folder exists.

We now need to check if the overall code coverage is below our threshold. To do this I have used.

image

The IF shape checks if our coverage is lower than our threshold. If it is then it will update the Build Test Status to failed and write a message to the Build Log as a Build Error.

Testing the Build

I have set the Code Coverage threshold at 80%. I have also selected the “Fail build on test failure” option

image

image

As usual I have included all the code as a sample. Running a build with 100% code coverage gives us.

image

No surprises there. If we now comment out some tests to bring out Code Coverage down to 50% we see.

image

Success we have failed our build!

Code can be found here.

4/14/2011

6 comments :

  1. Hi, I've attempted to follow your blog post to get my builds failing on a lower coverage threshold, however I am having problems with coding the coverage exe tool. Could you post your modified coverage exe somewhere so that we can try it? Many thanks in advance.

    ReplyDelete
  2. Hi AhBeng, yes will post tomorrow. Let me know how you get on.

    ReplyDelete
  3. Hi AhBeng

    I have updated the zip file to include the coverage exe tool.

    Let me know how you get on.

    ReplyDelete
  4. Hi Rob

    Nice work!
    I am more interested in the trend over time: How did Code Coverage change since last week? Since last month?

    Could you extend this to say: If Code Coverage drops by 2% since last check-in -> Fail the build!

    I have done something similar with "warning count"
    See here
    http://blog.gfader.com/2011/04/tfs-fail-build-when-warning-count.html

    ReplyDelete
  5. I tried using the exe in invokeprocess activity, however I don't get anything in stdOutput, so Convert.ToDouble fails in the Assign activity.

    ReplyDelete
  6. Is there a newer version of this tool and instructions for TFS 2012, preferably on Codeplex or Github?

    ReplyDelete