Visual Studio 2012 & Code Coverage

6/27/2012

Like Unit Testing, Code Coverage has undergone significant change in VS 2012.  Mostly the changes have simplified the process from VS 2010.  Want to see code coverage now?  How about right click the test and select Analyze Code Coverage.  Sure beats what you had to do in VS 2010.

NB This information is compiled from two Microsoft blogs plus my own tinkering.  The blogs have more detail than explained here.  Check them out at

http://blogs.msdn.com/b/visualstudioalm/archive/2012/06/19/visual-studio-2012-rc-what-s-new-in-code-coverage.aspx  and

http://blogs.msdn.com/b/sudhakan/archive/2012/05/11/customizing-code-coverage-in-visual-studio-11.aspx

Licensing

Let’s get this one out of the way quick.  Code Coverage is available in Ultimate and Premium editions.  Moving on…

Scope

So how does the new coverage tool know what to include or exclude?  There used to be the TestSettings file that was used to specify what to include / exclude.

The new tool includes binaries “as long as these binaries are a part of your solution, and get called by any of your test cases under execution

Ok makes sense.  But what about binaries that are not part of my solution?  What if I have a file reference to a different binary.  Well that won’t show up.  If you want to see that included in your code coverage stats then you need to do some work.

VS2012 has a new file for this called a .runsettingsfile.  This is an xml file that contains which binaries to include or exclude.  In RC you will need to add a new xml file to your solution and name it something.runSettings. (extension is important here)  The TestSettings file is not used anymore as that was bound to MSTest and would not support the new test runner plug in architecture.

You can then copy some boilerplate text into the xml file (I am sure that the tooling will be sorted by RTM), adjust what you want to include / exclude.  Finally tell you solution to use the runsetttings file and hey presto, all should be good.

Putting it all together

So lets have a go.  Suppose I have the following structure.

UnitTestProject A that has a project reference to SUT.dll.  These two are in the same solution so all good so far.  However SUT.dll has a file reference to Account.dll.  Runing code coverage from UnitTestProject A gives me.

image

As you can see we only get the code coverage for UnitTestProject A and the SUT.dll.  This is because they are in the same solution.

So we now right click our solution, add new item, xml file.  Call it CodeCoverage.runsettings (first part of the name doesn’t matter)

Now add the following xml and save.

<?xml version="1.0" encoding="utf-8"?>

<RunSettings>

<DataCollectionRunSettings>

<DataCollectors>

<DataCollector friendlyName="Code Coverage"uri="datacollector://Microsoft/CodeCoverage/2.0"assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">

<Configuration>

<CodeCoverage>

<ModulePaths>

<!--

About include/exclude lists:

Empty "Include" clauses imply all; empty "Exclude" clauses imply none.

Each element in the list is a regular expression (ECMAScript syntax).

An item must first match at least one entry in the include list to be included.

Included items must then not match any entries in the exclude list to remain included.

It is considered an error to exclude all items from instrumentation as no data would be collected.

-->

<Include>

<ModulePath>.*\\UnitTestProject1\.dll</ModulePath>

</Include>

<Exclude>

<ModulePath>.*CPPUnitTestFramework.*</ModulePath>

</Exclude>

</ModulePaths>

<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>

<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>

<CollectFromChildProcesses>True</CollectFromChildProcesses>

<CollectAspDotNet>False</CollectAspDotNet>

<!--

Additional paths to search for symbol files. Symbols must be found for modules to be instrumented.

If symbols are alongside the binaries, they are automatically picked up. Otherwise specify the here.

Note that searching for symbols increases code coverage runtime. So keep this small and local.

<SymbolSearchPaths>

  <Path>C:\Users\User\Documents\Visual Studio 11\Projects\ProjectX\bin\Debug</Path>

  <Path>\\mybuildshare\builds\ProjectX</Path>

</SymbolSearchPaths>

-->

<Functions>

<Exclude>

<Function>^std::.*</Function>

<Function>^ATL::.*</Function>

<Function>.*::__GetTestMethodInfo.*</Function>

<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>

<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>

<Function>.*::YOU_CAN_ONLY_DESIGNATE_ONE_.*</Function>

</Exclude>

</Functions>

<Attributes>

<Exclude>

<Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>

<Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>

<Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>

<Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>

<Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>

</Exclude>

</Attributes>

<Sources>

<Exclude>

<Source>.*\\atlmfc\\.*</Source>

<Source>.*\\vctools\\.*</Source>

<Source>.*\\public\\sdk\\.*</Source>

<Source>.*\\microsoft sdks\\.*</Source>

<Source>.*\\vc\\include\\.*</Source>

</Exclude>

</Sources>

<CompanyNames>

<Exclude>

<CompanyName>.*microsoft.*</CompanyName>

</Exclude>

</CompanyNames>

<PublicKeyTokens>

<Exclude>

<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>

<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>

<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>

<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>

<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>

<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>

<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>

</Exclude>

</PublicKeyTokens>

</CodeCoverage>

</Configuration>

</DataCollector>

</DataCollectors>

</DataCollectionRunSettings>

</RunSettings>

This is now a default runSettings file.  Next we need to make sure that our extra binary gets included.  To do this we can make sure that our Include section is empty (include everything.)  Change the include section to look like

image

Finally go to the Test menu, select TestSettings, select TestSettings file.  Select your runsettings file and re-run the code coverage.  We now see

codecoverage

All of our binaries are included!

6/27/2012

2 comments :

  1. How I can fix path and name for .coverage file output? It generated file in a temp name folder with name user_systemname_datetiime.coverage.

    "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "C:\Users\xyz\Test\Test.dll" /Platform:x64 /InIsolation /EnableCodeCoverage /Settings:"C:\Users\xyz\CodeCoverage\CodeCoverage.runsettings"

    ReplyDelete
  2. How can I exclude global functions declared in C files? Is this possible?

    ReplyDelete