TFS 2010 Build - Only run impacted tests

3/10/2011
***UPDATED NOW FOR 2012 & 2013 SEE HERE - http://blog.robmaherconsulting.co.nz/2013/09/team-foundation-server-build-only-run.html
***

Spoiler – Link to final xaml is here
If your tests take a long time to run, you may wish to only run the tests that have been impacted by code changes checked in with the build (and of course run a full nightly build that executes all tests :) Unfortunately there is no out of the box feature to do this, so we need to edit our build xaml to do it. Lets get started:-
  • Create a new Visual Studio 2010 class library project. This will be the container for our build xaml file. Add the following references :-
  • Microsoft.TeamFoundation.Build.Client.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0.
  • Microsoft.TeamFoundation.Build.Workflow.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies.
  • System.Drawing.dll which can be found at <Program Files (x86)>\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0
  • Microsoft.TeamFoundation.VersionControl.Client.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0
  • Microsoft.TeamFoundation.WorkItemTracking.Client.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0
  • Microsoft.TeamFoundation.TestImpact.Client.dll which can be found in the GAC at <Windows>\assembly\GAC_MSIL\Microsoft.TeamFoundation.TestImpact.Client\10.0.0.0__b03f5f7f11d50a3a
  • Microsoft.TeamFoundation.TestImpact.BuildIntegration.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies
  • Create a new build definition with a new build process template (xaml file) see here for more details.
  • Add the xaml file created in the previous step to your Visual Studio project and open it.
  • Locate and Expand the “Compile, Test, and Associate Changesets and Work Items Parallel” shape. Your screen should now look like this.
If you scroll down the workflow and find the “Get Impacted Tests, Index Sources and Publish Symbols” parallel shape you will see that the check for impacted tests happens after the run tests activity. This means that if we use the xaml out of the box all of the tests will run every time.
We need to reverse this order. If we can scope the Run Tests shape with the output of the Get Impacted Tests shape then we should be all good.
This just leaves one problem. At various points (maybe every night, maybe after each release) we need to gather the data for impact analyis by running all of the tests (to include any code changes since we last ran it) This should be configurable.
Right lets get started. Find the "If PerformTestImpactAnalysis" parallel shape as below:-
Drag this shape upwards and insert here:-
Things should now look like this:-
By moving this shape we have ensured that the Impact Analysis will occur before any tests are run. We now need to somehow scope the tests to only those that have been impacted by changes in the build. Fortunately we are in luck. The "GetImpactedTests" shape (msdn ref) has an output parameter called (funnily enough) ImpactedTests. This is defined as List<Test> and returns a list of the tests that were affected by the code changes between the previous build and this build.
The "MSTest" shape (msdn ref) has an optional input parameter called TestNames. This is defined as IEnumerable<String> . Using TestNames you can specify the names of the tests that you want to run. This property is equivalent to the /test option of the MSTest.exe.
All we need to do is enumerate through the list of tests returned by the GetImpactedTests shape and pass them as a list of strings to the TestNames parameter of the MSTest activity. The steps to do this are:-
  • Create a workflow variable called ImpactedTestList of type TestList.
  • Create a variable called TestNames of type TestList
  • Select the "Get Impacted Tests" activity and set its ImpactedTests property to be the ImpactedTestList variable that we just defined.
impactedtestvar
We need to convert the TestList into a list of strings for MSTest. To do this we will use a ForEach shape to enumerate the List and an AddToCollection shape to build the list of strings. We can then pass this list to MSTest.
  • Now grab a ForEach shape from the toolbox and place it directly after the "GetImpactedTests" shape. Set the DisplayName to "Create Test List".
  • Enter ImpactedTest after Foreach and enter ImpactedTestList after in.
  • Drag an AddtoCollection shape and add it to the body of the ForEach shape.
  • Set the DisplayName of the AddtoCollection to Add Impacted Test to List.
  • Set the Collection to be TestNames, the TypeArgument to String and the Item to ImpactedTest.TestName.toString()
  • Your ForEach shape should now look like this:-

We need to pass this list of tests into the Run Test shape. To do this find the “Try Run Tests” Try Catch shape below our current position. Impact analysis only works when MSTest is run with a test settings file so we only need to update the Run Test activity in the middle. i.e. the middle red circle below.
updatemstest
Set the TestNames property of the “Run MSTest” shapes to our TestNames variable.
testnames
This just leaves us with two problems. Firstly how do we get all of the tests to run for the first build? Otherwise we will never have any impacted tests. The second problem is how do we ‘reset’ the impact data to include new code. We may want to do this nightly or every release/sprint etc. Fortunately we can solve both of these problems with the same solution. We need a way to override the scoping. If the TestNames list is empty then MSTest will run all of our tests. This solves our problem but introduces an edge case. What happens when there are no tests impacted by our code changes. We don't want any tests to run in that case.
We can solve this by using a custom process parameter. Follow the guidance here to add this to the Process tab of the definition editor. I called my parameter Is Base Line Run, and placed it in the advanced section. The parameter is a Boolean.
isbaselinerun
We can now add an If shape like this:-
baselinetest
The Condition property on the If shape is:-
TestNames.Count = 0 And BaseLineRun = False
In the “Then” portion of the shape we add an Assign shape and set DisableTests = True.
This will mean that if there are no tests in the TestNames List and it isn’t a base line run then no tests will run (because nothing was impacted) However if the base line run parameter is set to True then all of the tests will run.

Putting it all together

  • Check your build xaml file in.
Right, time to queue a new build. Remember to set the Is Base Line Run parameter to True for the first build. This should run all of your tests. Here is the build output for my first build.
firstbuild
This ran all 7 tests in my solution.
Now lets change some code. Here is the Test Impact View in Visual Studio showing that my changes have impacted two tests. I now checkin my changes.
vsimpact
Now I queue another build, leaving the Is Base Line Run at False (the default.) The build output now shows:-
success
Success!! We could setup a nightly build with the Is Base Line Run parameter set to True and have out impact data being updated every night.
Link to final xaml is here
3/10/2011

8 comments :

  1. To get this working properly in an environment where there are multiple build agents available to run a build, would the base line run have to be ran once on each build agent to collect the baseline test data? If the test impact data gets stored on each agent, then couldn't the list of impacted tests change based on which agent the controller happens to pick?

    ReplyDelete
  2. Hi Omeganot
    Test Impact data is associated with a build not the build agent. So it would not matter which build agent executed the build.

    ReplyDelete
  3. just linked this article on my facebook account. it’s a very interesting article for all.



    Scrum Process

    ReplyDelete
  4. Great article, even though I knew very little about modifying the build process XAML the instructions made this easy. One interesting problem hit though was when your build has many unit tests, TFS times out during the TIA SQL query and this entire process fails. You can read about the errors received here:

    http://connect.microsoft.com/VisualStudio/feedback/details/635483/get-impacted-tests-fails-with-operation-timed-out

    After 5 minutes it dies. In my case we have over 3000 unit tests. The threshold for making this work using your XAML appears to be about 200 unit tests.

    ReplyDelete
  5. Its a very good article but one problem i downloaded the build definition provided but it is not giving me impacted test cases and it is not even running test cases on setting Is Base Line Run to True/False both. or there is any other thing which i need to do to get the impacted test. I have enabled the Code coverage in settings file.

    ReplyDelete
    Replies
    1. Hi Mehul

      Are you running this on TFS 2010? Can you share a build log with me?

      Delete
  6. Great article!

    I have a problem getting impacted test. I never get anything.

    In "GetImpactedTest" activity there is a field called BaselineBuildDropLocation. Is it necessary set anything?

    Is it necessary a business or ultimate edition?

    Regards

    ReplyDelete
  7. One question I have is how do you share the test impact data from a nightly build running all of the tests with the gated checkin build that I want to only run tests being impacted by code change when each build definition writes output under it's own folder on the build agent?

    ReplyDelete