Integrate MSpec with TFS 2010 Team Build

3/12/2011

BDD has been gaining traction for a long time and MSpec is one of my favourite frameworks.

Unfortunately there is no compatibility with MSTest for running within Visual Studio.  This also means that you cant run it easily using Team Build and have your test results recorded as part of the build.  Well now you can, as I am going to show you how to integrate MSpec and Team Build.

Firstly a disclaimer:-

  • This is an example solution that works well for me.  Your mileage may vary.

And now some acknowledgements:-

Background

This post will assume that you are comfortable customising Team Build etc.  I will try and include help links where I can but I am not going to explain the whole step by step process.

You can either follow along, or just grab the attached files and go to the How To Configure section to get it working for you.

Design Assumptions

  • Any MSTest unit tests should still work.
  • The build should handle multiple MSpec assemblies.
  • There should be a configurable way to tell the build which assemblies are MSpec (Similar to this)

regex

  • The Test Results should be able to viewed inside Visual Studio (e.g. a trx file.)

NUnitForTFS

We have a nice head start because even though this was written for NUnit, it is easy to extend.  There is an xslt file that transforms from the NUnit runner format to the trx format.  I wrote another xslt that transforms from MSpec.  The other code works pretty much untouched.  Unfortunately the MSpec xml output is not as rich as either MSTest or NUnit and so lots of information cant be populated.

The one change that I had to make concerns the test run start date.  This information is not supplied by the MSpec runner.  You also cannot create a date value with xslt 1.0.  The code in the NUnitTFS library expected a certain format.  So I took this function

private static DateTime GetStartDate(XDocument trx)
        {
            var name = GetName(trx);
            return DateTime.ParseExact(name.Substring(name.IndexOf(' ') + 1), "yyyy-MM-dd HH:mm:ss", null);
        }

and changed it to

private static DateTime GetStartDate(XDocument trx)
        {
         var name = GetName(trx);
            try
            {
                return DateTime.ParseExact(name.Substring(name.IndexOf(' ') + 1), "yyyy-MM-dd HH:mm:ss", null);
            }
            catch (System.FormatException)
            {
                return DateTime.Now;
            }

        }

This will return the current datetime for MSpec tests.

Everything else is a command line option.  I will go into these when we look at the xaml file.

Build machine Changes

Obviously MSpec will need to be installed onto the build server.  I copied all of the binarys into c:\Machine.Specifications-release.  I also copied NUnitTFS into this folder.

Build Changes

Open the xaml attached to this blog and navigate down to the Try Run Tests section.  There are 2 copies of the Run MSpec Tests shape in both branches of MSTest execution.  This ensures that regardless of TestLists / TestSettings our MSpec tests will be run.  The Run MSpec shape looks like this

runmspectests

The shapes break down like this.  The FindMSpec Tests mimics the MSTest method of using a regex expression to locate the correct test assemblies.

regextests

A build definition parameter has been created to take input from the user on the correct regex.  Again this is to mimic the MSTest behaviour.

A variable called MSpecAssemblies has been created in the build workflow to hold a List<string>.  This is the names of the assemblies that match the regex.

The rest of the workflow (with the expanded Run MSpec) is here.

restworkflow

If Test Assemblies Found checks if any test assemblies match the Regex.  If so we need to flatten our List<String> containing the assemblies to run.  We need to pass them to the MSpec test runner.

flatten

This shape builds up a string containing the list of assemblies.

invokemspac

We are now ready to call the MSpec test runner.  Our Invoke Process parameters look like this:-

Arguments

args

Nothing should be surprising here.  By putting the results file into the build output directory (binaries) we are making sure that there shouldn’t be any interference across builds.  The last parameter is our list of MSpec test assemblies.

Publish MSpec results

publish

This is our call to NUnitTFS.exe to transform and publish our test results.

Arguments:-

publishparmts

These parameters should be pretty self explanatory.  A word of warning here, you will need to specify a configuration on your build definition as below.

buildconfig

The last if shape should be pretty clear.  If the tests do not pass then the build will update the status, so that the build will be marked as partially successful.

Whew, ok so lets now run it.

How To Configure

Copy MSpec onto your build machine.

Copy NUnitTFS onto your build machine (ensure that it is a modified version – see above.)

Update the NUnitTFS.exe.config urls with your TFS addresses, e.g. change http://tfs:8080/tfs/TestManagement/v1.0/TestResults.asmx to point to your tfs instance.  There are 3 urls to update.

Create an new build definition.  Make sure that it has a  configuration set.

buildconfig

Complete the following properties in the build definition.

defparms

This is the path to the MSpec test runner on the build machine,  the regex format to find MSpec test assemblies, the folder where the xslt resides, and finally the path to the NUnitTFS.exe.

Run the build.

Output

Here is a screenshot of a finished build that contains a single MSTest and multiple MSpec test assemblies.

completedbuild

When you click the View Test Results link you see:-

testresult

These are our BDD tests!  If we double click on a failed test we see the following:-

failedtest

We can drill into the source just as with MSTest!  Success!

Limitations

  • Start / End time of individual runs – you may have noticed from the screenshot above that the test ran on 1/1/2008 at 3am.  This is because we are not getting the time from the MSpec output (because it doesn’t have it.)  If you need this information then another update to NUnitTFS will be required.
  • Fail Build on Test Failure – This is a property that can be set on the build definition.  It means that the build will be failed completely if any tests fail.  For some reason this does not currently work.  I think that it is looking for something MSTest specific.  I do not need it and so haven’t tracked it down.  Leave a comment if its important and I’ll see what I can do.

Exercises for the Reader

It might be nice if all of the xaml was packaged up into a custom activity.  I will leave this as an exercise for the future…

Links

xaml - here

NUnitTFS - here

3/12/2011

5 comments :

  1. This is fantastic. I am working on getting this running and will comment with more detail as soon as I am able to get it all wired up.

    WONDERFUL!

    ReplyDelete
  2. Nice work!
    Why does the platform has to be set to Any CPU?
    I work with Silverlight and it should say x86.

    ReplyDelete
  3. Hi mseeli - The platform does not have to be any cpu, you just have to specify at least one valid platform in the build process parameters. I chose any cpu for the example.
    The reason for this is because the Build Deploy Test xaml that microsoft released uses an api call that requires that parameter. People often leave that parameter blank as it is optional in a standard build process.

    ReplyDelete
  4. Thanks for taking the time to post this. It helped a lot.

    ReplyDelete
  5. Does this guide work for TFS 2013 as well?

    ReplyDelete