Integrate Fitnesse with TFS 2010 Team Build–aka How to use Build-Deploy-Test workflow with physical environment

3/16/2011

Fitnesse  is an Acceptance Testing Framework that uses a table driven approach.  It isn’t really one of the cool kids anymore having lost that title to the likes of SpecFlow, StoryQ and other BDD based frameworks.

You can however use FitNesse for BDD style tests.  Here is a video from uncle bob on the subject.

One of the main reasons that I still like FitNesse is that it is tester friendly.   By this I mean that non-coding testers seem to take to it well.

I am going to assume that you are familiar with Team Build, FitNesse (and Slim) for the remainder of the post.

I am also going to assume that you know the basics of MTM.

Another feature that I wanted to try out as part of this was the Build-Deploy-Test workflow on physical environments.  This has just been released by Microsoft (CAUTION: UNSUPPORTED)

Acknowledgements

I have taken from this excellent post from Jerry Bullard.

Also the workflow for Build-Deploy-Test was published here

Environment

Follow the steps in Darshans blog to get your environment setup.  NB Remember to configure the Test Agent as  process rather than a service.  It will need to interact with the desktop to invoke FitNesse.  Stop once you have the environment configured in MTM like this.

env

Fitnesse Approach

We will write an exe that wraps the REST call to Fitnesse.  This exe will then transform the Fitnesse result using an XSLT file.  The exe will be called from a Generic Test.  This gives us VS/TFS integration in terms of deployment, running etc.  All we then need to do is make sure the results are in a Summary Result Format.

I have used a Slim tutorial sample and have one test passing, one wrong and two ignored.  This should give us a good sample of results.

fitresults

Utilities

There are a few utilities that you will need to get this solution working.  I have attached all of them (including source) to the link at the end of the post.  None of the code is exciting or interesting so I am not going to go through it here.  The utilities do the following:-

GeneralFitTestRunner.exe – executes the FitNesse tests using the REST api.  Arguments are

  • Machine that FitNesse is running on.  This should be localhost or 127.0.0.1.
  • FitNesse wiki page
  • test or suite to indicate if we want to run a test or whole suite.

Usage example:- GeneralFitTestRunner.exe 127.0.0.1 8082 FrontPage test

ProcessStarter.exe – starts a process.  Arguments are

  • Path to exe
  • args

Usage example :- ProcessStarter.exe java.exe –jar c:\fitnesse\fitnesse.jar –p 8082

Run.bat – This invokes the ProcessStarter.exe and passes in the arguments to start FitNesse.  You may need to change the paths in here for your environment.

CopyWiki.bat – Uses xcopy to copy the FitNesse wiki from the droplocation of the CI Build onto the integration server.  You may need to change the paths in here for your environment.

Stop.bat – Stops FitNesse.

Deployment of Utilities

All these scripts assume that FitNesse is installed on the c:\FitNesse directory on the integration server.

Create a folder called c:\Tools on your integration server and copy all of the utilities into it except run.bat and stop.bat.  Here is the folder on my server.

tools

Place run.bat and stop.bat into a directory that can be referenced via a UNC from both your development machine and the integration server.

Setting up the Test

We can now create our Generic Test.  If you open the TestFitIntegration project and look at the FitTest generic test you will see:-

generic

We are pointing to our GeneralFitTestRunner and passing the arguments to run the FiNesse test.

Note the Additional files to deploy section:-

additional

This is the file that FitNesse will use to execute its tests. i.e.

fitnesse

We need to add this to (at least one) generic test to make sure that it gets deployed for us.

The Results settings indicate that we will generate a Summary Results file.  Make sure that the name is Results.xml as shown.

My solution file contains the following FitNesse artefacts.

solution

I am using this for deployment.  I have customised my CI build to copy the FitNesse folder to my build drop folder.  I am not going to step by step this.  The xaml is attached so you can use the same one.

If you don’t do it this way you will need to find a way to make sure that your latest wiki content gets pushed to your integration server.

MTM

Now we need to start building our test plan.  You should already have an environment setup so go ahead and create a new Test Plan and Suite.  Mine looks like this:-

suite

Opening the test shows us:-

testcase

If we navigate to the Associated Automation tab you can see that I have associated my generic test with this test case.  Here is one way to do this.

Next we have to sort out the Test Settings and Environment.  If you choose Properties of the plan

properties

From here we can update our Test Environment.  Choose the environment that you have created.  Make sure that this is for Automated Runs

autom

Click the Open link next to the Test Settings for Automated Runs.

settings1

Ensure that the Radio button is selected for Automated.  Select the Role that you assigned the machine in your environment.

role

Choose any relevant Diagnostic collectors

diag

Click on the Deployment menu underneath the Advanced option:-

deploy

I am using this to deploy the FitNesse folder (just the wiki) that has been copied to a UNC share from my CI Build – see above)  This ensures that I am running the latest version of my wiki.

Next move to Scripts.

scripts

These are the run and stop.bat scripts that we copied to an available UNC share earlier.

The run.bat script invokes the ProcessStarter exe which runs java – jar etc.  The reason for needing this is the way test settings work.  The test will not be run until the script finishes.  Obviously this is not good for us as FitNesse will stay open – and we need it to.  So we need the script to fire off an exe and then exit.  This will load FitNesse and run our test.

N.B.  Starting FitNesse this way means that it ‘unpacks’ each time.  To cope with that I have added a delay in the ProcessStarter.  A better solution would be to read the output and see when its finished unpacking.

The stop.bat stops FitNesse.

Configuring our Builds

We now need to setup our builds.

The flow that we are looking for is this:- 

  • Run a successful CI Build
  • Run an integration build.  This will deploy the CI Build to a physical environment setup in MTM and then run my generic tests – which in turn execute FitNesse.

Creating our CI Build

Create a new build definition for our CI Build.  Use the RunIntegrationBuildProcessTemplate.xaml that I have provided.   This should build the TestFitIntegration solution that I have provided.  In your build definition you will need to specify a directory to copy the FitNesse wiki to.  This should be a UNC accessible folder.  This folder is referenced in the copywiki.bat script.  Make sure that these folders match.

Creating our Integration Build

Create a new build definition and use the BuildDeployTest_Physical.xaml file. 

My Integration build is configured like this:-

fitserver

As you can see I have targeted a specific number of my CI Build (20110316.28.)  This is just for testing.  Unfortunately this xaml  does not support just getting the latest build, but depending on your build numbering that would be a trivial change to make.  The other option is to have the integration build run the CI build first.  If I chose this option my config would look like this:-

runbuild

We need to specify some properties for this build.  Firstly choose whether you want your CI build to run first or target a specific build.  These sections (1,2 and 3) are fairly self explanatory – if you need any more detail have a look at Darshans blog.   In Section 4 – Deployment – we want to add our deployment script – copywiki.bat.  See the screenshot above for an example.  We give the Build Agent Name that we want the script to execute on, and the directory and name of the script.  Make sure to set the Run Deployment Scripts parameter to True.

Section 5 is for testing.  Set Run Tests to True.  Test Environment name is the MTM environment that we setup in the beginning.  The Test Run Parameters has a nice dialog to help us.  Select the test plan and suite that we created in MTM.  Make sure that you select the correct Test Settings.

Right that’s it!

Running the Build

A successful integration build looks like this:-

intbuildsuccess

As you can see we have details of what build we have deployed from (our CI build) and the test run details.  Clicking on the View Test Results link opens MTM with the following:-

mtmresults

Great!  So we can see the actual FitNesse breakdown of results.  Because this was executed by a Test Plan we get all the features of MTM e.g. it will show up in the Analyse Test Runs View.

analyze

Not to mention all the testing reporting.  (But that’s another post) Smile

Conclusion

Hopefully this has shed a bit of light on two main techniques.  Firstly how we can execute an external test runner like FitNesse, but also how we can use the (unsupported remember) Build Deploy Test xaml on physical environments.

Links to all code here.

3/16/2011

1 comment :

  1. Hi, thanks for the great article execatly what I was looking for! I am also interested in the code files, but the google docs link is offline could you please fix it?
    Thx
    SH

    ReplyDelete