Fitnesse is an Acceptance Testing Framework that uses a table driven approach. It isn’t really one of the cool kids anymore having lost that title to the likes of SpecFlow, StoryQ and other BDD based frameworks.
You can however use FitNesse for BDD style tests. Here is a video from uncle bob on the subject.
One of the main reasons that I still like FitNesse is that it is tester friendly. By this I mean that non-coding testers seem to take to it well.
I am going to assume that you are familiar with Team Build, FitNesse (and Slim) for the remainder of the post.
I am also going to assume that you know the basics of MTM.
Another feature that I wanted to try out as part of this was the Build-Deploy-Test workflow on physical environments. This has just been released by Microsoft (CAUTION: UNSUPPORTED)
I have taken from this excellent post from Jerry Bullard.
Also the workflow for Build-Deploy-Test was published here
Follow the steps in Darshans blog to get your environment setup. NB Remember to configure the Test Agent as process rather than a service. It will need to interact with the desktop to invoke FitNesse. Stop once you have the environment configured in MTM like this.
We will write an exe that wraps the REST call to Fitnesse. This exe will then transform the Fitnesse result using an XSLT file. The exe will be called from a Generic Test. This gives us VS/TFS integration in terms of deployment, running etc. All we then need to do is make sure the results are in a Summary Result Format.
I have used a Slim tutorial sample and have one test passing, one wrong and two ignored. This should give us a good sample of results.
There are a few utilities that you will need to get this solution working. I have attached all of them (including source) to the link at the end of the post. None of the code is exciting or interesting so I am not going to go through it here. The utilities do the following:-
GeneralFitTestRunner.exe – executes the FitNesse tests using the REST api. Arguments are
- Machine that FitNesse is running on. This should be localhost or 127.0.0.1.
- FitNesse wiki page
- test or suite to indicate if we want to run a test or whole suite.
Usage example:- GeneralFitTestRunner.exe 127.0.0.1 8082 FrontPage test
ProcessStarter.exe – starts a process. Arguments are
- Path to exe
Usage example :- ProcessStarter.exe java.exe –jar c:\fitnesse\fitnesse.jar –p 8082
Run.bat – This invokes the ProcessStarter.exe and passes in the arguments to start FitNesse. You may need to change the paths in here for your environment.
CopyWiki.bat – Uses xcopy to copy the FitNesse wiki from the droplocation of the CI Build onto the integration server. You may need to change the paths in here for your environment.
Stop.bat – Stops FitNesse.
Deployment of Utilities
All these scripts assume that FitNesse is installed on the c:\FitNesse directory on the integration server.
Create a folder called c:\Tools on your integration server and copy all of the utilities into it except run.bat and stop.bat. Here is the folder on my server.
Place run.bat and stop.bat into a directory that can be referenced via a UNC from both your development machine and the integration server.
Setting up the Test
We can now create our Generic Test. If you open the TestFitIntegration project and look at the FitTest generic test you will see:-
We are pointing to our GeneralFitTestRunner and passing the arguments to run the FiNesse test.
Note the Additional files to deploy section:-
This is the file that FitNesse will use to execute its tests. i.e.
We need to add this to (at least one) generic test to make sure that it gets deployed for us.
The Results settings indicate that we will generate a Summary Results file. Make sure that the name is Results.xml as shown.
My solution file contains the following FitNesse artefacts.
I am using this for deployment. I have customised my CI build to copy the FitNesse folder to my build drop folder. I am not going to step by step this. The xaml is attached so you can use the same one.
If you don’t do it this way you will need to find a way to make sure that your latest wiki content gets pushed to your integration server.
Now we need to start building our test plan. You should already have an environment setup so go ahead and create a new Test Plan and Suite. Mine looks like this:-
Opening the test shows us:-
If we navigate to the Associated Automation tab you can see that I have associated my generic test with this test case. Here is one way to do this.
Next we have to sort out the Test Settings and Environment. If you choose Properties of the plan
From here we can update our Test Environment. Choose the environment that you have created. Make sure that this is for Automated Runs
Click the Open link next to the Test Settings for Automated Runs.
Ensure that the Radio button is selected for Automated. Select the Role that you assigned the machine in your environment.
Choose any relevant Diagnostic collectors
Click on the Deployment menu underneath the Advanced option:-
I am using this to deploy the FitNesse folder (just the wiki) that has been copied to a UNC share from my CI Build – see above) This ensures that I am running the latest version of my wiki.
Next move to Scripts.
These are the run and stop.bat scripts that we copied to an available UNC share earlier.
The run.bat script invokes the ProcessStarter exe which runs java – jar etc. The reason for needing this is the way test settings work. The test will not be run until the script finishes. Obviously this is not good for us as FitNesse will stay open – and we need it to. So we need the script to fire off an exe and then exit. This will load FitNesse and run our test.
N.B. Starting FitNesse this way means that it ‘unpacks’ each time. To cope with that I have added a delay in the ProcessStarter. A better solution would be to read the output and see when its finished unpacking.
The stop.bat stops FitNesse.
Configuring our Builds
We now need to setup our builds.
The flow that we are looking for is this:-
- Run a successful CI Build
- Run an integration build. This will deploy the CI Build to a physical environment setup in MTM and then run my generic tests – which in turn execute FitNesse.
Creating our CI Build
Create a new build definition for our CI Build. Use the RunIntegrationBuildProcessTemplate.xaml that I have provided. This should build the TestFitIntegration solution that I have provided. In your build definition you will need to specify a directory to copy the FitNesse wiki to. This should be a UNC accessible folder. This folder is referenced in the copywiki.bat script. Make sure that these folders match.
Creating our Integration Build
Create a new build definition and use the BuildDeployTest_Physical.xaml file.
My Integration build is configured like this:-
As you can see I have targeted a specific number of my CI Build (20110316.28.) This is just for testing. Unfortunately this xaml does not support just getting the latest build, but depending on your build numbering that would be a trivial change to make. The other option is to have the integration build run the CI build first. If I chose this option my config would look like this:-
We need to specify some properties for this build. Firstly choose whether you want your CI build to run first or target a specific build. These sections (1,2 and 3) are fairly self explanatory – if you need any more detail have a look at Darshans blog. In Section 4 – Deployment – we want to add our deployment script – copywiki.bat. See the screenshot above for an example. We give the Build Agent Name that we want the script to execute on, and the directory and name of the script. Make sure to set the Run Deployment Scripts parameter to True.
Section 5 is for testing. Set Run Tests to True. Test Environment name is the MTM environment that we setup in the beginning. The Test Run Parameters has a nice dialog to help us. Select the test plan and suite that we created in MTM. Make sure that you select the correct Test Settings.
Right that’s it!
Running the Build
A successful integration build looks like this:-
As you can see we have details of what build we have deployed from (our CI build) and the test run details. Clicking on the View Test Results link opens MTM with the following:-
Great! So we can see the actual FitNesse breakdown of results. Because this was executed by a Test Plan we get all the features of MTM e.g. it will show up in the Analyse Test Runs View.
Not to mention all the testing reporting. (But that’s another post)
Hopefully this has shed a bit of light on two main techniques. Firstly how we can execute an external test runner like FitNesse, but also how we can use the (unsupported remember) Build Deploy Test xaml on physical environments.
Links to all code here.