TFS 2010 - Test Impact Analysis Series - Part 4

12/30/2010
Well this series is getting longer than I thought. And still so much to cover!
Ok, we are up to the server portion. How does TIA (and also IntelliTrace) capture all that gold.
The answer is the CLR profiler! There is a good article on some of the internals here

How does the profiler work (You knew I was going to ask that didn't you!) Well the profiler gets invoked by setting some environment variables on a process (more on this later) When those are set, when the process is restarted the profiler tracks a whole bunch of data (as seen in the linked article) its this data that enables IntelliTrace and Test Impact analysis.

What are the environment variables? There are two. The first is called COR_ENABLE_PROFILING. When this is set to 1 then we are ready to profile. The second is called COR_PROFILER and this is the guid of the profiler being used. This is set to a guid. In our case that guid resolves to TraceLogProfiler.dll with a ProgId of Microsoft.VisualStudio.TraceDebugger. This is where the magic happens!

All sounds great so far. So what actually happens from MTM. Here is the order of events as I see them when TIA enabled.

  1. Run Test is selected in MTM.
  2. The Test Controller restarts IIS on the webserver hosting the application to be tested. (You always wondered why this needed to be an Admin account didn't you?)
  3. Tester Launches IE on local machine.
  4. Proxy setting is changed on local IE (see part 3 for more details)
  5. IIS worker process is started with correct environment variables set.
  6. Trace profiling is started.
  7. Tester ends test.
  8. Tester may re-run test
  9. Tester chooses Save and Close or just close in MTM
  10. The Test Controller restarts IIS again
  11. Trace profiling stops

Wow thats quite an intense sequence! So the observant amongst you will be saying What - restarting IIS - yes that is happening (not just once but twice!) Well ok, I can kind of see why...

But, and this is a pretty big But...

What can go wrong

Suppose you have more than one tester on your team. Large teams will have lots of testers. What happens if I am running a test with TIA enabled and another tester starts running another test case from the same plan which also has TIA enabled. Will IIS get restarted out from under me?

Funnily enough I tested this. When this happens, we get another TestRunCollectorErrorWarning.xml created and no TIA gets created. Inside this file is the following warning:-

The current Test Runner session has a conflict with another Test Runner session
that is already using the same environment. You might not get the expected
data and diagnostic information that is configured in the associated test
settings.

Ok...So best not do that then. But how would you know that before you started your test? This seems like an extremely limited feature to me. Basically it seems like TIA (and therefore IntelliTrace) are single user features. So to generate a good regression suite over a large application would take a very long time with only one tester at a time. There is also no automatic synchronisation method. The team would have to manually tag when one tester has finished and another one wants to start.

Disclaimer:-

This is the result of my investigations so far. I have asked questions of those brigher and better connected into the product units than I am. Hopefully I will be proved wrong. When that happens I will post an update here. Until then, well I am going to recommend that we use TIA very sparingly and carefully...

But theres more. Tune into Part 5 for details on Custom Account App Pools.

12/30/2010

No comments :

Post a Comment