Simple ROI Model for Testing Automation Projects

In a tutorial on self learning at CAST 2006, I was asked to analyze the first two printed pages of the article Simple ROI Model for Testing Automation Projects by Guy Arieli. To be clear, the material was not complete on purpose. The provided material stopped at

"I hope you didn't get lost in all the formulas. But in order to use this module on a project all you have to enter are the following parameters:"


The following is my general analysis based on what I was given. You will most likely want to have a copy of those two pages as I work through this.



I wasn't given the ability to ask context related questions, so I needed to make some assumptions in my analysis and figure out what my mission is.

MISSION:

For the purpose of my analysis, I'll ask the questions:

  • "Would I find this useful in my daily work?"

  • "Based on what I can interpret given the information available, can I imagine a use for this material?"

  • "How does this compare to my work on the same topic?"



Jon Bach was also given the analysis task. I have the deepest respect for Jon as a tester, so I want to impress him by doing a better job of analysis then he does.

TECHNIQUE:

The first thing I did was identify the variables in use. I created a list of all the variables I saw and how the author defined them. I could identify 16 variables. One of them (BMF) I could not find a definition for. Here is that list:

  • ROI = return on investment

  • TB = total benefits

  • TC = total costs

  • FC = cost to develop automation framework

  • TBB = cost to develop the building blocks of the tests

  • TTD = cost to develop the tests

  • TMF = test maintenance factor

  • TD = efforts to develop new tests for the new version

  • BMF = ?

  • BB = building blocks development efforts

  • M = manual efforts

  • NF = new feature factor

  • i = version

  • TDM = the ratio between tests development efforts to manual efforts

  • BBM = the ratio between building blocks effort to manual efforts

  • RF = the relevancy factor



Next I asked myself if I would know how to calculate (or come up with) all of these variables for any given project I am working on or have worked on:

  • Total benefits: How would I know this at all? What about new applications (not just new versions)? What about non-quantifiable benefits? What about the benefits I don't see?

  • Total costs: How would I know this? What about opportunity costs (the cost of the other things I could have been doing instead of this automation)? What about the costs to other projects or applications within the organization?

  • Manual efforts: Manual efforts for who? Doing what? How do I calculate this? Where do I get the numbers? Based on how many projected bugs? Based on what kind of a relationship with the developers? Based on what level of ineffective management?

  • and so on...



What do the ratios tell me, what are the units, and do they make sense:

  • The NF, or New Feature Factor, seems to be weeks over weeks. That one looks ok.

  • The TDM, or test development effort over manual effort, seems to be weeks over weeks. That one looks ok.

  • The BBM, or building blocks development effort over manual effort, seems to be weeks over weeks. That one looks ok.

  • The RF, or Relevancy Factor, seems to be weeks over weeks. That one seems ok as well.



While I don't know what they all represent, at least the units work out.

Next I scanned the document (or the portion of the document I had available) for inconsistencies. I noted the following:

  1. The title and the first paragraph both say that this is a simple equation. Look at the following equation (this is the full equation after substitution):
    ROIi = ((TB(i-1) x RFi x Mi) - (TC + ((1 + BMF) x TBB(i-1) + (NFi x Mi X BBM)) + ((1 + TMF) x TTD(i-1) + (NFi x Mi X TDM)))) / (TC + ((1 + BMF) x TBB(i-1) + (NFi x Mi X BBM)) + ((1 + TMF) x TTD(i-1) + (NFi x Mi X TDM)))
    Does that look simple to anyone reading this? I think not.

  2. In the first paragraph the author indicates that the question "Will it be profitable?" usually rises when working on a test automation project. This is inconsistent with my experience. I've worked on over 12 different large automation projects at over eight different companies and I've never heard anyone ask if automation would be profitable. This doesn't pass the BS test. It's a highly tuned heuristic for claims about automation.


  3. In the second paragraph the author states that when the ROI becomes positive, the project will be worth investing in. I don't understand this for two reasons. ROI will never be positive until later in time. Thus, it would never make sense to start an automation project. Odd... Second, let's assume I'm just misunderstanding things (which I may be doing on purpose at this point), instead why don't I examine the way I justify an automation effort. Here it goes: "Does the automation free me up to do some other more valuable testing within the time I have?" or "Does the automation allow me to run a test I can't run manually that I think would be valuable?" If either of those are yes, automate. If they are both no, don't. My equation seems much simpler then his...


  4. In the fourth paragraph, the author states that he found this model to be useful in determining what to automate first. I don't see anything in any of these equations that allows me to assign priorities to different automation items. Perhaps this is in the part of the article I'm missing, but it's certainly not justified by the ROI calculation we see in the first two pages. I have a hard time understanding how someone could use this to determine what to automate first.



Finally, I quickly applied my FCCCUTSVIDS heuristic to see what I missed:

  • Feature tour: This made me think of the variable listing I had already done. I couldn't think of anything else that I might call a feature.


  • Complexity tour: This is too complex to be called simple. Tour over.

  • Claims tour: Did this...


  • Configuration tour: How does the equation change based on different methods of automation and reuse? Is this just for regression testing only or is this suppose to also be useful for ad hoc test automation where I might throw a test away after I run it?


  • User tour: Is there a human that could use this? Possible users might include a test manager, project manager, or an academic who needs to publish a paper.


  • Testability tour: I could test this with Excel, real project data, and it would seem to be easy to automate (no pun intended).


  • Scenario tour: Apply real project data from past projects and see what the result is. Compare that to actual results.


  • Variability tour: Didn't spark any new thoughts.

  • Interoperability tour: What size projects is this intended for? Is this only for 10 year 20 person automation projects?

  • Data tour: Real project data seems useful and simple data seems like it would show that this isn't worth looking into. I don't even know where I could find all this data for my projects. I don't even know what all the benefits and costs are for automation, how can I plug in those numbers? What's the scope of this data? Is it just for my project? For a project that reuses some of my components? The organization?


  • Structure tour: Let me show it to you one more time...
    ROIi = ((TB(i-1) x RFi x Mi) - (TC + ((1 + BMF) x TBB(i-1) + (NFi x Mi X BBM)) + ((1 + TMF) x TTD(i-1) + (NFi x Mi X TDM)))) / (TC + ((1 + BMF) x TBB(i-1) + (NFi x Mi X BBM)) + ((1 + TMF) x TTD(i-1) + (NFi x Mi X TDM)))
    ... you have got to be kidding me.



Total analysis time was about 10 to 15 minutes.
Software TestingMichael KellyComment