Last weekend we held the April session of the Indianapolis Workshops on Software Testing. The attendees were:

• Joe Batt


• David Holscher


• Brian Marick


• Kenn Petty


• Charlie Audritsh


• Kartik Gaddam


• Rajasimba Admal


• Dana Spears


• Michael Kelly

The topic we focused on for the five-hour workshop was testing in an agile environment. About half of the attendees are currently working in an agile environment of some kind, and the other half were new to the topic and were interested in learning more. The following is a summary of presentations and ideas shared.

First to talk were David Holscher and Joe Batt. They wanted to preview their talk on Continuous Integration With CVSNT, CruiseControl, Ant, JUnit, JFCUnit and More on Microsoft Windows which is based on a talk they will be presenting later this year at the JavaOne Conference.

They have made their slides available, so I don't want to go to deep into the material, but their shared experience covered setting up continuous integration in a windows environment focusing on version control, build automation, automated testing and deployment, repeatability, and fault tolerance. They used a laundry list of tools to make it all happen (all open source I think), and they seem to be very pleased with the results.

All of the testing on the project was developer testing, they had no formal tester. They used JUnit for unit testing and TDD (which they did most of the time, but not all of the time) and JFCUnit for GUI level test automation. They relied (and still do I think) mostly on test scenarios derived from use cases and the test cases that result from TDD. One of the things noticed by Joe was the heavy focus on building testability into the system. Since it was the developers doing the testing (especially the GUI automation), a little more time was spent getting the design right and less changes occurred downstream.

Brian Marick recommended Mike Clark's book Pragmatic Project Automation for more information on the topic if you have questions after looking at their slides.

Following the continuous integration presentation, Brian Marick talked a little about a whole lot of things, ranging from Test First Programming to Test Driven Design (for the developers in the crowd) to the differences between "developer testing" and "tester testing" and where testing fits in agile methodologies in general. Brian, being the prolific blogger that he is, has most of his brain available online, so I can link to just about everything he talked about (and more):

Design-Driven Test-Driven Design:
http://www.testing.com/cgi-bin/blog/2005/03/17#presenter1
http://www.testing.com/cgi-bin/blog/2005/03/26#presenter2
http://www.testing.com/cgi-bin/blog/2005/03/30#presenter3
http://www.testing.com/cgi-bin/blog/2005/04/15#presenter4

Agile testing directions:
http://www.testing.com/cgi-bin/blog/2004/05/26#directions-toc

Brian of course mesmerized the workshop (myself included) but I had to cut him off so we could here the final experience report. I imagine we could have listened to him for at least another five hours! Check out the links above (especially the agile testing directions) and get a feel for where he's coming from. I'm certain Brain will answer any well thought out question on the material.

Following Brian's talk, Kenn Petty presented his experience implementing Session Based Test Management (SBTM) and pair testing in a rapid development environment. Kenn provided all the proper links and credits to where he harvested his ideas. He then jumped into what worked and what didn't.

Kenn's team recently finished their first iteration using pair testing and SBTM. Previously they followed (as best they could given time constraints) the IEEE 829 format for test documentation and the model that is commonly associated with the management and execution of those test plans and cases. Using pair testing meant that for most of the testing (90% maybe?) that takes place is sessions, there has to be two people: they can be two testers (this is happened a lot) or a tester and a developer (this happened some).

• Collaboration and mutual respect increased between the testing and development teams.


• The testers found more meaningful bugs faster.


• Overall bug counts increased as did the average severity of defects found.


• Because collaboration improved, the number of defect tickets marked "Functions as designed" went from 70 on the previous iteration down to 3.


• The testing team became more motivated because they focus shifted from documentation to finding bugs and collaborating with development. They felt like they were learning new skills from each other and they felt more challenged.

Testing took place in sessions, as outlined in the Satisfice material on the topic, and at the end of each session testers were debriefed by Kenn. Charters were initially generated by Kenn, and then during the execution of a charter, the testers could identify spin-off charters on their own, or during debriefing Kenn and the testers could identify new charters. Each requirement change (tracked mostly by email) generated a charter of its own.

Overall I thought it was the best workshop so far. I think I have said that each time, but I mean it each time. I would really like to thank Brian, Joe, and David for making the trip to Indianapolis for a five hour workshop. Brian didn't only come for the workshop, he also came to see the dinosaurs in downtown Indy. I don't know if the workshop was better then the Gorgosaurus (I've seen him and he's quite impressive), but I hope everyone had fun.

Last weekend we held the March session of the Indianapolis Workshops on Software Testing. The attendees were:

• Joshua Rafferty


• Cheryl Westrick


• Chad Campbell


• Jason Halla


• Allen Stoker


• Dana Spears


• Michael Kelly

The topic we focused on for the five-hour workshop was unit testing. It is important to note that half the attendees were developers and the other half testers, all working in the Indianapolis area. The following is a summary of presentations and ideas shared.

We started the workshop off with Allen Stoker providing a high-level overview of unit testing using a test framework (NUnit, JUnit, etc...) so everyone was on the same page. Nothing really eventful was discussed here, but several issues cropped up around unit test complexity and when unit testing should take place. We parked those issues and moved ahead.

Once everyone was working from the same understanding, we broke up into pairs (one developer and one tester) and actually did some unit testing. Each developer wrote some sort of widget code and then the developers and testers took turns writing unit tests for the widget. The developers guided the testers through the process, creating both negative and positive test cases and identifying issues and answering questions along the way. It was interesting to see what some developers considered a widget, the simplest widget belonging to Allen Stoker who simply returned the same value passed into the component. And the most complex widget going to Jason Halla whose widget had a random number generator included in it.

There were some issues identified after the exercise of writing the unit tests. Those issues were:

• How do we bridge the gap between testers and developers and actually get this level of collaboration on real world software projects?


• Developers suffer from the same oracle problem testers suffer from.


• It is difficult for developers (even though they do it every day) to explain unit testing to testers. All three developers struggled with explaining their thought process and their tacit knowledge around unit testing.


• Unit testing is done at such a low level that it is difficult to test real business functionality (unit tests test the code, not that the requirement is correct).


• Why isn't there more unit testing done?

We did not come up with many answers. We instead parked those issues with the others and moved ahead with the presentations. Some of the experience reports addressed some of those issues.

Cheryl Westrick then shared her experience helping developers with unit testing at her company. She is currently involved in a project where the testers were given the code as it was developed and they then logged defects as they found them. The unit testing was manual and the team identified and logged thousands of issues. Unfortunately, the testing still took place in an "over the wall" mentality. The testers would get builds, log defects, and then the process would repeat. No one worked side by side to resolve issues or establish continuous communication. Overall, Cheryl did not seem overly excited about the results of the effort, but during discussion, it seemed that her team was able to gain insight into the strengths and weaknesses of the developers, something previously difficult to do. We also talked about how the early insights into the problem areas in the software would affect their system testing going forward. Would they concentrate testing on areas where defects had clustered during unit testing? Would they scrutinize certain developer's code more closely?

Next, Chad Campbell shared his experience on testing private versus public methods while unit testing. Some believe in the process of only testing publicly visible methods. Others believe unit testing should cover all executed pieces of code. The testing of private methods can be difficult if you are not an experienced developer and Chad shared his experience using reflection in .Net to test private methods. Chad sparked some lively debate on the topic, with the Java developers curious about the specific details around his use of reflection. This took us into the very fascinating topic of unit test theory. Chad and I maintain that from a TDD standpoint, unit tests are there to simplify design and add confidence in your ability to refactor, so there is value in testing private methods. Allen and Jason sat on the other side of the fence offering the view that unit testing at that level is really not valuable in that all you are really interested in is the correctness of the functionality exposed by the public interface and that as long as that interface is thoroughly tested you have all the coverage you need. We agreed to disagree, concluding that different contexts for development will necessitate different unit testing methods (so really not much of a conclusion).

This gave us a good transition into Allen Stokers experience report on effective unit testing. Allen maintains that validating a build should be a convergence of a couple of pieces of information. You want to see a successful test run, but that's not enough. A successful run means that all of your test cases passed, but what you still have no real idea of what you tested. Allen made the case that this is typically where coverage analysis comes in.

The classic argument is that a detailed coverage report will give you a strong indication of the test case quality by identifying what code was exercised. Unfortunately, it will not tell you what was tested. The term tested implies far more than the execution of code. A proper test case requires that the results of the code execution are known ahead of time, and that the scenario actually verifies those results. A good test case will typically exercise a block of code multiple times by changing the conditions of the execution on each pass. From this, you can see that a good secondary indicator is the number of times each line of code is execution during the unit test execution.

High coverage % + High line execution count = success ?

According to Allen, no.

The problem is that one can write an entire suite of test cases that exercise the application at a high coverage level, and even execute the code multiple times (meeting the noted formula), but doesn't really verify anything. Building effective test cases takes a significant amount of time and thought to ensure that the right things are being tested. In the end, according to Allen, there is only one real solution, and that is human intervention. Allen recommended each use case include the following quality controls:

• Detailed design reviews before coding (if the design is bad, it may be plagued with issues forever).


• Initial code reviews (early identification of problem areas and bad practices)


• Final code reviews including:
  
  
  
  • Full source code of component
  
  
  • Full JUnit source code of component
  
  
  • Coverage analysis results
  
  
  
  

This may require the dedication of a senior team member to this activity alone during the entire project. It's a significant investment. Unit testing is no different from any other coding effort and it requires both learning about the API, and learning about effective testing. In addition, Allen advocated developers having their unit tests reviewed by an experienced tester as a mentoring process. There are probably many ways you could automate analysis of the unit test cases, but the only way to truly know that you have an effective test suite is to review the test cases manually.

Our discussion of Allen's experience focused mostly on the final recommendation of having unit tests reviewed by an experienced tester as a mentoring process. This generated discussion between the testers and developers at the workshop on the different sets of knowledge we own and cultivate and how that knowledge can best be leveraged. We tried to imagine how to effectively implement such and environment. Code is quite often serves as a wall between tester and developer collaboration. As a group, we felt that testers would almost need to have a developer background to get involved in the code at this level.

We also identified a second wall that was created by developers. Typically, if you go to a tester and engage them in this type of activity, "Hey, can you help me with my unit testing? I don't know if I'm testing the right stuff..." the tester will be excited to be engaged. On the other hand, if you attempt to engage a developer, our shared experience has been that the developer will respond, "Just let me code and leave me alone...." We talked about how to tear down this wall and we decided that this could only be done on a one on one basis. Individual testers and developers will need to decide to collaborate and to learn from one another. We see very little that corporate or project leadership can do to facilitate this behavior.

We then looked at a problem solving opportunity presented by Dana Spears. Dana asked the group how an organization might develop standards and metrics for unit testing. Was there a flexible enough process that could be implemented on the organizational level to address effective unit testing? The short answer to the problem is sadly, no. It was the conclusion of the attendees of the workshop that unit testing, like any testing activity, tends to be qualitative and not quantitative. We generated a lot of discussion on the topic, but we could only come up with more questions then answers.

Finally, we finished with a tool demo. We looked at component testing with Rational Application Developer. This tool is an excellent example of the types of tools needed to get developers and testers working together. Using this tool, developers have the ability to create complex component-test scenarios, select sets and ranges of data for input, and have ready access to the static metrics for coverage analysis. In addition, testers can easily provide input in the form of data selection techniques like boundary-value analysis, domain partitioning, and selecting data sets and ranges for optimal test case coverage.

We find tools like this one valuable in that developers and testers don't necessarily have to work side by side (although there wouldn't be anything wrong with doing that), but instead we envision a scenario where developers develop component tests on their own and then work with testers to define the test data to be used in the test. Data can be entered directly by the tester, thus extending the component test, or the tester can send comments on data selection to the developer and the developer can enter the data.

Working this way can result in the following advantages:

1. Increased appreciation and understanding of skill sets


2. Cross-training and knowledge dissemination


3. Better and more meaningful component tests


4. Possible reuse of component test data for functional testing downstream in the project lifecycle


5. Better coordination of all testing efforts based on coverage statistics and other static metrics


6. Better software

Next month's topic is on testing in an Agile environment. We already have a couple of experience reports lined up from the good folks at CTI Group, and I am hoping we can get some good lessons learned on the topic and identify some of the issues around shifting the testing culture here in the Indianapolis community. If you would like to participate in an IWST workshop, please feel free to contact me.