Some testers practice software testing. They think about it and research it. They write about it in papers or blogs. They practice actual testing techniques when they are at the dentist, movies, or airport. They want to achieve a higher level of performance.

In The New Brain, Richard Restak explains how practice affects our minds and what we do:

For the superior performer the goal isn't just repeating the same thing again and again but achieving higher levels of control over every aspect of their performance. That's why they don't find practice boring. Each practice session they are working on doing something better then they did the last time.

As he talks about practice and it's affects on the mind, he points out that modern research is starting to show that the best performers are those that practice, not those with some natural ability that they are born with. As I read this, I thought about all the times I've spoken with James Bach and how he is always practicing the art of testing and critical thinking.

There's another aspect to the superior performer, the ability to not respond on autopilot:

In order to achieve superior performance in a chosen field, the expert must counteract the natural impulse to gain an automated performance as soon as possible.

Because a superior performer is always trying to make something better, they never react with an automated response. If you think about something that appears to be automated like a heuristic response, you might be deceived. Great performers use heuristics to help them quickly generate ideas, but they don't rely on them without examining both the context of the situation and the heuristic itself.

For example, the other day I was practicing performance testing. It seems like a silly thing to do, but for me performance testing is incredibly interesting, I think it's going to be a required skill for testers in the future, and I simply want to develop better heuristic models for when I'm executing my testing. If you look at Scott Barber's patterns for scatter chart analysis, you will see that Scott has put a lot of time in developing heuristics that he can use to assess performance test results quickly.

While practicing with a performance test tool, I was simply running the same trivial script with different runtime settings to see if I could identify patterns between how my results appeared and what settings I used in the tool. If I can identify those patterns, I'm better equipped to debug script problems down the road when my tests are no longer trivial and time is much more important. While practicing, I noticed a couple of patterns and generated a couple of interesting ideas for performance test scenario development, so I sent them to Scott.

Now Scott could have looked at the data I sent him, compared them to his existing heuristic models, and provided me with an automated response as to what I was looking at. He did not. He asked what the script looked like that I was running? What settings was I using in the tool? Did I think about what the tool might be doing in the background? He asked about timing. He used a different heuristic that said, "When I see data, figure out what it is before applying pattern analysis to it."

If he had learned an automated response when he was first starting with performance testing and he had never developed heuristics beyond those initial patterns, he would not today be considered an expert in performance testing. Instead, he would be a really good performance tester, but he would most likely not be an expert and a thought leader in the field. By practicing performance testing, Scott continually tries to achieve higher levels of control over every aspect of his testing.

How do you practice your testing?

This month the Indianapolis Quality Assurance Association invited several prominent figures in the community to come and speak at our first program session of the 2005 year. We asked the panel to talk about their perceptions on the future of software testing and quality assurance. The panel was made up of:

• Scott Chesley - VP of Software Development, CTI Group


• Dr. Yuanshun Dai - Professor Computer Science IUPUI


• Charles Priller - Partner in MBA Performance Consulting


• David Schacht - CIO, Simon Property Group


• Bill Ritchie - Partner in Management Recruiters of Indianapolis

The panel was moderated by Mike Kelly . What follows is a summary of some of the questions that were asked and the responses of the panel.

Question from the audience: "Do you have any lessons learned in outsourcing?"

Priller:

"In my most recent project that I have been working on, and I sometimes don't like to talk much about this, I was engaged by a company to help them outsource about a 1,000 infrastructure jobs from the United States to a company in India. As we took a look at what they wanted to accomplish we found the main motivation for this was not financial like a lot of us often think it is, but it was for quality of service considerations. One of the things we found through this process is that quality on the first day, number of tickets closed, time opened, etc..., went down by a factor of 10. At the location in India, quality was a big issue. The people at the company, all the way to the top, talked about it and made it a big issue. They had a passion for making quality their number one priority. It made a lot of sense to do the outsourcing because they did do a better job of working problems, they did do a better job of developing a process, they did do a better job of training their people, and at the same time, their costs were less then the same services here in the United States. This company and other companies don't do outsourcing for financial reasons, it's about quality."

"Having gone through this recently, I think one of the lessons that I learned was that motivations of management were not truly bottom line financial, but really did revolve back to issues of quality. And I think better quality ultimately ends up in having a better financial situation. With all of the outsourcing efforts taking place, I think I have noticed that if you do them solely for financial reasons they tend to fail, if you do them to make quality better you have a lot higher chance of being successful."

Schacht:

"To echo some of what Chuck said, we have looked at offshoring, and when you look at offshoring just in terms of cost, for the most part what you see from the vendors is that they can do programming and IT services cheaper then you can and a lot of business people will hear a 50% reduction. What they're not taking into account, is that it's probably not practical to take the front-end business analysis and send it offshore. It's not really practical to take quality assurance and software testing, while perhaps it is on production support and packaged solutions, and take it offshore. I think there have been some successes there, and sometimes it makes sense, and not just from a cost perspective, but it's because they are highly educated, they do quality work, and there are some benefits of the time difference. It's not always lower cost, but the ability to turn things around more quickly and know that there is a high degree of quality."

Chesley:

"It looks really good on paper when you look at it initially, but when you look at some of the hidden costs, it doesn't materialize until much deeper in the process. You have to go into an outsourcing relationship for reasons other then for cost. Because many times those cost savings just aren't there. We have a really dedicated and highly skilled QA staff, and I wouldn't look at outsourcing them or the work they do on new development, but I would look at more when trying to outsource some of the more mundane work..."

Dai:

"There are two types of cost: the first type, are development costs and the second are risk costs. 60%-70% of development cost will be related to testing. Moreover, the risk cost may be much more than the development cost. For example, if a software product has a poor reliability and to often fails, then the company will lose its customers and the market, which is one typical type of risk cost. For some safety-critical missions, the risk cost will be huge even infinite, for example what if the software controlling the Nuclear Power Plant fails?"

In last November's CIO Magazine , from an article titled Fast Track Business Degrees comes the statement "Colleges are doing a poor job of preparing workers." The article claims that IT managers are looking for graduates that can apply the technology to solve business problems. What implications does this have for technology degree programs and do you find this to be a problem? What steps could someone in this position take to better develop their understanding of the implications of technology in business solutions?

Priller:

Mr. Priller spoke about his wife and her current experience in college. She will be graduating soon with an informatics degree from IUPUI.

"Not quite sure what she is being prepared for when she gets to the workplace. She does a lot with teams. I have a concern if I'm hiring a entry level person what have they really been prepared to do. From an educational perspective I have some concern that they may be being prepared in the wrong way."

Ritchie:

"Regardless of the technical ability developed in the college program; try to find an internship program that will help you integrate into the business community. I encourage anybody who can really ferret out a good internship to do so. We have found the business community to be very open to internships and they are much better preparation for the workforce."

Schacht:

"I tend to agree with Bill. I think to expect a young person to come out of a four-year program and to be able to apply technology to business problems is unrealistic. Applying business technology isn't just understanding the technology and it isn't just understanding the business, it's understanding how to work with people and how to deal with politics and many things. I don't see many people out of college equipped to do those things."

Chesley:

"One of the things I like to see, yes I like to see an understanding of technology that can be applied, but when they come talk to me that's not what I'm interested in. My focus is on is do they have a passion for what we are trying to accomplish? Are they truly energetic about the software quality profession and more importantly are they energetic and passionate about the organization they are applying for? Have they done any research on this organization? Resources are a premium and there are a lot of people out there looking for positions. I want somebody to come in there with a passion for the job and for the position to try to help my company excel at what we try to accomplish. Knowing they are just coming out of college it's important for them to have some understanding of the technology, but once that has been ferreted out, when they come to stand in front of my desk I want to make sure they have a passion for the position and for the product we make."

To follow up on what all of you have said, as a quality assurance professional, if I want to further my career (attain a higher position, more money, better job, or just not get outsourced), what types of skills should I be investing in? Should I be looking at technical skills? Should I be looking at business skills? Or should I be looking at soft skills (interpersonal communication and overall professionalism)?

Schacht:

"I think first of all I think you need to understand what you want to do. If you want to be a project manager then you will want to study with PMI..."

(My apologies. The recording cut out here and the rest of Mr. Schacht's response was lost. However, I thought it valuable to include that opening response.)

Ritchie:

"One of the things that I find that people tend not to do when they want to move out of a position or move up, is you have to make a point of it to tell people. You have to make people aware that you have that desire. A lot of people want to do something but they never tell anybody. You need to let people know what your plans are, what your desires are, and you need to get a career coach in that area and ask them, "How did you get there? What path did you take? Why did you feel you had the skills to be a project manager or the skills to be a developer?" I would encourage you to remember that education is an important point, but you need to continue growing and expanding your education. You need to make your desires known. Another thing, if you are willing to travel, often times opportunities are available where you can grow. Often times you will be able to be exposed to situations you would not be able to in one organization."

Chesley:

"You have to be willing to step out of your comfort zone and be willing to stretch your own ability. Challenge yourself to new endeavors. You can't be afraid to fail. You have to move forward and work to find a mentor to help you move forward and be willing to take the extra step to get things started. Don't be afraid to step outside of those boundaries. If you stay in your comfort zone, you're going to turn into one of those people when outsourcing hits and we're looking to reduce costs, we're going to take a long look at you and ask where have you grown and what have they done? The only way you are going to be able to grow and move in an organization is to step outside of that comfort zone."

Priller:

"In terms of outsourcing, if you do the same thing for more then two years in a row and you make pretty good money, someone may come along and I think you could be in trouble."

Question from the audience: "From your experience, what quality methodologies should I take the time to learn and get experience with?"

Priller:

"In 26 years in the business, I'm not the sharpest knife in the drawer when it comes to testing, but I have yet to see a tool that makes any of the stuff better. This last weekend, I was in the office of one of the executives of a local company that runs a PMO. They were showing us this diagram of how the whole system development lifecycle works, including testing and everything else, and they had this big chart on the wall. As I looked at it, they said well gosh isn't this great! To me it kinda looked like a design for a nuclear reactor. I said, "Well it's all great, but who the hell understands it?" Let's go down and get the vice president of customer service, let's go get COOs and those guys in here. They're the ones who need to understand how these processes work and need to be a part of them. Unfortunately what I run into in my business, especially from an outsourcing perspective, the guys in the CIO role tend to push off processes and methodology as the answer to all kinds of things. They understand them, and some of the business people understand them, but when you bring all of these things together to do something, they tend to complicate things even more. I've still not seen a tool or any type of produce that makes any of that easier."

Schacht:

"From my perspective, there are tools out there that make it administratively a little bit easier, but the person operating the tool still has to be as smart as they were before they had the tool and now they have to be smart enough to use the tool. At the end of the day, methodologies tend to level. What I mean by that is they tend to slow down the brighter people and elevate people who aren't quite as bright."

Ritchie:

"I think it's important from a career standpoint to get exposure to some of these methodologies. What I can say is I have organizations that are very happy with what they've done with certain methodologies and that some organizations are not going to be happy with any methodology. I've got to be honest with you, not everybody has as much control over what the market is going to require as a skill and you want to get as much exposure to these things as you can."

Question from the audience: "What tool(s) should I focus on?"

Ritchie:

"Without endorsing anybody's company, you might want to look at Mercury. They have been the most requested tool skill set for the last four year. Rational is a very good product to look at as well."

Question from the audience: "In your experience, what is the most difficult of quality management?"

Schacht:

"The largest problem is trying to find problems earlier. If you pay attention to the early parts of things, really all the way back to requirements and understanding the business drivers, that I think really pays the biggest dividends."

Chesley:

"Making sure quality assurance is involved in the initial design phase of the product. We have fallen short in the past and today we are trying to find places quality assurance can get involved in the initial stages and throughout the product. We need to figure out how to engage them early and then perpetuate that involvement throughout the project, working together, talking together, on a daily basis as we are putting out builds on a daily bases to get us more important information faster."

Question form the audience: "For general career positioning, in your experience what would be more competitive career wise: graduate degrees, certifications, certain training, certain experiences, what should we look at to better position ourselves in the market place?"

Schacht:

"I would tell you that I personally try to look for potential more then paper credentials. Paper credentials are important but the desire, the willingness, the aptitude are more important."

Priller:

"I've probably terminated more project managers with the letters PMP after their name then and anything I've come across. It's probably done a disservice, probably because people who have the ability to memorize things and pass tests can get a certification and are allowed to run projects without any real experience. As it was said earlier, it's really about people. It's not that I'm not an advocate of education, but the things that I look for are critical thinking skills and the ability to solve problems. Performance comes from an understanding of both the business and technology and we tend to look for both business skills and technical skills, not just one or the other."

Schacht:

"More important then textbook learning is a person's ability to think creatively and independently and to be good at problem solving. There is noting that pisses me off more then when someone comes into my office and tries to tell me about a best practice. A best practice in what situation? I've had a number of those people come to me and along the way I've helped them understand that a best practice or a textbook approach might be appropriate, given certain conditions. Let's talk about the appropriate solution for a given situation. We look for people who can think, and will think, and can leverage what they learn, not a reciter of textbooks."

Chesley:

"One of the things I look at is if you are posed with a problem what methods will you use to research and solve it. A lot of people will say, "Well I would go research it on the internet." You know what? That's not what I want to hear. I want to hear I have associates and a select professional network and I know just who to pick up that phone and call because they've been in a similar situations and I know they can help me through this. I want to know they are part of a professional network, they have contacts, they maintain those contacts, and make sure they utilize them because everything is not on the internet. I want to know the pros and cons of an actual experience."

Ritchie:

"We have a very enlightened panel here today and a lot of them seem to be looking for soft traits. I want to answer your question form a different perspective and it's a cumulative perspective of generally how you stay on top of things and is there a situation where people say, "If you don't have this, then we don't want to see you." And that is true. It's very difficult to get people in, and Scott said how he would evaluate you, but he also said "once you get your foot in the door." And I'm going to address the more general question of "How do you get your foot in the door?" You mentioned advanced degrees, and as it applies to the city of Indianapolis that applies to particular organizations, but definitely I would say that if you don't have a bachelors degree you need to obtain one. That would be my first advice. We are now seeing a big more towards that and it is a major factor. A masters is going to help you in certain organizations but as a general rule in Indianapolis it's not going to help you much. I would invest more in certification. Certifications are very important. I would get all of the certifications you can. If it's available to you, you should get it. You need to pay attention to what the marketplace is telling you. Right now that's the Mercury tools and any certifications around those tools."

Dai:

"A bachelor student is required to widely grasp the basic knowledge and skills in computer science, so that they are able to quickly work for any related companies after certain training. A master student is required to not only widely grasp the basic knowledge but also study one direction deeply as their thesis. Hence, they are able to apply the most advanced theory or technology where they go very deeply into practice. However, such specific topic on the other hand restricts them from all kinds of IT jobs if they want to make use of what they learnt in their Master study. A Ph.D. student is required not only the above two but also to contribute their own creative/novelty theories or techniques into the field. Therefore, Ph.D. is suitable for R&D dept of a company or becomes faculty in the universities. To be a programmer or system administrator will waste their talents and knowledge."

What's next? Five to ten years from now what software will I be testing?

Chesley:

"From my perspective, we will be doing more on the web. That means more security, more performance and reliability, and much more automated testing. The big thing right now is security and that needs to improve for money to be made on the web."

Schacht:

"Browser based applications are not going to go away. While I don't see my company going to grid computing (for security reasons) I do know that we have servers internally that are not fully utilized 100% if the time and I could see some effort made to better utilize those assists internal to us in some sort of grid environment."

Dai:

"In grid computing, it is impractical to rely too much on human configurations/test/debug/management because such systems are of very large scale and contain numerous components/users. So the autonomic testing/debugging/healing technologies are being developed now. The term autonomic computing is intentionally chosen because the idea is to mimic the autonomic nervous systems found in biology. The systems self-adapt, self-test, self-heal, and self-protect will be one of the future directions."

Chesley:

"You really need to concentrate on continuing education, stay up with current technology and what's current in the field. Don't rest on your laurels. Wireless will be huge. It's not coming, it's here. Everything we have today will be wireless tomorrow."

IQAA would once again like to thank each member of the panel for their contribution. If you attended the session, and would like to add comments, please do so in the comments section below.