Today was the first day of the WHET #4 (Workshop on Heuristic and Exploratory Testing). The topic for the workshop is boundary testing, and it's currently attended by:

• Henrik Andersson


• Ross Collard


• Tim Coulter


• James Bach


• Jon Bach


• Scott Barber


• David Gilbert


• Dawn Haynes


• Doug Hoffman


• Paul Holland


• Karen Johnson


• Cem Kaner


• Michael Kelly


• Sam Kalman


• Rob Sabourin


• Keith Stobie

I can sum my impression of the conversation today with some Francis Bacon. I think we spent a lot of time exploring idols of the Marketplace and idols of the Theater.

Idols of the Marketplace are errors arising from the false significance bestowed upon words, and in this classification Bacon anticipated the modern science of semantics. According to him it is the popular belief that men form their thoughts into words in order to communicate their opinions to others, but often words arise as substitutes for thoughts and men think they have won an argument because they have out talked their opponents. The constant impact of words variously used without attention to their true meaning only in turn condition the understanding and breed fallacies. Words often betray their own purpose, obscuring the very thoughts they are designed to express.

Idols of the Theater are those which are due to sophistry and false learning. These idols are built up in the field of theology, philosophy, and science, and because they are defended by learned groups are accepted without question by the masses. When false philosophies have been cultivated and have attained a wide sphere of dominion in the world of the intellect they are no longer questioned. False superstructures are raised on false foundations, and in the end systems barren of merit parade their grandeur on the stage of the world.

We spent a lot of time talking about definitions, the scope of "boundary testing", and looked at a couple of real examples. It was good, but I felt like we got hung up a lot on the idols. More can be found here.

I went to order a pizza last night and the website I normally use to do so had a new interface. It appeared to me that they had switched from what I believe to be a third-party solution to a homegrown solution.

Here's what immediately went through my mind when I saw the change:

"Great, the company is going to loose my personal data because they are new to this and they probably don't know anything about security."

"That also means I need to use the Discover Card instead of the check card, because I now suspect they'll loose my financial data as well. Discover is more forgiving of such things and has taken care of me in the past. Where's that card...."

Then I tried to login:

"Great, my user id doesn't work any longer. And that means I get to re-enter all the data that this company is going to then lose. Wonderful..."

"I assume that also means they won't have my order history. Now I have to recreate my custom pizza. Why they don't have a simple barbecue chicken pizza on the menu is beyond me, but I get to create one all over again (because it was so much fun the first time)."

Once I re-registered and got in, then the real pain started. I had flashbacks to dialing up on a 28.8K connection. Only, that might have been faster. Pages actually stopped trying to load before the download finished. They just gave up. I don't even know how you could program your site to make that happen, but they did.

Now, I'm really lazy. And when it comes to pizza at 8:30 at night after a 13 hour day, I'm really not out to find a new pizza place. But I was game, I tried two other local pizza places who I thought might have online ordering. No good. I was forced to use a phone.

I called one of the other pizza places on the principle that any company silly enough to release an application like this into production should lose my business and the business of everyone I know. I also called them because they had a barbecue chicken pizza.

Why is this interesting to me?

It's not often that I get to recognize my own performance expectations as an end user. I remember Rolland Stens sharing a technique he uses where he adds code to demo pages to slow them down on purpose. It's then that the customer who says "under six seconds" changes their tune to something like "That's six seconds! It needs to be three...."

Interestingly, I only walked away when I could no longer use the site. I probably would have finished the transaction if I could have, but then never gone back. I only gave up because I had no choice. I couldn't force the website to take my money. It just wouldn't load correctly.

Another aspect of this experience was embarrassment. I'm a performance tester. I cause pain like this when I do bad testing. I was also shocked by how little I trusted the site knowing that it was homegrown, and yet I still tried to use it. I was going to give them my money (or Discover's money, but they know how to collect from me). I expected them to lose my information. It's that common now. How sad is that for our industry?

All said and done, I filled out the feedback form. It was short and sweet. Probably not fit for print here. I received a reply from a customer relations person thanking me for the feedback and giving me a gift certificate. Apparently that's what's missing from my performance test plans. A risk mitigation strategy of discounts, rebates, and free food.