To Joel as promised:
Applied Chaotic Testing:
How many times have you run a test pass for your product and bugs have slipped by that could have been caught if others had been testing as well? Big corporations can afford large teams or extended beta programs. The little guys are strapped for money and resources are stretched with no time.
How to get the maximum return on testing?
One way is to shake up the patterns we all fall into. Implement what I like to call "Applied Chaotic Testing". As much as one may feel they are being creative and representing their user base as domain experts, there are certain techniques that are ingrained in us that make our testing results measurable and predictable. We all know that is does not reflect real world usage. How many times has customer support reported issues that come from the field that when researched come from environments that could not be predicted? I remember my days in support taking a call from the field where the user was running an English version of our software in a Danish version of Windows 95 running from a Japanese version of Dr. DOS. Who does that? This was an excellent example of an "Applied Chaotic User". Another was from a heroic fellow that had no hands and was struggling to calibrate my company's touch screen with a pen in his teeth. Turned out to be the wrong version of the driver.
As a tester with 22 years in software testing, I have seen many unique testing methodologies introduced into the field: Unit testing, Systems testing, Exploratory Matrix testing, Moge-Warwick Centralized. Analysis, Certified Emulation testing, RGB testing, Lowest Common Denominator testing (LCDT), and Micro testing. But overall there is still a need to capture the majority of chaotic users.
Remember that an IQ of 100 is average, so this should be the goal when test planning. To increase the target zone, plan for 85 instead. Don't forget to factor in one-off anomalies like managers of companies in the Fortune 100 and the elderly.
Applying Applied Chaotic Testing:
At irregular intervals, an Applied Chaotic Tester will change his/her testing approach by doing the unexpected. A sample scenario might look something like this:
Test with the mouse in the left hand
Reconfigure your keyboard to be Chinese
Turn off the monitor and try using the software via a screen reader with the volume set low
Try testing on an operating system in a language you can't read. I find Arabic or Hebrew most effective given these languages are right to left.
Try testing under duress. Many users have deadlines that keep them awake extra longs hours. Try testing in sleep deprivation of 24 hour increments. Eating poorly also can create a source of duress. Try an entire build cycle eating nothing more than Hostess HoHos and drinking Red Bull.
Create a software configuration that is prone to random crashing. A great solution is to test on virus-ridden operating systems. A quick source of viruses is AOL or several shadier porn sites, but clear this with the IT department to ensure it does not conflict with company policies.
Get small children as beta testers. Be aware and sensitive to the fact that criticism on their performance can make them cry. Same guidline goes for ESL (English as a Second Language) students.
Ask (if possible) to sit in on your company's support center. Listen with a careful ear for customers with attention deficit disorder preferably from non-English speaking countries and/or the clinically diagnosed bi-polar because they will have the most challenging questions and having the most questionable scenarios. Ideally the tech will share these same qualities. Take many notes and as detailed steps as possible. Don't worry if you miss a few steps here and there. It just leads to better Applied Chaotic testing.
To sum up, Applied Chaotic Testing taps into the undiscovered country of random and unmeasurable use cases. Forgot how your software is suppose to be, or even should be, think of how it is.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment