Chapter 5 Testing the Software with Blinders On

Testing the Software with
Blinders On
Chapter 5

Highlights
 What is dynamic black-box testing
 How to reduce number of test cases by equivalence
partitioning
 How to identify troublesome boundary conditions
 Good data values to use to induce bugs
 How to test software states and state transitions
 How to use repetition, stress and high loads to locate
bugs
 A few places where bugs hide

Dynamic Black-Box Testing:
Testing the Software While Blindfolded
 Dynamic black-box testing
 Just need to know input and output
 Product spec should tell you
 Defining test cases
 Selecting test cases is the single most important task
that software testers do
 Use exploratory testing if you don’t have a spec
 Simultaneously learning the software, designing the
software, and executing those tests

Test-to-Pass and Test-to-Fail
 Test-to-Pass
 Assure only that the software minimally works
 Applying the simplest and most straightforward test
cases
 Always run test-to-pass cases first
 Test-to-Fail
 Designing and running test cases with the sole
purpose of breaking the software
 Strategically chosen to probe for common weakness in
the software
 Example

Equivalence Partitioning
 Equivalence classing
 The process of methodically reducing the huge (infinite)
set of possible test cases into a much smaller, but still
equally effective, set.
 An equivalence class or equivalence partition is a set
of test cases that tests the same thing or reveals the
same bug
 Partitions
 Extreme case
 Similar inputs
 Similar outputs
 Similar operation

Equivalence Partitioning (2)
 Examples
 Goal of equivalence partitioning
 Reduce the set of possible test cases into a
smaller, manageable set that still adequately
test the software
 Subjective
 It’s science but it’s also arts
 As long as the partitions can cover the
software being tested

Data Testing
 Two parts of software
 Data
 Program
 A few key concepts
 Boundary conditions
 Sub-boundary conditions
 Nulls
 Bad data

Boundary Conditions
 Assumption:
 If software can operate on the edge of its capabilities, it
will certainly operate well under normal conditions
 Basic Example
Dim data(10) As Integer
Dim I As Integer
For i = 1 To 10
data(i) = -1
Next I
End

Types of Boundary Conditions
 Note: boundary conditions are those situations at the
edge of the planned operational limits of the software
 Look for the following types:
 Numeric, Speed, Character, Location, Position, Size,
Quantity
 First/Last, Min/Max, Start/Finish, Over/Under,
Empty/Full, Shortest/Longest, Slowest/Fastest,
Soonest/Latest, Largest/Smallest, Highest/Lowest,
Next-To/Farthest-From
 Tips: If you have a choice what data you’re going to
include in your equivalence partition, choose data
that lies on the boundary

Testing the Boundary Edges
 Testing the data points just at the edge of the
boundary line isn’t usually sufficient
 Good idea to test on both sides of boungdary
 Two equivalence partitions
 Work properly --- last one or two valid points
inside the boundary
 Expect to cause an error --- the one or two
points outside the boundary

Testing the Boundary Edges (2)
 Outside the boundary
 First-1/First+1
 Start-1/Finish+1
 Less than Empty/More than Full
 Even Slower/Even Faster
 Largest+1/Smallest-1
 Min-1/Max+1
 Just Over/Just Under
 Even Shorter/Longer
 Even Sooner/Later
 Highest+1/Lowest-1
 Note: The more boundaries you discover, the more
bugs you’ll find

Sub-Boundary Conditions
 Boundaries that are internal to the software
aren’t necessarily apparent to an end user
but still need to be checked by the software
tester
 Knowledge about how software works
 Powers-of-Two
 ASCII Table
 Unicode

Default, Empty, Blank, Zero and None
 Default testing
 Windows Paint Attribute dialog box example
 Always consider creating an equivalence
partition that handles the default, empty,
blank, null zero, or none conditions
 Why
 It’s likely a different software path is followed

Invalid, Wrong, Incorrect and Garbage
Data
 Garbage data --- Test to Fail
 Good software should handle whatever a
user can do to it
 For users are not always as smart as you
suppose
 Users won’t blame themselves, they blame the
software
 Software wants numbers, give it letters

State Testing
 Verify the program’s logic flow through its
states
 A software state is a condition or mode that
the software is currently in
 Example
 The software follows a path through the code,
toggle some bits, sets some variables, loads
some data, and arrives at a new state of
being

Testing the Software’s Logic Flow
 It’s usually possible visit all the states
 But it’s impossible to traverse all paths to all
states
 Solution:
 Apply equivalence partition techniques to the
selection of the states and paths

Creating a State Transition Map
 Should be specified in the specification
 State transition diagrams can be drawn by
using different techniques
 Examples
 Should show:
 Each unique state that the software can be in
 The input or condition that takes it from one
state to the next
 Set conditions and produced output when a
state is entered or exited

Reducing the Number of States and
Transitions to Test
 How to reduce the huge set of possibilities to
a set of test cases of workable size
 View each state at least once
 Test state-to-state transitions that look like the
most common or popular
 Test the least common paths between states
 Test all the error states and returning from the
error states
 Test random state transitions

What to Specifically Test
 Checking all the state variables
 Static conditions, information, values,
functionality, etc
 Windows Paint startup state example
 The Dirty Document Flag example
 Discuss your assumptions about the states
and state transitions with your team’s spec
writer and programmers

Testing States to Fail
 Testing-to-pass before
 Trying valid possibilities
 The flip side
 Find test cases that test the software to fail
 Race condition
 Repetition
 Stress
 Load

Race Conditions and Bad Timing
 Race condition
 Multiple processes racing to a finish line, not
knowing which will get first
 Bad timing
 Two or more events line up just right and
confuse software that didn’t expect to be
interrupted in the middle of its operation

Race Conditions and Bad Timing (2)
 Situation that might expose race conditions:
 Saving and loading the same document at the
same time with tow different programs
 Sharing the same printer, communications
port, or other peripheral
 Shutting down or starting up two or more
instances of the software at the same time
 Using different programs to simultaneously
access a common database

Repetition, Stress and Load
 Repetition testing
 Doing the same operation over and over
 Detect memory leaks
 Stress testing
 Running the software under less-than-ideal
conditions
 Determine what external resources and
dependencies it has
 Load testing
 Feed it all it can handle

Repetition, Stress and Load (2)
 Two important considerations
 Your team’s programmers and project
managers may not be completely receptive to
your efforts to break the software in this way
 Opening and closing your program a million
times is probably not possible if you’re doing it
by hand

Other Black-Box Test Technique
 Not standalone
 Finding stragglers
 Subjective and not necessarily based on
reason

Behave Like a Dumb User
 Throw out any preconceived ideas
 Make no assumptions

Look for Bugs Where You’ve Already
Found them
 Two reasons to look for bugs in the areas
where you’ve already found them:
 The more bugs your find, the more bugs there
are
 Many programmers tend to fix only the specific
bug your report, no more no less

Miscellaneous
 Think like a Hacker
 Seek to find vulnerabilities and exploit it
 Experience and intuitions can’t be taught
 Try different approaches