Lecture 6 - Classifying the Most Common Types of Testing -> Quick Intro -> By Knowledge Of The Internals Of The Software -> By The Object Of Testing– Black box testing
– White box testing
– Grey box testing
BLACK BOX TESTING
Imagine a soda vending machine. If you want a can of Coke, you insert money into the machine and get your soda. Nothing fancy. Now here’s another perspective:
– The coins that you insert into the machine are INPUT.
– The soda is an OUTPUT.
– The buttons to select type of soda and the slot where you insert your coins are the USER INTERFACE (UI).
– The internals of the vending machine is like a black box, because you have a vague or no idea about the concrete mechanism that exchanges coins for soda.
In regard to software, the black box (or area of unknown) is simply a software back end. From a black box tester perspective, the back end can be thought of as a virtual bridge that connects INPUT and OUTPUT.
There are two main things about black box testing:
1. The tester usually has no idea about the internals of the back end.
2. Ideas for testing come from expected patterns of user behavior.
Let’s look into the details.
1. The tester usually has no idea about the internals of the back end.
On the one hand, the tester has an advantage over the programmer, i.e., the author of the back end code. Why? It’s human nature to see desired things as reality. The programmer wants to see his code working. Every parent thinks that his child is the smartest and the most talented. The code is the child of a programmer, and in his reality the programmer often perceives his code as a problem-free creature.
BTW
Here is my favorite legend about the power of perception:
When the ships of Columbus’ expedition stopped in front of one of the Caribbean islands, the natives didn’t see those ships because their perception did not include images that were completely different from what they and their ancestors had been observing for thousands of years. Only their priest felt that something was wrong, and he kept scanning the horizon until he could distinguish the silhouettes of the Spanish frigates from the sea and sky. Then he said, “Check it out, dudes: Columbus with his expedition,” and only after that could those natives see reality as it was.
The black box tester has no “bonds” with the code, and a tester’s perception is very simple: a code MUST have bugs. Using the principle, “Ask and you shall receive,” black box testers find bugs where programmers don’t.
BUT, on the other hand, black box testing is like a walk in a dark labyrinth without a flashlight, because the tester doesn’t know how the back end was actually constructed. That’s why there are situations when
– A black box tester writes many test cases to check something that can be tested by only one test case.
– Some parts of the back end are not tested at all
Therefore, black box testing has the advantage of an unaffiliated opinion on the one hand and the disadvantage of blind exploring on the other.
2. Ideas for testing come from expected patterns of user behavior.
What we’ve been calling “steps” is a combination of 3 things:
– Actions
– Data and
– Conditions
combined together to achieve an actual result.
Example
ShareLane -> Do this:
1. Go to https://www.sharelane.com/cgi-bin/main.py
2. Type word “expectations” into the “Search” text field.
3. Click “Search” button.
Steps 1-3 inclusively are actions. The text “expectations” is the data. If scenario assumes that book is in DB, then the condition is: DB has data about book with word “expectations” in its title.
Scenario is a combination of actions and data applied to software under certain conditions
The purpose of a scenario is to bring test execution to the point where an actual result can be retrieved and compared with an expected result.
Expected patterns of user behavior are simply scenarios that we expect will be (OR are already) taking place as users use our software.
Thus, we can expect patterns of user behavior regarding software that
– will be released in the future
OR
– has already been released
to our users.
How can we figure out expected patterns of user behavior?
a. We can take them from the spec – e.g., the PM can assume the some users will enter an email address with two “@” characters.
b. We can figure them out by exploring. When you browse your Web site, you can imagine what a regular user would do.
c. We can develop scenarios by using a black box methodology. We’ll cover this extensively.
d. Scenarios can be a gift from our intuition. You can just wake up in the middle of the night and think: “What if a user would do that?” IMHO, intuition is one of the greatest assets that software tester might have.
e. The PM or programmer might give you some valuable ideas.
f. There are many other sources. For example, you can read an article about how users interact with similar Web sites.
BTW
In some cases, after we have released the software we might discover some actual patterns of user behavior. For example:
– The developer who works on reporting can share with you the results of live data about how users are using the site.
– The customer support folks can give you that information.
– You can read customer feedback on Internet forums, blogs, etc.
We can use knowledge about the actual patterns of user behavior to:
– Change our existing test cases (that we created based on expectations about patterns of user behavior) and/or
– Add new test cases
We would use these changes and/or additions to increase the probability of bug finding.
Let’s do a quick recap about conceptual issues regarding black box testing.
The black box approach assumes that the tester usually doesn’t know how the back end was written, so ideas for testing come from expected patterns of user behavior. Expected patterns of user behavior are scenarios that we expect will be (OR are already) taking place as users use our software.
Here’s another illustration of black box approach.
Example
Spec #2233: “5.1. If user adds from 20 to 49 books (inclusively) to the shopping cart, we will offer a 2% discount.”
ShareLane -> Now, do this:
1. Create a new account on ShareLane and login (you can use Test Portal>Helpers>Account Creator to create the new account and login automatically.)
2. Add any book to the shopping cart.
3. Click the “Shopping Cart” link.
4. Change the number of books to 20.
5. Click the “Update” button.
6. Check the value under the “discount” column.
Expected result: 2
We just performed black box testing by executing a scenario when a user adds 20 books to the shopping cart. The expected result was 2, but is the code really bug-free? Read on!
WHITE BOX TESTING
White box testing (also known as “glass box testing,” “clear box testing,” and “open box testing”) encompasses a number of testing techniques that require a comprehensive understanding of the software code. For example, a programmer can perform white box testing by comparing:
– the requirements from the spec
– a piece of Python code from ShareLane
Do you want to perform some white box testing now? Let’s do it!
Example
Spec #2233: “5.1. If user adds from 20 to 49 books (inclusively) to the shopping cart, we offer a 2% discount.”
ShareLane -> Now, do this:
1. Go to Test Portal>Application>Source code>shopping_cart.py
2. Click the “BUG #1” link under the “View bugs” section.
Here is what we see:
If q >= 12 and q <= 49: discount = 2
Programmer Billy used “q” as a variable to hold the value of the quantity of books added to the shopping cart. In human language the expression means:
IF the quantity of books is greater than or equal to 12
AND it is also less than or equal to 49
THEN the discount should be equal to 2
Congratulations! We’ve just found a bug using the white box testing approach! Billy made a mistake by writing 12 instead of 20. What is the bug summary? “Spec2233: shopping_cart.py: 12 instead of 20 was given as lowest limit for a 2% discount.“
During black box testing we used “20” as an input and it was a legitimate scenario, BUT we didn’t find a bug! We would have found a bug if we had tried “19” – i.e., a value that was not mentioned in spec. The bug was actually found during white box testing. That brings us to two very important conclusions:
1. If we simply develop test scenarios using direct ideas from the spec, we will create legitimate test cases, but those test cases won’t necessarily be effective bug finders. The job of the PM is to describe how the features should work, not how the features should be tested. In other words, the PM develops USE CASES, not test cases.
Brain Positioning
A use case is a description of :
– How software will be (or is being) used
– How software should respond to certain scenarios
Example
Here is an example of a use case: “During login, if a user submits the wrong password, an error message ‘Oops, invalid username or password’ should be displayed.'”
A functional spec often includes a collection of use cases.
Use cases and test cases are similar, because they share the same concept: a certain scenario should lead to a certain expected result.
Use case and test cases are different because:
– The purpose of a test case is to find a bug, while the purpose of a use case is to describe how the software should respond to a user actions.
– A test case is used to test the software, while a use case is used to describe the software.
You can read an excellent article about use cases on Wikipedia.
The real power in finding bugs is invoked when we use the professional body of knowledge known as the black box testing methodology. The value of “19” originates as a test idea because I used the special black box testing technique called Boundary Values. Read on, and soon you’ll become experts in many cool testing techniques!
2. Black box testing and white box testing are a great combination that helps to find bugs by improving:
– Test comprehensiveness – i.e., checking the software from different angles
– Test coverage
Brain Positioning
Let’s elaborate on test coverage.
Imagine a chessboard with 64 squares with white king on one of the squares. Each possible position of the king on the board is written on a separate card in the form of an instruction – e.g., “Move white king to E2.” There should be a set of 64 cards to cover all of the possible positions. If we start moving the king using the positions specified on our cards, then, after all cards are used, we will achieve:
– 100% of the possible positions of the white king on the chessboard
– 100% execution of the given instructions
Now, imagine that the chessboard is so big that the number of squares is incalculable. Let’s also imagine that, according to some logic, we selected 20 positions and wrote them down on 20 cards.
Can 20 cards cover 100% of the possible positions of the white king on the chessboard? No.
But can we achieve 100% execution of the given instructions? Yes.
Now, back to software testing.
The term “test coverage” means one of the following based on the context:
1. The coverage of possible scenarios
2. Test case execution coverage
1. For the coverage of possible scenarios, we have two situations:
a. Usually, the number of possible scenarios is incalculable. Thus, we usually don’t really know the exact percentage of possible scenarios that we are going to cover by our testing.
b. In some cases, the number of possible scenarios IS calculable – especially when we narrow down our testing to a small, isolated piece of the software. In those cases, as we prepare test cases we might confidently name the percentage of possible scenarios to be covered during our testing.
In both situations, the coverage of possible scenarios is improved if we add a valid unique test case – e.g., a test case that:
– Checks possible scenario
– Does not duplicate another test case
When we talked about how the black box/white box combo helps to improve test coverage, we also talked about improving the coverage of possible scenarios.
2. It’s much simpler with test case execution coverage.
We always know how many test cases we have and how many of them have been executed.
So, when you are asked about test coverage, ask what that person means: the coverage of possible scenarios, or test case execution coverage.
In real life, white box testing usually exists in the form of unit testing performed by the programmer against his own code.
GREY BOX TESTING
This is my personal favorite, and I consider it the most effective approach that I’ve seen during my entire career. One of the main reasons why I created ShareLane was to provide you with a grey box testing experience.
Grey box testing combines the elements of black and white box testing.
– On the one hand, the tester uses black box methodology to come up with test scenarios.
– On the other hand, the tester possesses some knowledge about the back end, AND he actively uses that knowledge.
BTW
Do you remember Test Case with Credit Card? We did DB verification – i.e., we used knowledge of the back end for our testing. That was a typical example of the grey box approach!
Below are two things to conclude our quick intro to black, white, and grey box testing.
1. During black box testing, the tester’s actions are NOT limited to actions that can be performed by the users. For example, ShareLane testers use Credit Card Generator (Test Portal>Helpers>Credit Card Generator) to generate a credit card for testing. Obviously, users don’t have access to our internal test tools.
2. One of the most critical things to keep in mind while doing black, white, or grey box testing is to come up with expected results that serve as true indicators of whether the software works or not. Please pay attention to this.
Example
In the Test Case with Credit Card, we checked the DB value. We did that because the true indication that the credit card transaction was successful was reflected as a second digit in the result column of the table cc_transactions – e.g., in the case of a successful Visa transaction, the value must equal “10”. Why is this a true indicator? Because if the transaction is successful, the credit card processor sends us a success code that equals “0”. We can say, “We used the grey box approach,” because we used DB verification.
If we used a pure black box approach, we would only be able to verify that, as a result of successful checkout, a user gets the confirmation page with an order ID. Will that confirmation page be a legitimate expected result? Yes. It IS an expected result that makes sense. But would it be a true indicator that the credit card transaction was successful? No.
Again, it does make sense to check if the confirmation page is displayed (in Test Case with Credit Card, we did that indirectly, because we got an order ID from that page), but a true confirmation of the success of the transaction can only be retrieved from the DB.
That’s why I’m such a strong believer in the grey box approach – it provides more opportunities to find those expected results that serve as true indicators of whether the software works or not. Next ->
Lecture 6 - Classifying the Most Common Types of Testing -> Quick Intro -> By Knowledge Of The Internals Of The Software -> By The Object Of Testing
