How to Become a Software Tester

Lecture 7 - Test Preps -> Quick Intro -> The Tester's Mental Attitude -> Intro To Special Skills In Bug Finding -> Black Box Testing Techniques -> When You Start To Implement Testing Techniques...

DIRTY LIST/WHITE LIST

This method has two stages:

            1. DIRTY LIST

This is very simple – type into a text file ALL ideas about testing that occur to you during (or after):

– Spec reading

            – Exploration

            – Getting feature info from other sources

Do not filter or analyze stuff; just allow your creativity and inspiration to flourish. The phrase “ALL ideas” has the widest meaning: test case ideas, ideas for scenarios, ideas about expected results, ideas for automation helpers, questions for PM and developer, WHATEVER. Don’t think that some ideas might be ridiculous or impractical. This is a brainstorm where you allow your imagination to soar without any restraints.

The content of the text file* will be the dirty list.

*Some folks (like me) prefer to use paper for the dirty list/white list method. Whatever works!

            2. WHITE LIST

Here you start to analyze what you’ve come up with during your dirty list creation. Things that make sense are transferred into another text file called a white list. Your analysis will be based on a variety of things: the requirements from the spec, common sense, other test methods (e.g., Boundary Values), communication with the developer and the PM, etc. The bottom line is that you allow on the white list only those things that in your opinion will help in some way or another to find bugs.

When you transfer items to your white list, you may want to:

– Modify them

– Group them

– Add more detail to them

Very often, the first white list turns into a second dirty list, so you create another white list based on this, and so on, until you get a finished product that can assist you to create great test cases and do other test preps (e.g., write an automation helper). It might be a little tricky to decide when your white list is finished – there is always room for improvement. My own approach is simple: I start working on test cases as soon as I feel that my white list gives me enough details to proceed with.

Let’s start mastering the dirty list/white list technique right now. Take a sheet of paper and spend 15 minutes creating a dirty list of ideas on how to test a soda vending machine. For example:

            – Does the buyer get the correct type of soda?

            – What if the buyer presses the “Buy” button two times?

            – Check to make sure the buyer gets correct change.

            – Check to see whether foreign coins are accepted.

After your dirty list is done, spend another 15 minutes creating a white list. And then, spend 30 minutes creating test cases using this format:

This exercise is really useful. Whatever you do in the future, the dirty list/white list method will serve you well in any kind of project that requires planning!

TEST TABLES

Here is a mock-up of the first page of “Sign up” flow (flow means scenario):

Let’s put possible conditions and inputs into the first table:

Table 1

As you can see, there are two subgroups:

ZIP code entered? is about condition.

Entered value is about data.

Each element (e.g., “Yes” for ZIP code is entered?) has its own unique ID (e.g., zip_elem_1). I chose “zip_elem” as a short form for “zip code table element”.

Now let’s combine the elements from Table 1 into finished scenarios and add some details to the new elements:

Table 2

Table 2 is harder to create than Table 1, because we have to start figuring out how the elements can be combined. For example, zip_comb_1 = zip_elem_1 + zip_elem_3, but we cannot combine zip_elem_2 and zip_elem_3.

Here again, each element has a unique ID. Why do you need those IDs? Mostly because it’s easier to refer to a particular element when it’s properly indexed.

We have come up with 7 scenarios (zip_comb_1 to zip_comb_7 inclusively) that we can use to create test cases. In the test cases with scenarios from zip_comb_2 to zip_comb_7 inclusively, the expected result is the error message:

“Oops, error on page. ZIP code should have 5 digits“

In case of zip_comb_1, it’s the next page:

Remember how we purchase books at ShareLane:

            1. Register

            2. Log in

            3. Select book

            4. Add book to shopping cart

            5. Checkout

With Tables 1 and 2, we came up with scenarios for the first page (“ZIP code”) of step 1: “Register”. The second page (“First Name” and other fields) will have its own test Tables 1 and 2.

Each Web page for each stage of the book purchase process will have its own Table 1 and Table 2.

Most of the black box testing (for Web-based apps) involves an interaction with a sequence of Web pages that are integrated with each other and create certain combinations that lead us to expected results.

In some cases, we must have Table 3, Table 4, and so on reflect complex scenarios.

In some cases, we have to integrate the last table of one Web page with the first table of another Web page.

In some cases, Table 1 is enough to start creating test cases.

So, there is a huge amount of situations and options here. Be creative!!!

I know that once you start using test tables for more or less complex scenarios, you might feel lost. My only advice is: Just keep practicing.

When I took a Unix class some time ago, there was a long lecture dedicated to regular expressions (the art of finding patterns in text). After the lecture, my pal said that he felt like he had had brain surgery, and all of us completely shared his opinion. I’m serious: my brain was hurting. Don’t be surprised if you feel the same trying to combine complex scenarios using test tables. But again, don’t worry – the Test Tables technique takes time, but once you get it, you’ll never forget it.

I don’t want to overload you with work, but please do this exercise for practice: Create Table 1 and Table 2 for the flow “Login”. Below are suggested subsections for Table 1:

Username entered?

Password entered?

Username valid?

Password valid?

FLOWCHARTS

When we talked about the SDLC, I mentioned process flowcharts as a means to improve specs. A process flowchart is a graphic representation of a specific process. Flowcharts allow for different levels of abstraction. For example, we can represent the process of registration in this form:

This flowchart and its sister from the lecture about the SDLC are:

            – Similar, because they show the logic of registration

            – Different, because they have a different number of details – or, in other words, a different level of abstraction

When you create your flowcharts, you’ll use the level of abstraction required for your concrete task. For example, if we are testing registration, we’ll need a more detailed flowchart of the registration process compared to a situation where we want to include a registration flowchart as a part of some other process – e.g., the process of making a book purchase.

Here are basic building blocks (or “shapes”) for flowcharts.

Below are several recommendations about flowcharting:

1. Before you start drawing a flowchart, give a name to the main process that is going to be described in the flowchart – e.g., “Registration”.

2. Draw the main process first, and then expand it with details. In other words, start from the high level.

3. Put short yet informative text into the flowchart blocks.

4. Give references for useful info. For example, we can put the spec name to the right of starting block.

5. Make the process flow from top to bottom and from left to right.

6. Avoid the crossing of arrows.

7. Always double-check your flowchart to make sure that it correctly reflects the process. Use the spec or another source of knowledge.

8. Flowcharting can get really fancy: there are tons of different shapes that can be used. I personally use 3 basic shapes and have no problems with this. If you learn 15 shapes, it won’t hurt as long as you accurately chart a process.

Let’s do a quick training. Draw a flowchart for making tea. This task seems easy and fast to do at first glance, but check it out and you’ll be surprised how many blocks you’ll have for such a trivial process. Below are some ideas:

            Is there water in the teakettle?

            Is the stove on or off?

            Is the water in the teakettle boiling?

Use Microsoft Visio (you can download and install an evaluation copy) or just an old good sheet of paper.

Flowcharts are amazing visual resources for generating testing ideas. Besides, as with other testing methods, the process of the creation of the flowchart: 

– Allows the tester to look at software from different angles

– Brings up a number of questions that didn’t come up during plain spec reading

RISK ANALYSIS

Imagine that we’ve just bought a hotel in California somewhere in mountains of Sierra Nevada. We’ve got no experience managing hotels, but we are pretty confident that we can handle it. Not a big deal, right?

There are 3 roads that lead to the hotel:

            Road 1 connects the hotel and the freeway.

            Road 2 connects the hotel and the ski resort.

            Road 3 connects the hotel and the local road.

Roads 1, 2, and 3 have the same length, width, and altitude.

Ten guests have already arrived, and 30 are going to arrive today. It has been snowing like crazy all night, and Roads 1, 2, and 3 are not usable. There is only one snow cleaning truck at our disposal, and it takes half a day to clean one road. The question is: What road shall we start with?

At first glance it doesn’t seem like a big puzzle to solve:

“It’s absolutely obvious that Road 3 should not be cleaned first, because it’s only used by our poker buddies. Screw them.”

“It’s absolutely obvious that Road 2 should not be cleaned first, because 10 (peeps who would want to go skiing) is less than 30 (peeps who will take Road 1 to reach the hotel).

“Therefore, it’s absolutely obvious that we should start cleaning up Road 1.”

Does it sound logical? If “yes”, note our confidence about how absolutely obvious this stuff is.

Here is my suggestion. Let’s put the implementation of our genius plan on hold for a minute and ask James H. (a manager who has worked at the hotel for 20 years) for his opinion:

Situation Number One

Question: “James, what road shall we clean first?”

Answer: “You know, boss, it’s very simple. Those 10 who have already arrived came here to throw snowballs, drink beer, and sleep. I know this, because those crazy bastards come here every year, and it’s hard to imagine people who care less about skiing. So there is no need to clean up Road 2.

“I also know that 16 of the 30 arriving today are from a company that will depart this morning from Reno (I talked to one of them yesterday). They are going to take the road [James shows the road on the map] that merges with the local road connected to Road 3. So they’ll take Road 3.

“Now, look at the reservations. Twelve of the 14 remaining guests live in San Francisco. I’ve just heard on the radio that the highway I-80 leading from San Francisco to us has terrible traffic. I’m going to assume that those 12 will leave after work, departing San Francisco at 5:00 p.m., and merge onto Road 1 no earlier than 9:00 pm.

“Therefore, I suggest that we clean Road 3 first and, after that, clean Road 1.

“BTW, 2 guys are coming from Arizona. I’ll call them right away, explain the situation, and decide on the best route for them.”

Situation Number Two

Question: “James, what road shall we clean first?”

Answer: “You know, boss, first we should start cleaning Road 2. All of our 10 guests are skiers. Besides, the 30 remaining people will most likely go to the ski resort first and come here only in the evening – I ordered 30 skiing passes starting with today, so they won’t want to lose the whole day.”

Situation Number Three

Question: “James, what road shall we clean first?”

Answer: “No problem, boss. We’ll clean both Road 1 and Road 2. They have the same importance for our guests. I’ll call my friend Steve who has his own snow-cleaning truck and lives next to Road 2, and he can help us out. His is a nice person, and he won’t charge us much. He’ll take care of Road 2, and we’ll take care of Road 1. That way, we’ll clean up both roads before noon.”

The moral of the story is: Conclusions based on subjective assumptions might lead us into situations where resources are wasted. At the same time, conclusions based on credible information will most likely lead us to effective solutions even if we don’t have many resources.

Technically speaking, our wise manager James did risk analysis. His risk analysis was accurate because he:

            – Had correct information

            – Knew how to use that information

Risk analysis is the evaluation of data or expectations with the purpose of setting priorities.

Here is the tester’s approach towards risk analysis:

            1. Get information about the subject of testing (e.g., the feature “Shopping cart”).

The tester can get information from specs, statistics, live data about the Web site usage, communication, etc.

            2. If possible, get an opinion from the person who knows the subject better than the tester does (e.g., the PM).

That person can be the PM, programmer, business analyst, somebody in accounting and finance, etc.

            3. Do risk analysis (e.g., evaluate different patterns of usage in the shopping cart and determine the best ways to test it*).

* Yep, ShareLane is a simple application, and there is not much to do here with Risk Analysis – just grasp the concept.

Here is another example. Let’s assume that we have to test the following functionalities (forget about ShareLane for a minute):

            a. Sales between users in America

            b. Sales between users in Japan

            c. Sales between users in America and Japan

Let’s assume that we are selling repair parts for cars, and our Web site connects buyers and sellers.

Let’s use Test Tables. First, let’s create subgroups of the elements:

Table 1

Second, let’s combine those elements into scenarios.

Table 2

Very nice! We have 4 scenarios. Let’s present them in the form of a flowchart:

So, we have 4 flows. Let’s play our favorite game, “absolutely obvious”:

            1. The most popular transaction must be Japanese Seller -> American Buyer, because the American market has a huge number of Japanese cars.

            2. Less popular is the flow American Seller -> American Buyer: Yes, there is a substantial demand for repair parts for American cars, but there are many other venues to get those parts besides our Web site.

            3. Next we have the flow American Seller -> Japanese Buyer. We put it in 3rd place because it’s less popular than items 1 and 2, but more popular than item 4.

            4. The least popular transaction is Japanese Seller -> Japanese Buyer. Why? Because it’s absolutely obvious that the Japanese have their own internal infrastructure of car parts sales, and nobody uses our site there.

It all seems pretty logical until we start digging.

Question: What is the source of our information – e.g., how do we know that our Web site is popular in the 1st scenario (Japanese Seller -> American Buyer)?

Answer: We’ve heard it somewhere or read it somewhere, or just have a gut feeling that it is the case.

Let me ask you this: What if whatever we’ve heard or read was outdated, used in a certain context, or just plain wrong? And how credible is that “source of data” called a gut feeling?

Answer: …

Another question: What does this really mean: “The American market has a huge number of Japanese cars?” How huge? What is the number or percentage of Japanese cars on American roads?

Answer: …

Our conclusions, based on the “absolutely obvious”, have been a house of cards…again…

Here’s the idea. Let’s go to the PM, Amy, show her our flowchart, and ask for her opinion. Oops, it appears that Amy doesn’t have live data about our Web site usage. But she tells us that the business analyst, Brian, must have the information we need. We take Amy with us and go see our man Brian.

“Hello, Brian. Can you help us? Here is a flowchart with 4 possible types of transactions based on the country of the seller and buyer. Can you give us an idea about the actual popularity of each of those types?”

Brian pulls up live data for the past two years, and off we go:

            1. Most of the transactions took place between Japanese customers (Japanese Seller -> Japanese Buyer). Amy adds that, according to last marketing meeting, the Japanese market was declared to be the highest priority for the further growth of our company.

            2. There are slightly less transactions for the American Seller -> Japanese Buyer segment. Amy adds that recently we closed two very important contracts with Ford and General Motors that makes us their single venue for Internet sales of repair parts for foreign customers.

            3. There are much fewer numbers for American Seller -> American Buyer transactions. Amy comments that our competitors dominate here.

            4. The least popular are the Japanese Seller -> American Buyer transactions. Amy says that Japanese car companies prefer to distribute repair parts directly to car dealers in America. That’s why American buyers shop at cars dealerships instead of our Web site.

After that, we ask Brian to give us the concrete percentage for each transaction type (among the 4 types) based on the number of actual transaction that took place for the past 2 years. If we have that percentage, we can prioritize each type.

Now, instead of conclusions based on unreliable assumptions, we have actual data and an expert opinion based on reality and experience!

Don’t be shy about asking for help regarding live data, opinions, etc. It’s normal, and people usually love to help you.

EQUIVALENT CLASSES

An equivalent class is a set of inputs that are treated the same way by the software under certain conditions. In other words, under certain conditions, the software must apply the same logic to each element of the equivalent class.

The ShareLane PM, Linda, wrote a spec about the feature that gives discounts to those customers who buy many books. Here is the table from her spec:

We can immediately detect two spec bugs:

1. It’s not clear what discount rate should be applied if the number of books is exactly 50, 100, 500, 1000, 5000, or 10000. For example, value “50” belongs to two baskets: one basket has a discount rate of 2%, and the other is 3%.

2. This table is not complete, because it’s not specified what happens if we have less than 20 books. Logically, it must be 0 (zero), but it makes sense to have that basket in the spec anyway, just to make sure that we have a complete list of inputs.

We discovered those bugs during our spec review and sent an email to Linda requesting that she fix them. Here is how the table looks after the fixes:

In other words, we have 8 equivalent classes:

Each value inside a given class (e.g., Class 2) must be equally treated by ShareLane software. For example, if one user buys 21 books and another user buys 43 books, they must each get a 2% discount because those values belong to the same class (Class 2).

Each class has two parts:

            1. Range of inputs – e.g., 20-49 inclusively for Class 2

            2. Logic for output – e.g., Class 2 inputs are treated with a 2% discount

So what’s so cool about equivalent classes? When we divide inputs into equivalent classes, we can:

– Clearly see the connection between the ranges of inputs and the logic of the code

– Apply the technique called Boundary Values to select a small golden number of inputs out of the enormous number of possible inputs, and use that small number to hit the software in places where it’s likely to break

BOUNDARY VALUES

Look at the picture below.

            The dashed vertical line is the first value of the class (lower boundary).

            The solid vertical line is the last value of the class (upper boundary).

For each equivalent class, we have only 1 out of 3 possibilities:

            1. There is only a lower boundary (Class 8).

            2. There are both lower and upper boundaries (Classes 1-7 inclusively).

            3. There is only an upper boundary (we would have that class if we had a class unrestricted by a lower boundary – e.g., “any value below 1”).

The Boundary Values technique relies on the idea that software tends to break at the upper and lower boundaries of equivalent classes.

Here is the full version of Boundary Values technique for one equivalent class.

            A. First, we test the lower boundary of the class (if the lower boundary exists).

            B. Second, we test the upper boundary of the class (if the upper boundary exists).

            C. Third, we test any value inside the class (if the class consists of 3 or more values).

            D. Fourth, we test the upper boundary of the class that precedes the tested class (if the preceding class exists).

            E. Finally, we test the lower boundary of the class that follows the tested class (if the following class exists)

Here’s the same thing in the form of a table:

A-to-E table

A, B, and C are positive tests. D and E are negative tests.

Let’s do boundary testing and see each of the 5 elements (A-E) in action!

ShareLane ->

1. Go to Test Portal>Application>Source Code>shopping_cart.py. You can find the list of bugs below the “View bugs” subsection.

2. Open another Web browser window, create an account on ShareLane, login, add a book to the shopping cart, and click the “Shopping Cart” link.

3. Start learning the material from the table below (remember to click button “Update” after you enter new value into the text box “quantity”)

Below is a set of inputs for performing boundary testing on all equivalent classes for discount functionality:

We have 23 test cases to cover 7 closed (each of Classes 1-7 inclusively has both lower and upper boundary) and 1 open (Class 8 has no upper limit) equivalent classes. There are two very cool facts here:

1. We covered an unlimited number of inputs with 23 test cases.

2. We used a methodology that hits the software right at its breaking points!

Now, what if we wanted to test Class 2 in isolation from its sisters? We would need 5 test cases:

We’ve used the most advanced examples to learn Boundary Values technique. Now, let’s see how simple this can really be.

Lecture 7 - Test Preps -> Quick Intro -> The Tester's Mental Attitude -> Intro To Special Skills In Bug Finding -> Black Box Testing Techniques -> When You Start To Implement Testing Techniques...