Lecture 8 - Bug Tracking -> Quick Intro -> The Bug Tracking System -> Bug Tracking Procedure
Let’s forget about software for a minute. Let’s say that you are test-driving a car. You check out how fast the car gains speed, how it handles turns, if controls like the audio volume are reachable, etc. After the test drive is finished, you write down on a notepad whatever didn’t meet your expectations.
That notepad is the simplest version of the BTS.
Here is a definition:
The BTS is the infrastructure that enables to
– view and
information about bugs.
As a rule, software companies use professional BTS software. Some BTS software is free, like the most popular BTS, Bugzilla; some can cost tens of thousands of dollars (for example, Test Director by HP). I have used many of them, and although I personally prefer Test Director, I must say that Bugzilla is an ideal bug tracking solution for most software start-ups.
As we know, the fact that a bug has been discovered has no practical value until we share information about that bug. How can we share the information? We can make a phone call, send an instant message or email, talk personally, etc. The standard way to report bugs in software companies is to file a bug into the BTS. So, another important function of the BTS is the communication medium between SDLC participants. Of course, bugs are communicated in other ways, too (e.g., we can talk about bugs at the Go/No-Go meeting), but the rule remains: Each and every bug found must be filed into the BTS.
Question: How can we get information about the bug itself and the activities surrounding it?
Answer: We must check the bug Attributes in the BTS.
ShareLane -> Now, do this quick exercise:
1. Create an account on ShareLane and login.
2. Go to Test Portal>Bug Tracking>Training BTS>Submit new bug.
3. File a bug about anything: e.g., “Today the weather is bad” (fill up only Summary).
4. View that bug and change its status to Closed.
5. View that bug and change its status to Re-open.
Bug Attributes are needed to:
1. Describe the problem with the software/spec
2. Reflect activities made towards solving the problem
3. Provide other useful info about the bug
Let’s learn about those Attributes! I urge you to actively use Test Portal to file/modify your own bugs during this lecture.
Here is the list:
The bug ID is the unique identifier of each bug in the BTS. The BTS automatically assigns next available ID after a new bug is filed. The bug ID is not changeable. Among other usages, the bug ID is the universal way to refer to a concrete bug: “Hey Billy, how is the situation with 998?”
The Summary is a short, informative message about the gist of the bug. As a rule, the text field for the Summary doesn’t exceed 80 characters with spaces. The way a tester writes a Summary is a very good indication of his professionalism.
Example of a good Summary:
“Spec2345: 2% discount is given for 12-19 books inclusively”
Example of a bad Summary:
“problem with discount”
The purpose of the Summary is to give the BTS user a quick, powerful description of the bug. A bug Summary is to a bug as a title is to an article. Professional journalists are specifically trained in the art of creating good titles. I recommend that you start practicing right now. It’s not a big deal, really. Just start writing short Summaries of all the bugs you see in real life;
“I wanted a beer, but my wife bought me an ice cream.”
“I wanted an ice cream, but my husband bought me a beer.”
(BTW, bad Summary for 2 cases above would be, “I didn’t get what I wanted.”)
Here are some guidelines about Summaries:
1. Depending on the situation, start a Summary with one of the following:
– The spec ID; e.g., Spec2345
– The name of the feature (or component); e.g., Registration
– The name of the file with bug; e.g., shopping_cart.py
There are at least two reasons to do this:
a. The ability to unite related bugs by the same first word(s) in summaries. Bugs can be related because they were found while testing the same spec (e.g., Spec #1111 “New User Registration”) or the same functional area (e.g., “Checkout”), etc.
– On the one hand, the benefit here is that the BTS user can sort a list of bugs by the first word in the Summary and thus see all related records in one group.
– On the other hand, if certain keyword (e.g., spec ID) is presented in the Summary, then the BTS functionality “Search” can be used to find and list in one group all the bugs with that keyword in the Summary.
b. The ability to provide instant info regarding a buggy area. For example, if I put “shopping_cart.py:” as at the beginning of the Summary, then Billy, who is responsible for the shopping cart, will immediately know where to start looking for the problem.
2. While creating a Summary, think about your target audience – i.e., PMs, programmers, and testers. For example, if a certain component has an internal nickname like “cc” for “credit cards” you shall use it only if you are 100% sure that people from your target audience will instantly understand it. Always remember that bug filing is the process of sharing of information, so be clear. Concerning the level of abstraction, I prefer to pack as many details as possible into the Summary.
3. Sometimes it’s really hard to create a good Summary. In this case, end the Summary with the phrase “(see description)” or “(see attachment)” – whichever is applicable. We’ll talk about bug descriptions and attachments in a minute. An example of a Summary might be: “Homepage: wrong layout of the logo (see attachment).”
The Description is a text area where the full details about the bug must be given.
ShareLane -> I suggest the following format for bug Descriptions (check out the Description of Bug #1 in the Bug Vault):
Description: <provide information about what happens>
Steps to reproduce: <provide steps to reproduce the bug>
Bug: <state what’s wrong>
Expected: <state what’s expected>
Always remember that nothing is set in stone. In some cases, the bug Description can be just simple repetition of the bug Summary, or just the phrase, “see attachment” (when the attachment is self-explanatory); in other cases, you’ll really need to make an effort and provide great detail.
An important thing to remember about steps to reproduce: some things, like professional slang or abbreviations adopted in the QA Department and widely used in test cases, may be absolutely incomprehensible to the PMs and programmers; always remember the purpose of the Description is to share your knowledge about the bug in a way that is understandable to others. Yes, sometimes it can be boring, and good Descriptions do take time and effort, but bug reporting is the main way to address bugs, so each tester must make sure to provide crystal clear info about bugs.
Below is a table describing the most common Web page elements. This table will:
– Help you provide better Summaries, Descriptions, and test case steps
– Deepen your technical knowledge
– Give you an idea of the typical problems with each element
ShareLane -> You can play with these elements here: Test Portal>More Stuff>Basic Web page elements.
Name of the element
Example of the element
Example of HTML code of the element
Description of the element
Text is the most common element of the Web page. Problems with text include:
1. Wrong text (e.g., wrong text of the error message)*
2. No text (e.g., the code doesn’t display the title of the book)*
3. Wrong text property: font, color, size*
Of course, the problems of “wrong content”, “absence”, or “wrong property” can happen with other elements too, e.g.:
1. Wrong image of the book
2. No image at all
3. Image with wrong size parameters.
<a href=’./register.py’>Sign up</a>
Link (or hyperlink) is a navigation element. Link can point to
– Position on the Web page. Read on for examples.
Here are typical targets of links:
1. File, e.g., HTML file (Web page), PDF file, TXT file, Python file, etc.
2. Anchor on the same Web page
3. Anchor on a different Web page
4. “mailto” value
So, when a user clicks a link:
1. A certain file is opened (e.g., HTML, PDF, TXT files) or launched (e.g., the Python file shopping_cart.py dynamically generates the Web page “Shopping Cart”).
2. The Web browser scrolls down to a certain anchor on the same Web page.
3. The Web browser opens a new Web page and scrolls down to a certain anchor.
4. The Web browser invokes a new message instance of the default email client.
1. Click the link Link to another page on Test Portal>More Stuff>Basic Web page elements.
2. Click the link Bug #3 under the section “View bugs” on the Test Portal>Application>Source Code>shopping_cart.py.
3. Click the link Link to another page with anchor on this page: Test Portal>More Stuff>Basic Web page elements.
4. Click the link Mailto link on the Test Portal>More Stuff>Basic Web page elements.
Below are some typical problems with links:
A. Misleading link: The link points to the wrong file and/or anchor, or mailto value.
B. Broken link: This term is usually used for problems with linked files. Here we have 2 variants:
– Web server responded that linked file doesn’t exist at specified path. In that case we get the “404 – File Not Found” error.
– DNS cannot find IP address associated with that domain name. In this case we get “DNS error – Cannot Find Server”.
BTW, the DNS (Domain Name System) is the translator from a domain name to the IP address of a physical computer on the Internet.
C. Misspelled mailto value: e.g., ‘mailto:contactsharelane.com’ instead of ‘mailto:email@example.com’ (note missed “@” sign).
See examples on Test Portal>More Stuff>Basic Web page elements inside the table, “Bad Links”.
Check out the Downloads section of qatutor.com for automation tools that check for broken links.
BTW, linking the URL to a file can be presented in two ways:
– Relative URL: i.e., we assign a path to the file related to the directory where we are right now.
For example, if we are on the page https://main.sharelane.com/cgi-bin/main.py it means that we are inside the Web server directory of sharelane.com called ‘cgi-bin’. The directory cgi-bin contains the file register.py, so we can put a link like that to register.py: <a href=’./register.py’>Sign up</a>on the HTML page generated by https://main.sharelane.com/cgi-bin/main.py.
– Absolute URL: e.g., we give the full path.
For example, if we are on the page http://www.google.com and want to reach register.py on sharelane.com, we should do this: <a href=’ https://main.sharelane.com/cgi-bin/register.py ‘>Sign up</a>
Images are graphic files (as a rule, in a GIF, JPG or PNG format) that are referenced from the HTML code of Web pages. When a Web browser loads HTML code and “sees” a special tag: <img>, it tries to download the referred image to the hard disk of the user. If the file exists, the user can see the picture. If the image is corrupted or the path to the image is wrong, then we have a situation called “broken image” and we can see only an image placeholder on the Web page that looks
like this: (IE) or like this: (Firefox).
<img src=’../images/product_8_small.jpg’ height=’150′ border=0>
This is a link represented by an image instead of text. The linked image can have maladies of both images and links.
In the example of HTML code above, bolded is the code for a link and not bolded is the code for an image. If we put text The Moon and Sixpence instead of <img src=’../images/product_8_small.jpg’ height=’150′ border=0> then we would get a standard textual link.
<input type=’text’ name=’keyword’ size=’50’>
The text box is a one-line field for text input. The text box (as well as other elements that we’ll discuss further) serves as an input mechanism for a Web form. A Web form provides a way to submit one or a set of key-value pairs to a concrete file within the application core. Here is an example of HTML code for a Web form with a text box and a submit button (we’ll learn about this button in a minute). The opening and closing tags for the form are bolded. Underscored is the code for the button.
<input type=’text’ name=’keyword’ size=’50’>
<input type=’submit’ value=’Search’>
Here is what happens: after a user types expectations into the text box keyword and presses the “Search” button, key-value pair “keyword=expectations” is sent via the Web server to be processed by the application core file search.py.
– If you submit that form by using the Search functionality on main.sharelane.com or
– If you just type in this URL: https://main.sharelane.com/cgi-bin/search.py?keyword=expectations
You’ll get the same Web page as a result.
Let’s get back to the text boxes. Besides standard things like the ZIP code or email fields, text boxes are also used to enter the value for captcha. Captcha is an image with some alpha, numeric, or alphanumeric characters on it. Captchas are used to prevent bots (automated scripts) from submitting Web forms: the premise is that humans can distinguish text on an image, while bots cannot (some of them already can, BTW). Here is an example of a captcha from the Yahoo site. This captcha is displayed to prevent automatic registration of email accounts:
Regarding bugs connected with text boxes, we usually have
1. The wrong value for the text box attribute name. For example, if a programmer types name=’keywor’(note: no “d” letter) instead of name=’keyword’ then no search would take place, because search.py expects the variable called “keyword”. Go to this URL to check it out: https://main.sharelane.com/cgi-bin/search.py?keywor=expectations
2. The wrong value for the attribute maxlength or no such attribute at all. Maxlength is needed to limit the number of characters (including spaces) that can be typed into the text box. For example, Willy wanted to type this to limit search keywords to 25 characters:
<input type=’text’ name=’keyword’ maxlength=25>
but erroneously he typed:
<input type=’text’ name=’keyword’ maxlength=5>
As a result, a user can enter only a keyword with equal or less than 5 characters:
The first problem is discovered during functional testing; the second problem is discovered during UI testing.
BTW, very often a bug occurs because the DB column is able to store fewer characters than the user is able to submit. The grey box approach can help us find that bug. Do this:
1. Go to Test Portal>DB>Schema
2. Scroll down to table below this line: “mysql>describe users;”
3. Note the value of Type for Field first_name. It’s varchar(50).
Varchar(50) means that the column first_name can store up to 50 characters, including spaces. So, if user enters 51 characters or more as his/her First name, 51st and following characters will be truncated (i.e., cut off), because the first_name column has a limit of 50 characters (for security purposes this exercise cannot be done on ShareLane).
The easiest fix for this problem is to introduce a maxlength parameter to limit the number of characters that a user can type into the text box.
<textarea columns=10 rows=5 name=’message_body’>
I’m some text inside test area. I can take 1 or more lines
The text area is a multiline field for text input. An example of usage is the body of a message on an Internet forum. Usual text area problems include the wrong value for an attribute name or the wrong values for the columns and rows attributes.
Password input box
<input type=’password’ name=’password’>
The password field is a special one-line text box where the input is masked by asterisks or dots as the user types text. Character masking is required for security to prevent some friendly fellow from seeing your password and cleaning out your bank account.
The usual problems with password fields are just like those with text boxes. In some cases, the programmer makes a mistake and uses a text box instead of the password field to submit a password. See Bug #1 in the Bug Vault.
Below are two additional things that I want to mention here:
Professional ethics: When somebody types a password in your presence you should explicitly turn your head (or your whole body) away from the person. It’s extremely unprofessional to stare at the monitor/keyboard while somebody types a password. Your colleagues will surely appreciate the fact that you respect their privacy.
Your privacy: Many software companies install software to capture keystrokes and screenshots on their employees’ computers. This means that whatever you type or see on your monitor can be viewed by your employer. Just FYI…
A drop-down menu is a list of values to choose from. BTW, there are 2 bugs in the code above! Here is what happened: Each credit card has an internal index at Sharelane:
1 for Visa
2 for MasterCard
3 for AmEx
(See Test Portal>DB>Data>cc_types)
The developer made 2 mistakes. He put 1 for both MasterCard and AmEx. When we click “Make payment” button to complete the checkout, the content of the Web form is sent to checkout.py for processing. Part of the content is the key-value pair card_type_id=<internal card index, e.g., 1 for Visa>. Here is how these 2 bugs echo in other parts of the system:
– The first digit of the value in the column result of table cc_transactions will always be equal 1 (Test Portal>DB>Data>cc_transactions) and
– The second value on each line in the log file cc_transaction_log will always be equal to “V” (Test Portal>Logs>cc_transaction_log)
In this case, I would file 2 bugs with summaries like these:
“checkout.py: value 1 is given for MasterCard in drop down menu card_type_id”
“checkout.py: value 1 is given for Amex in drop down menu card_type_id”
Why don’t I recommend doing something like “checkout.py: value 1 is given for all cards in drop down menu card_type_id” in our case? Because the MasterCard bug is not related to the AmEx bug – if you fix the problem with MasterCard, the bug with AmEx will not be fixed. There is more to this situation, and we’ll talk about it in several minutes.
I recommend that you to go to the Training BTS right now (Test Portal>Bug Tracking>Training BTS) and file those 2 bugs yourself (fill in only the Summaries and Descriptions).
Back to the drop-down menus…. Now we know that the wrong value of the value attribute can give birth to a bug. In addition, we can also have problems with the wrong value of the name attribute, or with the completeness of the text parameters (e.g., if we accepted “Discover” then it should have been on the list, too) or their correctness (e.g., “Viva” instead of “Visa”).
<input type=”radio” name=”morning” value=”fishing”> Go fishing
<input type=”radio” name=”morning” value=”kids” checked> Play with kids
<input type=”radio” name=”morning” value=”dishwasher”> Repair dishwasher
The radio button is an element of a Web form that allows the selection of one, and only one, value from the group with the same name (within the same Web form). The group must have 2 or more values. In the code above, the group’s name is “morning”. In other words, logically the elements of the same group are mutually exclusive; e.g., in the above example you can select EITHER Go fishing OR Play with kids OR Repair dishwasher.
Typical problems with radio buttons include:
– Incorrect value for name
– Parameter checked specified for a wrong value
– Incorrect label (“Go hiking” instead of “Go fishing”)
The radio button took its name from those old huge radio receivers with the big buttons: each button was associated with one frequency, so you could only listen to one radio station at any given moment.
It’s bad programming style to use a radio button if a group consists of only 1 element. In that case, if you select the button, you cannot clear it.
<input type=”checkbox” name=”option1″ value=”bread” checked>Bread
<input type=”checkbox” name=”option2″ value=”lettuce”>Lettuce
<input type=”checkbox” name=”option3″ value=”cheese” checked>Cheese
Checkboxes have the same maladies as radio buttons.
<input type=’submit’ value=’Search’>
The Submit button is needed to submit Web form content to the Web server. Usually it’s referred to as just a “button.” For example, when you press the “Search” button on the ShareLane home page, you send whatever was typed into the text box on the left of the Search button. A common problem with the Submit button is entering the wrong value of the attribute value.
<input type=’reset’ value=’Reset’>
The Reset button returns the values inside the Web form back to their defaults.
Here is a UI once a particular Web page is opened:
As you can see, the option “Play with kids” is checked by default. If we select the button “Go fishing” and then press the “Reset” button, the selection will automatically shift back to the “Play with kids” option.
Here is another Web page:
As you see, the default setting here is that none of the radio buttons are selected. If we select “Play with kids” and press the Reset button, then all the radio buttons will automatically become unselected (another term is cleared); i.e., they’ll return to their default state.
I hope that this little intro to Web elements was useful! Now, let’s get back to the BTS attributes.
In some cases, it may take a long time to write a good textual Description of the bug (especially when the tester has to give details about what happens on the UI). It all depends on concrete situation, but as a rule, one of the best solutions is to illustrate the bug with an attachment.
Do this exercise (instructions are for Windows OS):
1. Go to Test Portal>More Stuff>Basic Web page elements.
2. Press the keyboard button PrtScrn (or whatever the abbreviation is on your keyboard for “Print Screen”); this button is located on the upper right side of a standard keyboard.
3. Now, go to Start/Programs/Accessories/Paint.
4. Do Ctrl+v on your keyboard; i.e., press the “v” key while holding down Ctrl.
5. Save the file with the extension .jpg (.jpeg).
This is how you create an image that can become an attachment to the bugs in the BTS.
Please note that attachments are not restricted to images. You can also attach other types of files (e.g., text files with logs, etc.) . The idea is to provide the best possible illustration of the bug.
The BTS automatically fills in this attribute with an alias of the person who filed the bug. The bug can be filed by any person who has an account in the BTS and the bug filing privilege – i.e., the tester, developer, PM, etc. This attribute’s value is not editable.
The BTS automatically fills in the value of this attribute with the precise time the bug was filed. This attribute’s value is not editable.
Here we come to the very important concept of the bug owner. The bug owner is the concrete person responsible for the next step towards bug closure.
The “Assigned to” drop-down menu is needed to specify an alias of the bug owner. When filing a bug, the tester should always select from Assigned to:
A. The alias of the appropriate developer (e.g., “Billy”)
B. The tester’s own alias
A. How do you know who the “appropriate developer” is? In the majority of cases, it’s the developer who wrote the buggy code. But what if he has left the company (or the world), or moved to another team inside Development, so he is not responsible anymore for fixing that particular bug? In that case, we can “take our business” to
– Either the person currently responsible for that functional area (e.g., “Search”)
– Or to the person inside Development who would reassign the bug to an appropriate developer. That person is usually the dev manager.
The logical question is this: How do we get information about current responsibilities? You can ask questions and get answers, but it’s not the best solution here. The best solution is to create and maintain two very important documents: Who Does What and Who Owns What. Needless to say, these documents should be posted on the Wiki.
Who Does What relates to current projects (a separate Who Does What can be created for each major release.)
Who Owns What relates to areas of expertise.
Example: Who Does What
“Security Improvements to sign-up flow”
Example: Who Owns What
You can get templates from Downloads on QATutor.com.
It’s critical to keep these two documents up-to-date and fill them up with the correct info.
If a company consists of 3 persons, it’s more or less clear who does what and who owns what, but once a start-up begins to grow, there are more people, more projects, more communication (and more miscommunication), more things to do, more information to digest, etc. It becomes very important to have centralized sources of information about current projects and areas of expertise. It’s not only about the Assigned to attribute. It’s about keeping track of things and knowing where to go for help. I strongly recommend both of those documents.
B. In some cases (especially in a start-up mess), the tester has no idea who will be fixing the bug. But that “no idea” should not prevent the tester from filing the bug report! The tester just needs to select his name from the Assigned to list, file the bug and then find out who is “Da Man” or “Da Woman” responsible for fixing the bug. I know of a situation where on a daily basis the dev manager would simply go through the list of freshly filed bugs assigned to testers and change the Assigned to values to the names of the appropriate developers.
Below are several BTWs related to Assigned to.
The value of Assigned by is the alias of the person who assigned the bug to its current owner. As is the case with Submitted by, this is an automatically populated attribute. This attribute’s value is not editable.
The Verifier is a drop-down menu with the same list of aliases as Assigned to. The Verifier is the person who must regress the bug once it is fixed. By default, the value of the Verifier is the alias of the person who filed the bug (Submitted by).
Remember, a bug can be filed by anyone who has an account in the BTS and a bug filing privilege; e.g., by the PM. Of course, it’s not the responsibility of the PM to regress the bug. So, if the PM (or another non-tester) files a bug, then that person simply selects the tester’s alias from the Verifier menu and lives happily ever after without thinking about bug fix verification. BTW, that “happily ever after” can be disturbed, because sometimes a bug is returned to the bug filer – e.g., if the assigned developer cannot reproduce the bug. Read on!
As a rule, this is a drop-down menu with a list of the functional areas of the Web site. For example, at ShareLane we have this list of components:
– Shopping Cart
The benefit of having this attribute is that we can sort bugs by their Component values.
This is drop-down menu with a list of the environments where the bug was found. At ShareLane, we also added Spec to the list for spec bugs:
– Other environment
Of course, the worst thing is when a bug is found in production.
Here are the benefits of having this attribute:
– It’s clear where to reproduce the bug.
– Bugs can be sorted by Found on values.
This is the release version of the environment where the bug was found; e.g., if the application version is 1.0-23/34, we put 1.0.
This refers to the build number of the environment where the bug was found; e.g., if the application version is 1.0-23/34, we put 23.
The DB schema version of the environment where the bug was found: e.g., if the application version is 1.0-23/34, we put 34.
This is multiuse text area. This attribute has 2 main usages:
– Comments about the bug itself; e.g., give more info about the steps to reproduce.
– Comments about bug-related actions; e.g., when a developer reassigns a bug to another developer.
As a rule, Comments is an optional field, but in some cases it makes sense to make it required – e.g., when the developer cannot reproduce the bug and reassigns it back to whoever submitted the bug.
This is a drop-down menu with values S1-S4 inclusively. Severity is the magnitude of the bug’s impact on the software. Severity is a technical characteristic. In order to determine Severity, we must find a match between
– The Severity category
– The type of bug’s impact
The document we use for this is Bug Severity Definitions (see QATutor.com Downloads for the template).
Here is an example of the document (explanations will follow):
Type of impact
– data loss
– security issue
– site hangs
– blocking issue
– functional issue
Details and examples
This is situation when a Web site is completely nonfunctional because of a software problem.
DATA LOSS usually happens in the following cases:
1. Data is lost halfway to the DB
2. DB is populated with partial data
3. Data is deleted from the DB
Example 1: A user submits his data via the registration form register.py, but the module db_lib.py has a bug in the function update() and the data is not inserted into DB.
Example 2: A good example is the earlier situation when the first_name column in the DB table users couldn’t hold 51st+ character while the first name text field allowed the user to type in 51+characters. Of course, this was an extreme example; not many people have a first name with 51+ characters. In reality, data loss of this type usually happens when a programmer allows an unreasonably small capacity for the DB column; e.g., Varchar(5) – only 5 characters for the first_name.
Example 3: This may happen when the application core file mistakenly deletes/updates data. For example, in 2.0 we introduced the “Update user info” functionality. Willy wrote buggy code that would update all the records in the DB table users instead of updating only one record: precisely, the record of the user who is changing his info. So a bug resulted in this situation: if a user changes his last name to “Lee”, the last name of every ShareLane user becomes “Lee”.
BTW, bugs of this kind are another reason to set up daily DB backups in both the testing and production environments!
Please note that data loss might not be related only to the DB; e.g., because of the bug the content in the log file can be overwritten with new lines instead of being appended with new lines.
Example: Because the bug site that accepts credit cards stopped using a secure connection between the Web server and the Web browser, unencrypted credit card data can be discovered by special software and stolen.
This is usually a performance issue; e.g., it takes a long time to load the Web page.
The term “blocking issue” has a wide meaning. It refers to a bug that blocks:
3. Site usage
Example 1: Willy is supposed to write functionality A. Functionality A should use the Python classes from Billy’s code. But as it appears, those classes have a number of serious design bugs. Therefore, it just doesn’t make sense to start working on functionality A until Billy has fixed those bugs. So Willy is blocked from coding.
Example 2: Willy wrote functionality A. Billy wrote functionality B. B depends on A, but A doesn’t work. Thus, tester Rajeev cannot test functionality B. In other words, Rajeev’s testing is blocked.
Example 3: Users are blocked from buying books on ShareLane because Checkout doesn’t work.
A very important term that is associated with a blocking issue situation is workaround. For example, if the Account Creator (Test Portal>Helpers>Account Creator) used by testers for automated account generation doesn’t work, it’s not an issue that blocks testing because new user accounts can be created manually. So, in that situation, manual account creation is a workaround. If there is a reasonable workaround, then an issue cannot be considered as blocking.
This category is for all functional bugs that don’t fall under S1 or S2. It’s no wonder that majority of bugs will be S3. That’s why S3 is, as a rule, the default Severity when testers file a bug.
This includes all bugs that don’t fall under S1, S2, or S3. As a rule, here you’ll find UI bugs (e.g., layout problems, spelling, etc.).
This is a drop-down menu with values P1-P4 inclusively. Priority is the magnitude of a bug’s impact on the company’s business. There is often confusion between Priority and Severity, so let’s point out the differences:
Severity refers to the technical aspect of a bug.
Priority refers to the business aspect of the bug.
In other words, Severity uses technical criteria to grade the bug, while Priority uses business criteria. That’s why it’s almost always clear which severity to assign to the bug, while the Priority of a concrete bug is often the subject of arguments and political games.
The best tool that helps to reduce any controversy associated with assigning Priority is called the Bug Priority Definitions
Please look at the example of Bug Priority Definitions below (you can get the template from Downloads on QATutor.com).
Bug Priority Definitions:
Types of issues
1. Critical reliability/performance issue*
2. Critical legal issue*
3. Critical business issue*
4. Critical security issue*
5. Core functionality** is not working; no workaround
Site is dead
Legal suit is filed (or about to be filed) against the company
Integration with partner’s site is broken
Intruder’s back door is discovered
500 – Internal Server Error when user clicks link "Shopping Cart"
1. Major reliability/performance problem*
2. Major legal issue*
3. Major business issue*
4. Major security issue*
5. Core functionality is not working; workaround exists
Unreasonably long time to load a Web page, e.g. 10 seconds.
Owner has demanded to stop sales of the book.
Inaccuracies in banner rotation
Under certain condition, name and address of another user is displayed in Checkout page.
Cannot complete purchase if card expiration date is this year.
1. Unpleasant but bearable reliability/performance issue*
2. Legal issue of moderate importance*
3. Business issue of moderate importance*
4. Security issue of moderate importance*
5. Functionality doesn’t work (cases that don’t fall into P1, P2, P4 categories)
Intermittent delays loading a Web page.
Age verification during user registration was not implemented.
"Show Book" page is missing info about the publisher.
Log out button doesn’t work in some browsers
Cannot search for keywords that consist of 3 characters or less
Cosmetic issue* of low importance that doesn’t and won’t affect reliability/performance, or legal, business, security sides of company operation.
Low importance/probability functional problems.
Problems with layout
* Issue can be present or potential
** Core functionality is functionality that is essential for the book sale, e.g., Shopping Cart or Checkout.
There are 2 more very important documents that are closely associated with Bug Priority Definitions.
1. The Bug Resolution Times document defines concrete deadlines for the programmer to fix the bug. Deadlines depend on Priority.
ASAP. Drop whatever you’re doing.
Fix if possible
* P3s are to be fixed in the next minor or major release
** P4s are fixed if possible and thus are not suitable for patch releases.
2. The Go/No-Go Criteria document is a set of conditions under which the release goes to production. As a rule, P1s and P2s must be fixed before releasing code to production. So, in some cases P2 bugs are changed to P3s to meet the criteria (talking about political games!).
You can find both Bug Resolution Times and Go/No Go criteria under Downloads on QATutor.com
Always remember that Priority is a powerful tool that affects the developer’s schedule, so do your best to assign a Priority as close to the Bug Priority Definitions as possible.
To conclude our talk about Priority let me give you an important BTW.
This is usually a text box where you can put the aliases of people who, in your opinion, would be interested to get emails containing:
– Notification that the bug was filed
– Updates about changes that happened to the bug
While testing code that has been written according to a concrete spec, I usually put the PM’s alias into the Also Notify list. PMs appreciate this gesture.
This is an automatically populated field in case any events happen to the bug:
– The fact of bug filing
– Any change to the bug
The Change History usually includes:
– Date and exact time (to the second) of the bug filing/change
– Alias of the person who filed a bug/made a change
– Fact that the bug was filed, or what was changed and how it was changed
Therefore, whatever someone changes about the bug is automatically logged, and that log (called Change History) is available to everyone who has a privilege to view bugs in BTS.
This is drop-down menu with two values:
– Feature Request
We select “Bug” if it’s about a bug. We select “Feature Request” if we want to suggest and/or keep track of some kind of non-bug-related improvement. Please, note that EFR (Emergency Feature Request) is usually filed with type “Bug”.
This is drop-down menu with 3 values:
– Open is automatically assigned after a bug is filed.
– Closed must be selected by the tester when he closes the bug.
– Re-open must be selected once the bug is back. (Read on…)
When do we have a situation where we need to re-open a closed bug?
This is a drop-down menu with the following list:
>Fix in progress
>Fix is verified
>Not a bug
>3rd party bug
>No longer applicable
Resolution is one of the most important attributes. If Status is about global things like “was born,” “died,” and “got reincarnated,” Resolution provides details like “graduated from college”, “got married”, “bought a condo” etc. Resolution details the stages of a bug’s life.
Question: What drives a bug from one life stage to another?
Answer: Certain activities geared towards bug resolving.
Question: Where can we find information about those “certain activities” and the associations between them and the concrete values for resolution?
Answer: In the BTP.
Let’s learn about each Resolution. BTW, because the majority of bugs are found in the software (not in the specs or other documentation), we’ll be talking about situations where the person responsible for the bug fix is the programmer.
This Resolution must be chosen when the person who files a bug doesn’t know who will be fixing the bug. This is the situation when a tester assigns a bug to himself.
ShareLane -> Bug Vault>Bug #2
This Resolution means that the programmer in Assigned to should investigate the bug.
Example 1: The person who files a bug knows who is going to fix the bug, so that person selects name of the programmer from Assigned to and selects the Assigned for Resolution.
ShareLane -> Bug Vault>Bug #3 – please note that this bug was invented to illustrate the point. ShareLane app. doesn’t really have a bug like this.
Example 2: The person filed a bug with Reported status, but changed Reported to Assigned (and put programmer’s alias into Assigned to) once he found out who was going to fix the bug.
ShareLane -> Bug Vault>Bug #4 (see Change History)
Fix in progress
The programmer selects this Resolution as soon as he starts working on the bug fix.
ShareLane -> Bug Vault>Bug #5 (see Change History)
The programmer selects this resolution as soon as he has checked the bug fix into the CVS. Along with that, the programmer must select the same alias as in Verifier from the drop-down menu Assigned to.
ShareLane -> Bug Vault>Bug #6 (see Change History)
Fix is verified
Remember that bug regression consists of two parts:
1. Verification that the bug was really fixed
2. Checking if the bug fix hasn’t broken some other part of the software.
First of all, we just try to reproduce the bug following the instructions in the Description:
– If the bug is NOT reproducible, then it was really fixed.
– If bug IS reproducible, then it wasn’t fixed, so we send it back to the developer with the Resolution “Verification failed” – and we DO NOT execute Part 2 of the bug fix verification.
If the bug is NOT reproducible, we move to Part 2. This is where it gets interesting. In the case of more or less complex software, it’s sometimes extremely hard (even for a programmer) to confidently predict how a certain change to a certain piece of code will affect other parts of the software. So, the only way to do comprehensive regression testing is to perform 100% coverage of all possible scenarios (what’s usually impossible!).
The standard way to execute Part 2 is to perform a basic end-to-end test of the feature that contained the bug. In special cases (e.g., in case of EBF) you can:
a. Ask the developer who fixed the bug his opinion of what could have gotten messed up as a result of the bug fix and what he recommends that you check out in particular.
b. Next follow his instructions.
c. Then perform a basic end-to-end test of the feature that contained the bug.
Please note that you can:
– Change the Resolution from “Fixed” to “Fix is verified” AND
– Close the bug (change its status to Closed) as soon as Part 1 is finished and you have verified that the bug was really fixed.
Why bother about Part 2? We need Part 2 simply as a safety measure. Of course, in the case of Part 2 we’re not talking about serious testing; we just do a quick check that usually takes several minutes. What happens if we find a bug? Simple – we file new bug (or re-open closed bug).
ShareLane -> Bug Vault>Bug #7 (see Change History)
The verifier changes the Resolution to “Verification failed” and “Assigned to” to the alias of the responsible developer if the bug is reproducible; i.e., if Part 1 of the bug fix verification failed.
ShareLane -> Bug Vault>Bug #8 (see Change History)
This unpleasant situation occurs if the developer tries to reproduce the bug assigned to him, and cannot do it. The bug is usually not reproducible for 1 of 3 reasons:
1. The tester didn’t provide a comprehensive Description.
2. The tester’s environment and the developer’s environment are different.
3. There is no bug.
Let’s look at these reasons one by one.
1. The tester didn’t provide a comprehensive Description.
One of the main concepts about bugs is the idea that if a bug exists, then it is reproducible. NEVER file a bug until you reproduce it at least once after you discovered it. Read the following Brain Positioning, and remember it for the rest of your testing career!
Again, if a bug exists, then it’s reproducible. Each concrete bug is a resulting combination of a certain scenario, i.e.,
Usually people are more focused on providing info about actions and data rather than about conditions. Try to avoid that pitfall! In many cases, it’s the condition that makes all the difference. In some cases, a condition can be very unusual or hard to grasp, but if you pay attention you’ll find that condition. Please read the story below to better understand my point.
Let’s get back to our testing. In many cases, a bug is not reproducible not because the tester didn’t know the scenarios/conditions, but because he just didn’t write a comprehensive Description.
It really sucks when a developer must put off his coding, spend time trying to reproduce a bug, and is not able to do it because the tester forgot to include a little detail right in the middle of “Steps to reproduce”. You must avoid situations like that! Do your best to avoid “merchandize returns” that come with the Resolution “Cannot reproduce”!
2. The tester’s environment and the developer’s environments are different.
Again, it’s all about conditions. There are situations where some kind of underlying item is different in the dev environment (e.g., billy.sharelane.com) than in the test environment (e.g., main.sharelane.com). For example, the dev environment could have a more recent version of the Python interpreter, so Billy could use the Python libraries that exist in his environment but are missing on main.sharelane.com. The tester sees a bug and files it, only to get a “Cannot reproduce” Resolution, which leads to a waste of time going back and forth with Billy trying to understand what’s really going on!
3. There is no bug.
Here’s a potential situation: The DB on main.sharelane.com is down for maintenance, and the tester doesn’t know about it. He sees the bug, tries to reproduce it (with success), and then files it. The programmer tries to reproduce the bug when the DB outage is over, but the bug doesn’t exist anymore. So the programmer sends the bug back with the “Cannot reproduce” Resolution. BTW, a similar unpleasant situation might happen when the testing is done while a push of a new build is under way. ALWAYS check the build status page before starting testing!
In case of a “Cannot reproduce” Resolution, whoever selects that Resolution should also assign the bug back to the person who filed it; i.e., the bug owner must be a person from the “Submitted by” field.
What happens if a tester receives a bug back? He should either close it OR provide more details and assign it back to the developer with an “Assigned” Resolution.
ShareLane -> Bug Vault>Bug #9 (see Change History)
This Resolution is selected if another bug was already filed for the same issue. In software companies, the BTS may contain thousands of open and closed bugs, and sometimes it’s not physically possible to review each of them to avoid filing a duplicate. The best way to avoid duplicates is to keep track of:
– All new bugs OR
– New bugs that meet certain criteria. For example, bugs found in particular area, e.g., “Checkout”
A professional BTS usually allows you to modify the settings for your BTS account and sends you an email if any new bug is filed or a filed bug meets certain criteria. That way you’ll know what was filed and what wasn’t.
On the other hand, there can be not-so-obvious situations. For example, two or more filed bugs can be the result of the same root cause.
Black box testers don’t look into the code, and situations like this do take place. So when the developer returns your bug with a “Duplicate” Resolution, explaining (in Comments) that your bug was caused by the same code as another bug, don’t take it personally – that happens, and it’s okay.
Once a bug is marked as Duplicate, the person making that change must also put the original bug ID in Comments and assign the bug back to the person who submitted it.
ShareLane -> Bug Vault>Bug #10 (see Change History)
Not a bug
Start-ups, especially in the early stages, are usually a mess.
In some cases, it’s really hard to find out the correct expected result, and our old friends: common sense and life experience might be different from what the PM tried to communicate during his talk with the developer some time ago.
In some cases the PM communicates a product change to the developer and forgets to:
– update the spec and
– mention that product change to the tester.
In some cases, the tester is correct that it’s a bug, but the developer thinks that it’s a “Feature” and sends the bug back to the tester.
So be prepared to answer the question: “Why is that a bug?” If you were wrong about it, but you had reasons for believing that it WAS a bug, then there is no problem.
If you are not sure and it’s a good time to ask (“Hey Linda, is this a bug or not?”), then ask. If nobody is available, then file a bug report. IMHO, it’s better to file a bug (if you have reason to believe that it is a bug) and later find out that it’s not a bug, than be scared of the “Not a bug” Resolution and ignore the real problem.
In any case, when you receive your bug back with the “Not a bug” Resolution, read the Comments and try to understand what the developer tried to communicate. If you still think you are right, go talk to the developer, and then:
– If you both agree that it is a bug, send it back to the developer (Assigned to: developer’s alias; Resolution: “Assigned”) with your explanations in “Comments”.
– If you both agree that it’s not a bug, just close it.
In some cases when a spec sucks, the tester and developer cannot come to a consensus about what the heck the PM meant. In that case, reassign the bug to the PM and let him decide whether it’s a bug or not. In a situation like this, I’d also recommend filing a separate bug against the spec.
See an example of ping-pong with “Not a bug” here:
ShareLane -> Bug Vault>Bug #11 (see Change History)
3rd party bug
The code in a software start-up cannot rely upon only in-house software, e.g., code written by the start-up’s programmers. We use databases, compilers, interpreters, and Web servers that have been designed and developed by others. In many cases, we also integrate out software with the software of our partners, e.g., with credit card processors. So, here is the situation: you file a bug, the bug programmer returns this bug to you with a “3rd party bug” Resolution and the Comment that the bug is not in the software that he wrote, but instead in the software written by others. Here we might encounter two situations:
1. We CANNOT influence “others” to fix their software – e.g., if the bug is in the Python interpreter, we cannot call Python’s father, Guido van Rossum and ask him to fix the bug before our release goes out. So, the only way to fix the bug is to find some programming workaround. So, if a programmer returns your bug with a “3rd party bug” Resolution, it doesn’t solve the problem, because whether he likes it or not, help is not coming, and we have to find a solution ourselves. In a case like this, talk to the programmer and re-assign the bug back to him.
ShareLane -> Bug Vault>Bug #12 (see Change History) – please note that this bug was invented to illustrate the point. ShareLane app. doesn’t really have this bug.
2. We CAN influence “others” to fix their software.
ShareLane -> For example, in checkout.py we have the function get_ccp_result() (see source code here: Test Portal>Application>Source code>checkout.py). This function connects to the payment processor to process credit card payments (of course, in case of ShareLane there is no real processor – it’s all training software). That payment processor software belongs to another company, which is our service provider (also called “vendor”). If there is a bug in the vendor software, then we can call them and ask them to fix the problem ASAP. As a rule, the tester usually doesn’t make this contact. As a rule, a contact like this is the responsibility of the project manager (PjM). BTW, in start-ups PMs are often also PjMs. So, if a developer returns a bug to you with a “3rd party bug” Resolution in this case, you should reassign the bug to the PjM and that person would be the bug owner who is responsible for resolving the situation.
ShareLane -> Bug Vault>Bug #13 (see Change History)
No longer applicable
This is usually selected for bugs found in features that have been deprecated; i.e., features that are still available to users but no longer officially supported.
Lecture 8 - Bug Tracking -> Quick Intro -> The Bug Tracking System -> Bug Tracking Procedure