Lecture 3 - Test Cases and Test Suites -> Quick Intro -> Test Case Structure -> Results Of The Test Case Execution -> Useful Attributes Of The Test Case -> Data-Driven Test Cases -> Maintainability Of Test Cases -> The Number Of Expected Results Inside One Test Case -> Bad Test Case PracticesOur test case verifies only one concrete thing: “Payment can be made by Visa” and in an ideal situation there is only one ER. If I was a test theoretician, I’d tell you to never include more than one ER in a test case, but I’m no theoretician and I promised you to teach you real stuff. Here is a live example for you:
Example
Let’s assume that according to item 12.b. of the document “Code Design for the Spec #6522”, we can say that a payment with Visa is successful if-and only if-two conditions are met:
1. In the DB, the column result of the table cc_transactions has a value of “10” for the record with our Visa transaction.
2. The credit card balance is reduced by the amount equal to the amount of the payment.
So, in order to verify only one concrete thing, we have to check to see if the reality meets two expected results.
We have two ways:
1. Split the test case IDEA into two IDEAs and create two test cases OR
2. Don’t change the test case IDEA and have two ERs in one test case. The test case would pass if, and only if, BOTH ARs match the corresponding ERs. In all other cases, the test case would fail.
Here is an example of the second way:
IDEA: Payment can be made by Visa | TC ID | ССPG0001 | |
Priority | 1 | ||
SETUP and ADDITIONAL INFO Go to Test Portal>More Stuff>QA KnowledgeBase to “See QA KB“ In order to run SQL1 go to Test Portal>Helpers>DB Connect Utility. SQL1: select result from cc_transactions where order_id = <order id>; How to get credit card balance: Test Portal>Helpers>Credit Card Balance Viewer | |||
Created (date/name):11/17/2004/O.Ferguson | Reason: new way to verify that credit card transaction was successful. | ||
Modified (date/name): 02/03/2005/I.Newsome | Reason: modified steps to improve test case maintainability | ||
Modified (date/name): 02/15/2005/I.Newsome | Reason: modified steps and added second expected result to verify that card balance was reduced. | ||
1. Generate Credit Card (See QA KB) and copy card number into the temporary text file. 2. Get credit card balance and write it down: ____ 3. Create New Account (See QA KB) 4. Login 5. Find Search Keyword (See QA KB) 6. Do search 7. Add found book to the Shopping cart. 8. Copy card number from the temporary text file and Do Checkout (See QA KB) !!! While doing it, write down amount of payment: ____ 9. Write down order id: ____ 10. Make DB query: SQL1 | 10 | ||
11. Get credit card balance and write it down: ____ | Step 2 – Step 8 | ||
How will we execute this test case? Simple:
– Execute first 10. Compare the AR and ER.
– Execute Step 11. Compare the AR and ER.
The test case would pass if, and only if, both conditions are true:
1. The AR after Step 10 equals “10” AND
2. The AR after Step 11 equals “Step 2 – Step 8″-i.e., the result of subtracting the amount of payment (Step 8) from the card balance before payment (Step 2).
In theory, it would be cleaner to split the test case into two parts and create two test cases:
1. IDEA: “The correct success code is inserted into the DB if Visa is used”
2. IDEA: “The correct amount is debited from the Visa balance during Checkout”
If possible, it’s better to create two test cases, BUT in the vast majority of situations, it’s more practical to combine two or more ERs into one test case. The rationale:
– You probably will not have time to write, execute, and maintain TWO test cases. If we have only one situation where two ERs can be combined, writing, executing, and maintaining two test cases are not a problem. But what if you have HUNDREDS of situations like that, and thus hundreds of extra test cases?
– The saved time and effort can be spent writing, executing, and maintaining at least one more test case. As time goes by, you’ll create HUNDREDS of additional test cases that would make a huge difference on how thoroughly your application is tested.
I’ve been working with test cases that include more than one ER for many years. And the software that I’ve been testing is very complex financial software of PayPal that processes billions of dollars worth of payments. So, with all confidence I can tell you that two or more ERs in one test case is normal practice, and your effectiveness will only increase if you follow this approach for your projects.
Very often, when you have two or more ERs, you have to check for:
– a certain value on the Web page and/or
– a certain value in the DB and/or
– a certain value in the log file
BTW
A log file is simply a text file in a specified format that keeps track of particular events.
For example, a programmer can write a code that appends a text file in the following format every time someone successfully pays with credit card:
<UNIX timestamp*>,<card type: ‘V’ for Visa, ‘M’ for MasterCard, ‘A’ for AmEx>,<4 last digits of card number>,<amount in cents>,<order id>
Here is an example of the content of such log file:
1184970195,V,8277,1000,762
1184970197,M,7844,1000,763
ShareLane -> See Test Portal>Logs>cc_transaction_log.txt
*A UNIX timestamp is the number of seconds that have passed since midnight on January 1st, 1970. Programmers often use a UNIX timestamp to log the precise time of a specific event, like the creation of new user account.
This means that you need to check both the front end and the back end of your software to determine if that test case passed or not.
Brain positioning
The front end is the interface that customers can see and use; e.g., text, images, buttons, links. Compared to a car, the front-end pieces are items like the steering wheel and the dashboard.
The back end is the software and data that are hidden from the user; e.g., the Web server, the DB, the log files, etc. Compared to a car, the back-end pieces are items like the engine and the electrical circuit of the car.
One last note about two or more expected results: We looked at a situation requiring a step (“11. Get credit card balance”) between ER1 and ER2. In practice, you’ll also encounter situations when you’ll have two or more ERs in one place; in other words, your ERs will not be separated by steps. Here is a schematic steps-ER portion of such a test case:
1. Do something 2. Do something 3. Do something | Verify something Verify something |
Lecture 3 - Test Cases and Test Suites -> Quick Intro -> Test Case Structure -> Results Of The Test Case Execution -> Useful Attributes Of The Test Case -> Data-Driven Test Cases -> Maintainability Of Test Cases -> The Number Of Expected Results Inside One Test Case -> Bad Test Case Practices
