Lecture 10 - Test Execution: Regression Testing -> Quick Intro -> How to Choose Test Suites for Regression Testing -> How To Resolve The Contradiction Between Our Limited Resources And The Ever-Growing Number Of Test Suites -> Automation Of Regression Testing: Do It Right OR Forget About It -> When Regression Testing Stops -> Lecture Recap -> Questions & Exercises1. We need regression testing because:
a. It’s often extremely difficult for a programmer to know how change in one part of the software will echo in other parts of software.
AND what’s even worse…
b. Programmers sometimes change software without even trying to figure out if their changes will break something.
2. Testers usually know (or can make an educated guess) which features will be directly affected by a change.
3. The 1st group of test suites for regression must contain the test suites checking legacy (i.e., existing) features that are directly affected by a change in the software.
4. The 2nd group of test suites for regression must contain the test suites checking legacy features that may somehow depend on the changed features.
5. The 3rd group of test suites for regression testing contains the test suites checking legacy features that don’t fall into the first or second group.
6. The fact that a test suite falls into the 1st, 2nd, or 3rd group doesn’t mean that we are going to execute it. For example, we might be short of time.
7. Below are the ways to settle the contradiction between our limited resources and the ever-growing number of test suites:
a. Prioritization of test suites and test cases
b. Test suite optimization
c. New hires or outsourcing
d. Test automation
8. Prioritization of test suites and test suites/cases is the most inexpensive and (in most cases) the most effective way to resolve this issue.
9. TA for regression testing comes in two forms:
– Helpers
– Automation scripts
10. Helpers are loved by everybody, because they are a real help in the manual execution of everyday routine tasks. Sometimes testing is not possible without helpers (Credit Card Generator).
11. Automation scripts come in the form of:
– Tools for component automation
– Scripts for end-to-end automation
12. The number one enemy of TA is MAINTENANCE.
13. TA maintenance has two roots:
– Start-up software is always changing, thus the TA must be changed too.
– TA can be poorly designed, so even a small change in the tested software can affect the maintenance of that TA.
14. Helpers are less prone to maintenance that automation scripts, because they don’t test software; they automate tasks around testing.
15. Component automation must use short, isolated test flows in order to be less prone to maintenance.
16. In most cases, end-to-end automation in start-ups is a BAD idea.
17. In majority of cases, if the TA produces an error, it doesn’t mean that the software has a bug.
18. If TA produces an error and that error is not related to a bug in the software, then time must be spent on maintenance.
19. Good TA saves money. Bad TA is a hungry monster that can eat up hundreds of thousands of dollars with nothing or little in return.
20. Questions to ask before automating something:
a. How stable is the tested piece of software?
b. What are the frequency and length of manual execution of this candidate for automation?
c. What is the priority of the test suite/case to be automated?
d. How much longer will that task be around?
e. How hard will it be to write TA?
21. There are 2 things that affect the end of RT:
– The deadline set for RT completion
– The test cases that we have decided to execute for RT
22. The challenge is to execute all test cases that we have decided to execute for RT before or at the deadline set for RT completion.
23. The decision about whether to release or not is made at the Go/No Go meeting. Next ->
Lecture 10 - Test Execution: Regression Testing -> Quick Intro -> How to Choose Test Suites for Regression Testing -> How To Resolve The Contradiction Between Our Limited Resources And The Ever-Growing Number Of Test Suites -> Automation Of Regression Testing: Do It Right OR Forget About It -> When Regression Testing Stops -> Lecture Recap -> Questions & Exercises
