Lecture 9. Test Execution: New Feature Testing -> Quick Intro -> Test Estimates -> Entry/Exit Criteria
As a rule, software companies have a Release schedule. Release schedules can come in many forms, from:
– “Everyone at ShareLane knows that we do a release every Thursday”
to
– Complex document with a list of new features, references to specs, start/end times for each stage of SDLC, etc., for several future releases.
Naturally, the QA schedule is linked to release schedule.
Let’s assume that, according to QA schedule, we have:
– Two weeks (10 working days + 4 weekend days – which can also become working days) for test preps
– Two weeks for test execution
In other words, we have:
– Two weeks to write test cases and do other test preps (e.g., create a test automation helper)
– Two weeks for NFT and RT
The problem is that even if testers stop eating, sleeping, and watching YouTube videos, there is a limit of how much testing can be done. Thus, there is an eternal conflict between:
– The avalanche of new features that marketing and PMs want to push
and
– The abilities of testers to do proper testing
To find the status quo between the desired and the possible, testers submit test estimates. A test estimate includes:
– Time for test preps
– Time for the NFT
After the spec is written, the test manager asks the tester to read it and evaluate how much time it will take to:
– Write test cases and do other preps for the NFT
– Execute the NFT
The tester reads the spec, talks to the PM and the programmer, and – based on all that info – he submits two figures to his manager: one figure is the estimated time for test preps and the other figure is an estimated time for the NFT. The time is usually represented in hours.
Example
A tester was asked to create a test estimate for Spec #8112 “New search features”. The tester gave the following to his manager:
50 hours for test case generation + 20 hours to write an automation tool
60 hours for the NFT
If we have 2 weeks (80 hours) for Test preps, then the tester has 10 hours left (80-50-20). Those 10 hours can be spent helping other testers, or generating more test cases. It’s also good to have some time reserved in case the estimate is not correct or in case something doesn’t go as planned, e.g., if the PM has to change a spec.
In this example it is more complex with test execution. There are only 20 hours left (80-60) for RT. Will it be enough? It depends on number of factors. For example:
– How many test cases do we plan to execute for RT?
-What is the overall load on the testers? Maybe some testers have a lesser load and can help their amigos.
How do we write test estimates? The challenge here is that test estimates are created after the spec was simply read. But the difference between spec reading and the actual testing of the spec’s feature is like the difference between reading about skydiving and actually jumping from the plane: HUGE. Therefore, when you are writing your test cases, or executing those test cases, hundreds of nuances come to light…nuances that you couldn’t even imagine during spec reading.
Below are the factors I recommend that you consider while writing test estimates:
1. What is the complexity of the feature to be tested?
2. Do you have any experience with testing similar features?
3. Is there any planned integration with the vendor’s software?
1. What is the complexity of the feature to be tested?
The rule is: The more complex the feature, the more nuances will emerge; the more nuances that emerge, the more time needs to be spent on testing.
2. Do you have any experience with testing similar features?
For example, if you are experienced with testing Checkout, you’ll spend much less time testing the addition of another type of credit card compared to a person who has never tested Checkout.
An experienced person does a lot of things automatically; he knows where to look for answers and who can help.
An inexperienced person MUST spend time exploring things, reading documentation (which often sucks or is outdated), bothering various people with questions, etc.
The bottom line is: If you are inexperienced with some feature (especially a complex one) make sure to build your learning curve into your test estimates.
BTW
In the case of complex software, there are people in the QA department who become experts in particular parts of that software. This is a great thing, but there is a danger that an expert can become ill or even depart this wonderful world right in the middle of something important – for example, during test case generation. So a QA manager must make sure that the testing of complex features is well documented and that this documentation is properly maintained. Also, it’s a good idea to assign 2 testers (at least) to each complex area – this way we have some “insurance” in case of an unfortunate incident with one of them (unless of course they get into the accident together).
3. Is there any planned integration with the vendor’s software?
Let’s say that we integrate our software with a payment processor. The test architecture looks like this: Our test environment “talks” to the vendor’s test environment. In case there is a bug (or another type of problem, e.g., a server is down) on our side, we can solve the problem like this: “Hey, Billy, can you sort this out?” But in the case of a problem on the vendor’s side, it can take hours or even days before they might fix it; our PjM must talk to the vendor’s PjM, the vendor’s PjM must talk to their developers; their developers might be busy with other stuff, etc. Therefore, the gist of the problem is that our issues are not their issues. Thus, if there is a planned integration with the vendor’s software, you should always increase your test estimate for the NFT. How much should the increase be? Each situation is unique; the complexity of software and of the integration, and any previous experience with the same vendor are important factors here.
BTW
After you have your final test estimate, increase it 10%, just in case of any unforeseen circumstances.
BTW
A very simple and working approach that I’ve seen is to dedicate 1 day of test preps + 1 day of NFT for each page of the spec. Next ->
Lecture 9. Test Execution: New Feature Testing -> Quick Intro -> Test Estimates -> Entry/Exit Criteria