What To Do If You Are The First Test Engineer At A Software Start-Up

Lecture 12 - Bonus Tracks -> Quick Intro -> How Much Technology Knowledge Must A Beginner Tester Have -> What To Do If You Are The First Test Engineer At A Software Start-Up -> What If You Are Asked To Do Test Automation As Your First Task

lol-doggie-i-can-walks-on-water-300x217Below is a list of first steps for that happy camper who is the first and only tester at a software start-up. The order of execution doesn’t matter, but 1 and 2 are the most important.

1. Install a bug tracking system

2. Learn the product, write a set of Acceptance Tests, and start executing them before each release

3. Bring good standards and processes to the attention of your colleagues and management


Your purpose as a tester is to find and address bugs, so a bug tracking system is one of the most important instruments you need to do your job. When you begin working at your start-up, it’s likely that:

1. The developers already use some free BTS (let’s assume that it’s Bugzilla) OR

2. Bugs are being tracked by something like an Excel spreadsheet

In the first case, create a Bugzilla account for yourself and ask the Bugzilla admin to

a. Give you permission to close bugs (common users cannot do it) and

b. Make you the default QA contact (in this case, you’ll be getting emails when bugs are filed/modified/closed).

In the second case, ask the IT guy or gal to help you install Bugzilla (http://www.bugzilla.org/) on some server on the intranet. After Bugzilla is installed, do steps a. and b. listed above.


Bugzilla is not the most sophisticated BTS out there in terms of functionalities, but it’s free, reliable, and easy to install and maintain.

The Bug Tracking Procedure (BTP) of Bugzilla is simple and very easy to understand. Spend some time playing with it: e.g., file bogus bugs and try to follow different flows of BTP. You can become a Bugzilla expert (in terms of BTP) within an hour.

One of the challenges for you will be to encourage the developers to follow the BTP, because while they tend to like the idea of Bugzilla, they tend not to like the fact that they have to use it properly -i.e., not only file bugs, but also update the Resolution and Status (or “state” in Bugzilla terms) as bugs are being repaired. Just have the patience, tact, and wisdom to communicate the necessity properly Bugzilla usage. You can find a diagram with the Bugzilla BTP here: http://www.bugzilla.org/docs/3.0/html/lifecycle.html.

Another important thing is to ask your coworkers from other departments than dev and QA (i.e., marketing, product management, business development, etc.) to file bugs using Bugzilla. As a rule, these folks prefer to communicate bugs via email or a friendly chat. This is not a good practice for many reasons – for instance, the difficulty of tracking the status of a bug. So make a presentation of Bugzilla usage for your start-up brothers and sisters, and be ready to answer questions and provide Bugzilla educational support during and after your presentation.


Bugzilla actually has the functionality that allows filing bugs by sending an email to Bugzilla. This functionality is LOVED by non-technical folks – talk to your Bugzilla admin about it.


Let’s assume that you have been hired to do manual functional testing.

Testing activities can be started in many different ways. Your manager might ask you to work on just one part of a Web site: e.g., start testing the Checkout. But in the majority of cases, you are simply told, “Start testing,” and it will be up to you to develop the strategy. Let’s talk about that strategy.

Here are the steps that I recommend:

– Learn the product

– Create a set of Acceptance Tests

– Start executing those Acceptance Tests before each release

As you’ve already learnt, exploration is one of the best sources of information about a Web site. If you are hired by some big corporation, you usually get a temporary mentor who helps you to get started. In a start-up, as the first tester you’ll get a lot of sympathy, but nobody will be babysitting you because everyone has more important things to do. So it will be up to you to dig up data and start testing. Of course, you’ll be asking for help, but as a rule, your first knowledge will come from exploring. Besides, once you are familiar with Web site functionalities, your questions will make much more sense to the developers. Make sure to take notes about any testing ideas that occur to you.

Let’s assume that you’ve done enough exploring and questioning, and you are now ready to start creating the test documentation. Your first piece of documentation should be a test suite with Acceptance Tests. Why Acceptance Tests? Let’s say that your Web site already has n number of important functionalities: e.g., Registration, Search, Checkout, etc. In the future, you’ll need to write test cases to perform deep functional testing, digging into the nuances of each of them, but right now you don’t have the time for this, especially if there are frequent releases (many start-ups have one release per week). So you have a dilemma:

– Either you write test cases in order to do a thorough testing of only one or a few chosen functionalities: i.e., go into DEPTH

– Or you single out the most important flows for each existing functionality and write test cases to check all of those flows: i.e., go into BREADTH

Again, each case is different, but I find it logical to go into breadth when you are just starting your testing.

It’s usually good idea to create an Acceptance Tests in the form of an Acceptance Checklist (see downloads on qatutor.com). The idea is to have a document that can be used to check out the general health of the software before each release. Another important aspect of an Acceptance Tests is that it is an actual testing document; i.e., a set of information stored, not in someone’s head, but in physical form.

Your efforts to create and execute Acceptance Tests will be largely welcomed by your coworkers, because by executing those tests you’ll be preventing a lot of nasty bugs from going into production.


In terms of politics, this is the hardest part of all for both the beginner and the experienced tester. Start-ups are famous for their atmosphere of democracy and openness, and even a slight attempt to put some limitations on those great qualities can meet with opposition – not because of the nature of the limitation, but because for many start-up employees the word “process” is associated with evil. But even in start-ups, we DO need good processes and standards (further P&S). The reason is simple: good P&S make our work more efficient and easier to perform. The very important prerequisite of a good P&S is: “The only reason a P&S should be introduced is to improve things.” This motto sounds very natural, but you’d be surprised how many P&Ss are introduced for other reasons, like “At my old company we did it that way.”

Now, how can a P&S be introduced in the first place? When you come up with new ideas, especially about new processes, the smart way to do it is to collect feedback first and then make your idea public. The reason is that start-up folks are usually very smart, creative, and active. So, if your idea comes out of the blue during an engineering meeting, you’ll probably get at lot of opposition; there’s nothing personal about it – your colleagues will just try to express their opinions. Instead, let them do it before the meeting: meet privately with some of the influential folks within your company (PMs, developers, managers), share your idea, ask them for feedback, and try to incorporate their opinions into your future presentation. This way, when you go public with that presentation, many respected folks will already know about your idea and support it. Another approach, which can be combined with the one above, is to ask one of your more senior colleagues to help present your idea in public.

It’s different in every company, but usually a quality related P&S is considered to be officially accepted once it’s been voted for at the engineering meeting.

The first pieces of a P&S that you can gradually, gently, and smartly introduce are:

a. Bug Priority Definitions

b. Bug Tracking Procedure

c. Bug Resolution Times

a. Bug Priority Definitions

Bug Priority Definitions is basically internal standard (within a company) about what to call P1, P2, P3, and P4 bugs. Definitions create a guide to help assign correct priority for each particular bug. There WILL be room for dispute (about the priority of some bugs) even after the Bug Priority Definitions are accepted, but in the many cases you can simply point to this document and say: “We’ve all agreed on our Bug Priority Definitions, and this particular bug matches the criteria for P2.”

b. Bug Tracking Procedure

The Bug Tracking Procedure (BTP) is set of rules about how bug tracking should function from the moment a bug is found and filed (or re-opened) to the moment when a bug is closed. Everyone who uses the BTS for more than simple bug filing should understand the BTP. The following is an illustration of the importance of proper BTP use.


There is a P1 bug that stops the release (a showstopper). At 8:00 pm on Friday, the development manager checks the BTS and finds out that the bug wasn’t fixed. The responsible programmer, who is also the only person capable of fixing that bug, has already left the office in stealth mode and doesn’t pick up his phone. So the manager tells his manager: “We cannot go live; we’ll have to wait till Monday.”

In this situation, everyone suffers:

– the programmer, who didn’t fix the bug

– the manager whose people don’t care

– and the rest of the company, because the release doesn’t take place

But to make matters uglier: in reality, the bug WAS fixed, but the programmer didn’t change its Resolution to “Fixed “! So, not taking a simple fifteen seconds of action – changing the bug Resolution – stopped the release from going out. This happened because the programmer

– either didn’t know about the accepted BTP at his company

– or negligently ignored it.

The above example is an extreme case, but in reality, not knowing about or ignoring the BTP by those who should know and follow it brings A LOT of inconvenience, miscommunication, and wasted energy.


If you have Bugzilla as your BTS

– you can use the default Bugzilla BTP (see link above) or

– you can develop some kind of custom process, but this will require a Bugzilla code change, and I don’t think you need that.

If you use commercial BTS software like Test Director (Mercury Interactive), it usually comes with a default process with the possibility for easy process customization (no BTS code rewriting is needed).

c. Bug Resolution Times

Why do we need bug priorities? Is it only about making some bugs more important than others? Let’s say that bug #234 has priority P1 and bug #211 has priority P2. Naturally, #234 should be fixed before #211, but here is the question: What is the time frame for fixing that P1 bug #234? Is it two hours, three days, or one week? The point is this: bug priority doesn’t make much sense if there’s no concrete bug fix time frame associated with each bug priority. The Bug Resolution Times document is where these associations are made.

Brain positioning

If there is no Bug Resolution Times standard in the company and developers don’t fix their bugs within a reasonable (whatever it is!) period of time, the test engineer can be easily blamed for not putting enough pressure on them or for not effectively “selling” bugs to them. But if that standard IS in place, it becomes the problem of each particular developer who doesn’t comply; testers cannot be blamed anymore. If the company has a standard, and the developers don’t follow it, it becomes the development manager’s problem, not the test engineer’s. That’s how the P&S take care of the situation and unloads unnecessary problems from the shoulders of the testers. Next ->

Lecture 12 - Bonus Tracks -> Quick Intro -> How Much Technology Knowledge Must A Beginner Tester Have -> What To Do If You Are The First Test Engineer At A Software Start-Up -> What If You Are Asked To Do Test Automation As Your First Task