QA: A Black Box?

Ever interact with a Quality Assurance (QA) organization and feel like your teeth are getting pulled? That should never be the case. Communication is the key to resolving this matter.

After I took over the reins of the QA automation team at Zoosk, I noticed there were two things that needed to be addressed: 1) Other teams did not know what the automation team did and 2) Other teams did not know how the automation team contributed to the company. We were essentially a black box. Offering transparency into our daily workings would alleviate these problems.

The main purpose of the automation team at Zoosk is to maintain and update automated regression tests. Having an all-encompassing automated test pass allows the manual testers to be agile and ready to test new features that come down the pipeline. To showcase the daily workings of the automation team, we created an automation dashboard!

We display two different types of test passes in the dashboard – a daily pass and a custom pass.

The daily test pass runs at a specified time every day and with the same subset of suites. A suite is a list of tests that are grouped together. Currently, we have a little over 520 regression tests in our daily test pass. We run the daily pass against the same set of QA owned pre-production servers. This allows us to easily compare the results between the various runs and see trends over time. In order to keep maintenance time down for the automation team developers, we typically like to keep the pass above an 80% passing rate.

Daily runs showing an overall view. By hovering over a pass, a tooltip with the breakdown of results and a description of what occurred in the test pass will be seen.

Daily runs broken down by suites

The custom test pass is an “on demand” test pass. The custom pass typically tests a smaller subset of the suites and can be run on a developer’s own pre-production server. By letting developers test on their own servers, we allow them to test their own source code branches and validate that they did not break any other features.

Adding these two items into our dashboard has given the company good visibility into what the automation team does. Frequently, we get developers who want us to do passes on their branches to quickly validate that they did not break anything. We’ve also had other teams come up to us and ask about what they could do to help in our development and lower our maintenance costs.

While having transparency for your QA team is great, think about the bigger picture here. Having transparency across all organizations in the company brings good insight to the other teams. It offers a different means of communication than the normal channels. With communication open, other teams can come in and help out with the inefficiencies within your processes and see what is going on.

Communication is vital for a good work environment and a successful company.