Engineering Blog - May 7, 2012

Selenium Conf 2012 – Reliably Automating a Moving Target

Dan Cuellar

Two weeks ago at Selenium Conference 2012 in London, I presented a talk on writing functional test automation across multiple platforms in a fast-paced and experimental development environment. The presentation is available on YouTube and the slides are available on the conference web site, but I thought I’d share some of the content here on the blog.

Common Issues With Functional Test Automation
Before writing infrastructure to combat a rapidly changing platform, it’s important to take a look at the most common reasons functional automation testing fails. From my experience automating web sites using Selenium I’ve noticed a few themes.

  • Not enough information is retrieved to reliably reproduce the error
  • Hooks into the application move, disappear, or become ambiguous
  • The business logic changed and your test is now out of date
So, How do we combat these problems? If we are having trouble retrieving adequate information to diagnose failures, we need to ensure that we are logging any information that could be possibly be useful and doing so in an organized and uniform manner. As for hooks or UI locators breaking, there are two things we can do. The first is to centralize all hooks for the UI into a single location to ensure that every script and library is identifying the element using the same technique. We also need to develop a means to test our collection of hooks independently of regular automation so we can isolate hook failures from actual test script failures. My presentation gives an example of how to test your entire site UI model using reflection or introspection. As for frequent business logic changes, you can combat this by consolidating all your business logic into libraries and avoiding placing it in the more than one test script when possible.

Proper Abstraction of Functional Test Automation Infrastructure
Even with the techniques mentioned above, the whole problem space becomes a lot more manageable when you properly organize your automation infrastructure. At Zoosk we break it up into a few different components which allow for cross-platform test scripts. The components all draw clear lines as to which code goes where.

  • Hooks are means by which test automation can identify pieces of the user interface. To enable consistency and to reduce the cost of test repair, it’s important to centralize your hooks into a single location. Now when a piece of UI moves around or changes, there’s a single place you need to update it to repair all of your scripts.
  • Controllers are pieces of code that control a component or platform. Methods that perform low-level actions to manipulate a component belong here. For example: if you are writing a web browser controller, it could perform tasks like clicking, going to urls, gathering cookies, etc. If the controller were operating on an iPhone it would do things like tap, swipe, and pinch.Controllers should be rich with logging so that when investigation time comes around you have more than enough information to trace back what happened step by step.
  • Libraries are methods (groups of calls to controllers) that perform product-level actions using a controller on a particular platform. A library contains methods that orchestrate controllers into higher level actions. A login method in a library would check if the user is signed in by looking at a cookie first. If it then determined that the user was not logged in, it would then fill out the login form and submit it and validate the proper cookie is set.
  • Interfaces are used to remove the platform context from libraries. If you think about Zoosk we have several platforms; there is a website, a Facebook app, a mobile website, native iPhone and iPad apps along with an Android app, but all of these platforms share common actions. Each platform supports tasks like authentication, messaging, searching, and more. Implementing an interface allows for tests to be written in a platform-agnostic fashion. For example: When a user deletes their Zoosk account, they can no longer sign in. We can write a test that deletes an account, then verifies on every platform using a signing method in the shared interface to valid authentication fails on each platform.
  • Test Scripts coordinate interface calls (or library calls if the test is platform specific.) Their purpose is to test a particular product scenario and report back if the behavior is as expected. When tests call the interface, the business logic of the test can be maintained in a single place, minimizing the amount of code changes that need to be made in test automation when business logic changes.

If you’d like more information, feel free to check out the slides linked above or post a comment and I’ll get back to you.