The mainstream adoption of agile development methods is driving an increased emphasis on test automation. But will test automation allow businesses to more successfully implement agile methodologies, or erode in-house quality assurance (QA)?
As more companies move from a traditional waterfall software development approach to an agile one, suppliers are offering more test automation tools and services.
When IBM acquired cloud-based software testing company Green Hat at the start of the year, analysts said the move signalled pressure on suppliers to provide automation tools for agile environments.
Michael Azoff, principal analyst at Ovum, says: “The drive towards agile development with higher frequency of testing and testing earlier in the lifecycle is producing huge pressures on quality assurance.”
Green Hat, with customers such as BP, British Airways and Scottish Power, allows IBM’s Rational Suite development products to support agile development.
One of Green Hat’s customers, T-Mobile, used its testing tool, GH Tester, to solve production issues caused by its service-oriented architecture (SOA) after finding Tibco’s BusinessWorks platform lacked consistency across individual teams. Jeroen van der Sman, manager of in-house software at T-Mobile, says the standard ad hoc test-as-you-go approach adopted by developers for new functionality was inadequate and left the SOA open to faults. The company decided to build a test case library, creating automated tests for all its processes. As a result, development productivity increased 30% while the company reduced errors by 40%, he says.
Research firm Forrester expects testing-as-a-service (TaaS) to emerge as a managed test service from an increase in test automation, as large suppliers such as HP seek to support customers with test automation tools and outsourcing options, including security testing and services for SAP. Better test automation is intrinsic to agile development
Dorothy Graham, a software testing consultant and co-author of Experiences of Test Automation, says: “You can have automation without being agile, but you can’t do agile development without automation. As people get better at doing good automation in agile development, we will see increased velocity and consistency.”
However, Graham says automation has its limitations: “A repeated test is much less likely to find a new bug than a new test.” Hence, manual testers are still needed.
What future for Google’s software test engineers?
“Simply put, we don’t believe there is a future software engineer in test (SET). The SET is a developer. Period. Google pays them as developers, calibrates their performance reviews against developers, and calls both roles software engineers. So many similarities can only lead to one conclusion: They are the exact same role.
“As doomed as we believe the role is, the work itself cannot go away. The magic in the Google formula is the work the SET performs. SETs provide features such as testability, reliability, debugging capability and so on. If we treat these things as features in the same sense that we treat the user interface and other functional components as features, then SETs are nothing more than developers who own these features. This is the evolution of the role we think will happen at Google and other mature software shops in the near future and what better way to make test development a first-class citizen than treating it just like any other feature?
“Indeed, this is the part of the process that is flawed in its current state. Every user-facing feature is managed by product managers (PMs) and built by software engineers. Code for these features is tracked, managed, and maintained by a well-defined automated workflow. However, test code is managed by test engineers and built by SETs. Why? This is a relic of the history of how the roles evolved. But the evolution has peaked and it’s time to treat test code as a first-class citizen: that it be managed by PMs and built by software engineers. Instead, ownership of testing features should fall to new team members, particularly the more junior ones.
“Here’s our reasoning. Testing features cut across the entire footprint of a product. As such, the developers involved in building testing features are forced to learn the product from interfaces to APIs. What better way is there to take a deep dive into a product and learn its design and architecture quickly?
“Owning the test feature (either building it from scratch, modifying, or maintaining it) is the perfect starter project for any developer on any team. Arguably it is even the best starter project. As new members are on board, existing test developers move to feature development to make way for the new engineers. Everyone is fresh and engaged and over time, all developers are test savvy and take quality seriously.”
This is an excerpt from chapter 5 of How Google Tests Software, by James Whittaker, Jason Arbon and Jeff Carollo, published by Pearson/Addison-Wesley Professional, ISBN 0321803027
Shift in test ownership
But James Whittaker, development manager at Microsoft and former engineering director at Google, believes test automation will eventually shift test ownership out of the IT department and into the hands of the users.
Whittaker’s co-authored new book, How Google Tests Software, was released in March 2012. He says: “The biggest difference between Google and Microsoft is test ownership is distributed differently. At Microsoft, testers own test. At Google, developers and test teams share ownership.”
But this will change in the future. According to Whittaker, while repetitive and precise tasks should be automated, details and intuition in testing requires too many manual testers.
“Testing by using your own product and getting users involved in test is very important,” says Whittaker.
“I think that once all the automation is written and the crowd engaged, there is very little left for testers to do,” he adds.
Companies such as Google, Microsoft, BBC and NHS are making use of crowd-sourcing platforms for testing. The usability testing for NHS Direct’s symptom-checker application for Apple iPhone and Google Android devices was conducted via uTest , an online community of over 50,000 QA professionals.
Faced with a lack of in-house resource, Charlie Young, the lead consultant for NHS Direct, says: “People use smartphones in different ways, so we needed to make sure the user experience lived up to expectations.”
But beyond usability testing, uTest is also used by businesses to conduct functional testing, traditionally conducted by in-house QA professionals. Bob Doubell, The Met Office’s senior tester, received around 150 reported defects in 24 hours and says crowd-sourcing in this way allows valuable defects to be sent to developers in a short amount of time.
Despite the benefits of automation and crowd-sourcing, Amy Phillips, QA lead at silicon roundabout startup Songkick, says, ultimately, tests can only be used to carry out testing.
“Tests are a retrospective view of the code you have written so you still need to have the communication channels and processes in place to build quality into the code in the first place,” says Phillips.
Greater automation and increased crowd-sourcing look set to change agile development and quality assurance processes. While experts continue to debate the extent to which test automation and user crowd-sourcing may eliminate the need for in-house test expertise, the need to continuously ensure code quality and the need for fast and reliable reporting remain obstacles.
Top three tips to agile testing
- Continually evolve -Agree on the simplest possible process and then evolve it to add complexity as you need to. Robust processes and automated tests are essential. Nothing slows an agile development cycle down as much as finding 100 bugs to fix. Testers need to do everything they can to test early and make sure developers know the risks and code to deal with them.
- Build your own methodology - More companies are using home-grown agile approaches to build and release software. Agile relies on individual team members, using previous experience to create the best possible process, whether that’s the best parts from scrum, lean, extreme programming and even waterfall being mixed together to form an agile approach.
- Communication is key - Unlike more traditional methodologies agile testing is all about culture and communication, so successful agile teams are those who work together to continually evolve their process.
Source: Amy Phillips, QA lead at Songkick