Exploring the World of Test Automation
In this article
There are several options for development teams to choose from when automating their functional testing. These tools or frameworks can be sorted into three categories:
- Click-and-record
- Automation-as-code
- Artificial intelligence/machine learning (AI/ML)
We explain these different types of automation tools by breaking down their benefits and weaknesses then providing our opinion on their value for different types of organizations.
Click-and-record tools
Click-and-record tools make automation easy by recording a session of user clicks and interaction within an application. The recorded session from the QA is turned into a repeatable test that can be easily played back at any time. The software also provides a code version of the test. Examples of these tools include Selenium IDE, TestComplete, Test Café and even some Chrome extensions like iMacros.
Benefits:
Click-and-record tools are a good choice for companies that do not have the technical QA (Quality Advocate) staff with programming experience to write automated tests through coding. The tests take less time to create than with code-based automation tools. These tools are also good for use with older applications or applications written in languages besides Javascript.
Weaknesses:
Tests created with click-and-record tools are often fragile and require a lot of maintenance in order to provide durability and long-term value. Creating repeatable functions and page objects, to enhance automation, are not straightforward to add and require staff with some programming experience. Some of these tools also require expensive user licensing.
Our opinion:
This is a good way for non-technical QA departments to start with test automation and as a gateway to automation-as-code. We believe that non-technical QA departments would get better ROI by training their people and/or hiring people with the skills to write automated tests as code.
Automation-as-code
Automation-as-code is where you are writing tests using a library of commands structured around a programming language. These tests are written like code with syntax, frameworks and assertions. For web applications, these tools include Cypress, Google Puppeteer, NightWatch and Selenium-based tools like WebDriver. Test Café even offers the ability to write tests through a code library without paying for the click-and-record license.
Benefits:
Automation-as-code allows you to create strong, dependable and reusable code which you get to create and control. Debugging is typically easier than click-and-record tools and the test runs are quicker to execute. These tools can be used with Windows, Mac and Linux whereas many click-and-record tools are Windows-only. Tests can also be run in parallel whereas other types of tools can only run test suites serially. Some of these are open-source tools and have a low barrier to entry because they do not require a user license.
Weaknesses:
Automation-as-code tools do require basic programming experience to create the test and configuration files. They also do not provide the same support that a licensed tool does. Open-source tools rely on an online community for help whereas licensed tools provide support directly from the software company.
Our opinion:
Even with these constraints, we believe that having the technical QA staff to support automation-as-code tools will result in better ROI. These tools are proven and affordable and will give you the best test automation experience for your web application, at this moment in time.
Artificial intelligence/machine learning tools
AI/ML tools are newer to the market and are considered by some to be the future of functional test automation . Examples of these tools are Eggplant, Test Sigma and Mabl. These tools not only incorporate AI/ML but combine them with click-and-record technology. Some of the tools also allow for the tests to be written through automation code.
Benefits:
These tools use AI/ML technologies to both fix broken tests and create new tests. Automated tests can often break because of software code being updated and the automated tests still referring to the old code. The AI/ML tests can heal themselves by recognizing the changes to the software code and updating the tests automatically. These technologies can also create new tests by tracking user analytics and discovering user paths that the test suite does not already incorporate.
Weaknesses:
The usefulness of AI technology is an open question as AI/ML tools are still in their infancy. The perceived beneficial implications are enormous, but time will tell if these tools live up to all their hype. Licenses for these tools can be even more expensive than the click-and-record tools.
Our opinion:
We believe that these tools are good for companies that have large budgets to take on the license fees and can allow the time for their developers or technical QAs to take advantage of all the features. As mentioned with the click-and-record tools, we believe that some experience with programming concepts is still needed to enhance the automation suite with repeatable functions and page objects.
Conclusion
Using an automation-as-code approach has allowed us at WWT to create stable and reliable automated test suites that we are able to run through continuous integration and create shorter feedback loops for our defects. We have experience with click-and-record tools if your organization is tied into an expensive license and is struggling to use it. We are excited about the future of AI-based tools and continue our research and experimentation with them. Contact an expert today to find out how WWT can spearhead your functional test automation.