According to Wikipedia, “Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.”
Test Automation, contrary to perception, is an eco-system of automated test scripts, including automation libraries, frameworks and processes. Test automation is done when:
– Tasks to be tested are highly redundant.
– Software is stable and has been tested several times.
– Minimal changes are expected in developed code.
– Tasks are so repetitive; avoiding human error is advantageous.
Establishing the connection: Improved test automation and better software
Even though test automation provides benefits in terms of repeatability, reduced time and costs, reusability, reliability and better quality of software being developed, there has been very little focus on improvement and maintenance of test automation. One of the main arguments substantiating this thought is “Automated test scripts and libraries that I develop is not delivered to the end customer. Customers won’t review my code and not find faults with it. Why do I care so much?”
Well, here are some facts. Test scripts are developed and executed to validate the product shipped (developed application). If the quality of test automation is not maintained, we can’t be sure of the quality of production code. If a test case doesn’t catch a product defect, there is no way to find if the production code was developed well or the test script was faulty.
From the above arguments, it can be deduced that the quality of test automation is directly related to the overall product quality.
Suggestions to achieve improved test automation
During the course of this section, the author would suggest practices the automation teams should follow to improve the quality of test automation.
Test Automation Architecture
Architectural diagram of test automation helps in getting an overall idea of how test automation is mapped with the Test Architecture. While preparing this document, automation teams often realize the need of automation scripts and libraries for user scenarios (from end-to-end testing perspective) that they might not have thought of. This results in improved test and automation coverage and thus improved confidence in Dev/QA teams and other project stakeholders. Hence, it is imperative that automation teams sufficient time in this activity.
Automation Planning
Automation plan should involve detailing of the type of automation framework that would be used to automate test scripts. Typically, automation teams work with one of the following four models:
- Keyword driven automation
- Data driven automation
- Modular automation
- Hybrid automation
Automation of few test scripts by using the decided type of automation framework is definitely useful during planning as this would validate the applicability of chosen automation framework for testing the developed application. This also ensures there is clarity on the structure of test scripts among automation engineers before the actual automation begins.
Building a Comprehensive & Robust Automation framework
1. Infrastructure to support cross platform, compatibility, security, performance tests and code coverage runs should be made an integral part of automation framework.
- Cross platforms testing can be achieved with the help of libraries or framework components that can perform Power On/Off, snapshot operations on virtual machines installed with different Operating Systems that application supports.
- Standard penetration tests including SQL Injection, cross site scripting, buffer overflow can be developed as test tools or libraries and can be executed on applications (wherever applicable) with the help of automation framework.
- Scripts can be written for installation of set of software against which the application needs to be tested for compatibility. It would be simple to call the installation scripts and check for successful installation for each build.
- On similar lines, libraries or automated tools can be made use of, for performance tests including soak, load and stress tests.
- Including capability of executing code coverage runs in the framework would boost the overall project productivity as this activity is jointly owned by automation, quality assurance and development teams.
By building this infrastructure, automation teams would ensure that they have improved test coverage (in turn finding more defects in the product) significantly with implicit advantages of test automation.
2. Monitoring and Reporting components of automation framework would not only give an indication of tests running at particular instance but also would help in getting the results of test runs at a single location. Graphs depicting results of test runs for product components would help in mining information including:
- Stability of component under test.
- Coverage of each component (results of code coverage runs).
- Type of testing that uncovered more defect.
- Overall product quality.
3. Logging and log analysis are important aspects of automation framework. Report of executed test runs may not be helpful in triaging bugs. Execution logs need to be analyzed to root cause and fix defects. Automated log analysis (log analyzer tool) would definitely improve turnaround time by development to fix defects and thus improve product quality as a whole.
Design review of test scripts and libraries
Automation frameworks provide infrastructure to execute tests, monitor & report test executions, analyze test results and improve test coverage. But sufficient care must be taken while designing automated scripts and libraries. Here are few points that should be taken care of:
- Methods in the libraries must be designed taking care of requirements provided by quality assurance engineers. Limited functionality or modular tasks must be developed in the form of methods. Tasks that fall under bigger umbrella should be part of automation library. For example, method would be written for testing a functionality of component, and all the methods for testing functionalities of a component would be part of component library. Test Scripts can be divided into following parts: setup, test and teardown where setup part handles the pre-requisites, configurations required for test case, test part handles the actual test case scenario and the teardown section performs the clean up and reverts the environment to the original state. This gives a structured look to the test script, thus improving readability and ease of maintenance.
- Design patterns are general reusable solution to a commonly occurring problem in software design. Considering design patterns while designing libraries would pay rich dividends. Library reviews should point out the need and the applicability of relevant design patterns. Aligning with such standard development practices help in easy maintenance of code within and across teams.
- Embedding the element of re-usability in scripts and libraries is imperative. Code reusability can be achieved with:
- Modular Scripts that can be useful to automate larger number of test cases.
- Library files for Project components.
- Functions catering to less complex project functionality.
Automation engineers should design test script and libraries to exploit the plug-and-play flexibility where each step of test case is only a matter of calling an already developed function, script or method in the library. Design reviews must take care of this aspect.
Automation code review
- Automation code reviews must emphasize test coverage aspect. For example, if a reviewer is assigned with test scripts for a single component, s/he can identify slippages in test coverage during reviews.
- Strict review of test scripts and library design would also help in standardizing documentation in turn substantiate easy understanding of script/library flow. Maintenance of documented artifacts is also easier.
- Test script and libraries can be executed and profiled for memory and cpu cycle times. This can be useful to remove bottlenecks in scripts/libraries that hamper faster execution of tests.
Conclusion
During the course of this article, we understood the needs and advantages of test automation. We established the impact of quality of test automation on the quality of production code. The author also provided suggestions and tips for achieving improved test automation.
References
- Wikipedia.org – Wikipedia, the free encyclopedia – Article on ‘Test Automation’, taken on 8 Jun 2011.
- Microsoft.com – Article on ‘Quality in the Test Automation Review Process and Design Review Template’, taken on 8 Jun 2011.
