THINK BEFORE DOING: THE TEST AUTOMATION SOLUTION

Developing automated tests is not difficult. What is challenging is delivering a solution that is effective (it works!) and is sustainable (it can be mantained at low cost). The test automation solution is a crucial component.

Developing test automation solutions (TASs) is not trivial, as just like other software systems, they have to be modular, scalable, understandable, reliable, and maintainable.

For example, the CICD pipeline topic that is often being discussed by development teams is part of the TAS, but that is only one component. What should we be looking at when designing it? What is the appropriate kind of “design thinking” that we should be doing in this case?

In this post I’d like to explore the notions of Test Automation Architecture (TAA) and Test Automation Solutions (TAS) as defined and elaborated in the Test Automation Engineer Syllabus of the International Software Testing Qualification Board.

I think that for the any complex software project, some aspects of the TAS are important to be considered in the early stages of the development. For example, key components of the TAS need to be identified, good practices for designing and maintaining the TAS need to made explicit, an appropriate architecture of the TAS need to defined. This could help identifying new requirements that are important for the development and testing environment and therefore that might be wiser to consider sooner rather than later.

What follows is a brief summary of some of the main parts of the Test Automation Engineer Syllabus of the International Software Testing Qualification Board.

The Test Automation Solution

A TAS consists of both the test environment (and its artifacts) and the test suites (sets of test cases including test data). A test automation framework is used to realize a TAS: it provides support for the realization of the test environment and provides tools, test harnesses, or supporting libraries. The configuration management of a TAS may need to include the following testware:

Test models
Test definitions/specifications including test data, test cases and libraries
Test scripts
Test execution engines and supplementary tools and components
Test adaptors for the SUT
Simulators and emulators for the SUT environment
Test results and test reports
Tools for monitoring the testing process (such as ARGOS).

Normally, a TAS is being developed in various iterations/versions and needs to be compatible with the iterations/versions of the SUT. The testware comprising the TAS must be at the correct version to match the version of the SUT. Testing of an SUT should be synchronized with its development and synchronized with the TAS development. Therefore, it is advantageous to coordinate the processes for SUT development, TAS development and for testing.

The TAS must support the test management for the SUT: test reports including test logs and test results need to be extracted easily or automatically and need to be provided to the test management of the SUT.

Success factors for an effective deployment of the TAS include:

Incremental rollouts. In this way support to new users occurs iteratively rather than all at once. Also the usage of the TAS increases by steps. Bottlenecks and issues can be identified and solved early in the deployment.
Support to the testers and developers should be provided. Guidelines, FAQs, usage manuals should be provided to prevent extensive support effort.
Monitoring the usage of the TAS over a certain period of time indicates whether the TAS is indeed used. This could include also ways to gather information about how the TAS is being used and which components are used and which are not.
Gathering lessons learned from all teams.
Adapting and improving processes to fit with the use of the TAS: when different users use the TAS, different processes come in touch with the TAS, and need to be tuned to the TAS, or the TAS may need (small) adaptions to the processes.
Providing training and coaching/mentoring for new users: New users need training and coaching in the use of the new TAS.

Maintainability of the TAS is a crucial requirement in order to achieve a sustainable testing process. The TAS has to designed and kept modular, so that its components can be easily replaced and changed. Good practices include:

The TAS must be under configuration management.
Naming standards and other coding conventions should be defined so that TAS components are easy to understand, change and maintain. This would also make it easier to introduce new people to the TAS. The naming standards can refer to variables and files, test scenarios, keywords and keyword parameters. Other conventions refer to pre-requisites and post-actions for test execution, the content of the test data, the test environment, status of test execution, and execution logs and reports.
Well written documentation of the TAS can be used as a basis for training material of the TAS. This can include a specification of the TAS, its usage, with practical examples. Needless to say, such a documentation needs to be kept updated.

Maintainability is enhanced by adoption of a proper architecture, which is covered next.

The Test Automation Architecture

It is the architecture implemented by the TAS. To simplify the TAS development, evolution and maintenance, the well known SOLID software principles should be observed:

Single responsibility: Every TAS component must have a single responsibility, and that responsibility must be encapsulated entirely in the component. In other words, every component of a TAS should be in charge of exactly one thing, e.g., generating data, storing test scenarios, executing test cases, logging results, generating execution reports.
Open/closed principle. Every TAS component must be open for extension, but closed for modification. This principle means that it should be possible to modify or enrich the behavior of the components by extending them rather than by modifying them, therefore without breaking the backward compatible functionality.
Liskov substitution : Every TAS component must be replaceable without affecting the overall behavior of the TAS. The component can be replaced by one or more substituting components but the exhibited behavior must be the same.
Interface segregation. It is better to have more specific components than a general, multi-purpose component. This makes substitution and maintenance easier by eliminating unnecessary dependencies.
Dependency inversion: The components of a TAS must depend on abstractions rather than on low-level details. For example, high level test scenarios should not depend on actual lower levels components of the TAS (like the actual detailed sequence of actions needed to exercise the SUT). Instead they should depend on abstractions of these components (for example, a software layer that isolates details of the control or observation points of the SUT).

There are several other generic requirements for the TAA:

The TAA needs to be able to touch the control and observation interfaces of the SUT that are likely to be involved in test procedures.
An SUT uses configuration data to determine how it is instantiated and run. It uses data provided by users that will be processed when it is run. It may use also external data supplied by external services. All these data need to be defineable, possibly versionable and instantiable by the TAA.
At an abstract level, the TAA might comprise four layers for which it needs to provide support:
- The test generation layer consists of tools supporting:
  - Manually designing test cases
  - Developing, capturing, or deriving test data
  - Automatically generating test cases from models that define the SUT and/or its environment (i.e., in case of automated model-based testing)
  - These tools are used to edit and navigate test suite structures, to relate test cases to test objectives or SUT requirements, to document the test design.
- The test definition layer consists of tool support for:
  - Specifying test cases (at a high and/or low level)
  - Defining test data for low-level test cases
  - Developing test cases and maintaining them.
- The test execution layer consists of tool support for:
  - Executing test cases automatically
  - Logging the test case executions
  - Reporting the test results
- The test adaptation layer consists of tool support for:
  - Controlling the test harness
  - Interacting with the SUT
  - Monitoring the SUT
  - Simulating or emulating the SUT environment.

Conclusion

To what extent the current TAS for your project is well designed and ready to be deployed?

What should you do next to improve it?

IDS SrL Part. IVA 02077400301

Privacy Policy – Cookie Policy

Via Linussio 51, 33100 – Udine Italy

Phone: +39 0432 629717

contact@designcoaching.net

THINK BEFORE DOING: THE TEST AUTOMATION SOLUTION

The Test Automation Solution

The Test Automation Architecture

Conclusion

Follow us