A CASE STUDY IN EFFICIENT IMPLEMENTATION OF END-TO-END AUTOMATED TESTS

What does it mean to implement a test development and maintenance strategy that has to be sustainable? What is the required effort?

We have previously seen some metrics that can be used to determine the business value of a software test automation initiative based on end-to-end tests.

Here I illustrate how such an initiative can be implemented, and how the required effort can be assessed. Doing this is necessary to determine the extent to which the test approach is sustainable.

This is how we approach the problem with the 121T service.

The test harness

The system under test (SUT) is a web application in the financial sector, the same application mentioned in a previous post.

The data presented below come from one month of work by our team, and derive from analysing the commits of the code base that includes the test cases themselves and the software test harness that supports their execution.

This test harness (testware based on JUnit and Cucumber) is automatically generated by a proprietary tool that compiles into Java source files the finite state automata models specifying the expected behavior of the SUT. In particular we use UML State Machines. Each state of the model represents a state of a widget of the UI, and each transition represents an action that a user can do.

Our tool transforms states and transitions into Java classes and methods that can simulate an execution of the automata against a sequence of events, each sequence representing a test case.

The test harness can “touch” the SUT in a number of touch points, which correspond to action points of the SUT (buttons, links, text edit fields, etc.) and observation points (other elements of the user interface, called oracles, that are used in test conditions). Both action and observation points need some way to identify the specific elements in the DOM of the user interface, which are called “locators“; they are XPath expressions involving unique ID, CSS classes, HTML tags and attributes.

Part of the model of Gmail.

Collected data

The analysis comprises 66 commits done last January in 18 consecutive working days, 13 of which involved the model and locators.

Each model being committed was based on an average of 926 states, 790 pseudostates, 535 regions and 1950 transitions.
Each commit added 3 new states, 2 new pseudostates and regions, 6 new transitions. Some commit added a maximum of 42 states, 32 pseudostates, 25 regions and 80 transitions.
On average, for each commit, there were 4 new locators for actions and 3 new locators for oracles, 1 change to actions and no changes to oracles. In some cases there were as many as 40 new actions being added.
In total, there were 54 new actions, 40 new oracles, 17 changes to actions and 6 changes to oracles.
On average, per each single working day, there were 3 new actions, 2.2 new oracles, 0.9 changes to actions and 0.3 changes to oracles.
Each commit led on average to 160 changes spread on 11 source files and to 12 new files being created. The maximum was 739 changes spread on 55 files, and 108 new files being created.
In total, for the 13 different commits, there were 2077 changes spread on 146 files and the creation of 154 new files.
On average, per day, there were 115 changes on 8 source files and with 8 new files being created.
Each commit, on average, included 2619 source files, with a maximum of 2698; on average each commit included a total of 247,000 lines of source code, with a maximum of 270,000.
Each new action required 38 changes to source files, and each new oracle led to 51 changes. Each action changed 2.7 files, and each oracle 3.6.

Conclusions

All the changes to the source files of the test harness were generated automatically, by our model compilation tool. The equivalent manual effort is substantial:

Each working day our developers would be required to implement 3 new actions and 2 new oracles. This would require 115 changes to source files across a total of 2619 files.
Adding new actions and oracles leads to changes that are relatively widespread (on 38/51 files respectively). This is due to the nature of these actions and oracles, rather than the way in which they are implemented in the source code: very often the same action or same oracle is available on several states, and this requires appropriate changes to the source code.
Adding changes to the model has a considerable impact on the source code: for example, adding a new region may lead to 83 coordinated changes on the source code.

In each of these situations, developers would have to decide which files to change, make the change, and then test them. The compilation tool, being already substantially tested, produces code that is assumed to be correct. The developers have to make sure that locators and the model are correct, and this is done via specific debugging tools.

This analysis was carried out on commits that are typical of an ongoing test automation project.

A reasonable conclusion is that for end-to-end tests that involve the user interface of the SUT, a naive approach to automation of tests is likely to lead to a substantial maintenance effort, which undermines the sustainability of the testing process. Model-driven techniques are effective in reducing this effort.

IDS SrL Part. IVA 02077400301

Privacy Policy – Cookie Policy

Via Linussio 51, 33100 – Udine Italy

Phone: +39 0432 629717

contact@designcoaching.net

A CASE STUDY IN EFFICIENT IMPLEMENTATION OF END-TO-END AUTOMATED TESTS

The test harness

Collected data

Conclusions

Follow us