MODEL-DRIVEN TESTING FOR ACCESSIBILITY

In his posts Karl Groves makes the interesting point that accessibility testing of a website could be framed in the context of software testing, and therefore could be based on the same tools and practices that developers adopt. In particular he suggests using the Tenon service and building testing scripts for accessibility that conform to certain standards so that these scripts can be run automatically, and automatically produce test reports; even included in continuous integration services such as Jenkins.

The expected benefits for developers are that they can get these tests to be run automatically whenever a new build of the site is made available, and therefore get to know if applied changes affect somehow the accessibility status of the website.

In this post we’d like to make a step forward, and explore some issues related the ideas of model-driven testing. Model-driven testing is the practice of using models and automatically derive from them the source code of the test-harness (TH) system (in our case testing scripts and the possible testing infrastructure) that can be used to test another system, the system-under-test (SUT), in our case the website/webapp.

The TH is needed to cope with fragility of testing scripts against changes in the details of the user interface implementation. If we were to follow a record-and-play approach (such as when using Selenium IDE to record a user interaction session, and then replay it again), the resulting scripts would be both:

non-parametric with respect to data, requiring us to add variables and parameters, as well as datasets, so that the same script could be reused more than once; such as a login script that could accept more than one set of credentials, and
fragile against changes in the DOM structure: in fact, recorded test scripts would refer to specific element IDs, XPATH expressions or CSS selectors to identify elements in the DOM in order to activate links or buttons or evaluate test assertions. Any change in the DOM could potentially disrupt most of our test scripts, requiring us to maintain and debug not only the SUT but also test scripts and the TH.

The TH could help reduce such a maintenance burden, by providing appropriate abstractions that insulate these details from the code. Similarly to the page-objects approach used when applying Selenium to websites, we here suggest following an approach which could be called State-Objects, where each state identifies a state of the SUT user interface.

The expected features and benefits from these practices are that:

the source code can be automatically generated from models, and therefore it can be done as often as needed at no cost;
test cases are independent from the details of how the user interface is implemented; as a consequence they can be easily written, they can be reused across changes of the user interface, they are stable against many changes of the user interface;
test cases can be made parametric, with data specified separately from scripts, so that these can be reused;
no particular skills are needed to write the TH nor the test cases.

While in many cases these model-driven practices are used for behavioral or acceptance testing, here we want to explore their viability in the context of accessibility.

Let’s focus on a concrete example.

A typical date picker, a user interface component that requires accessibility features.

The figure shows the user interface component of a website; the component implements a date picker. From a functional viewpoint that date picker can be used to select a date from the current month, to select today’s date, to move back and forward between months, to open a submenu (via the pull down menu on the month name) and choose a particular date of the selected month or close the submenu.

This is exactly the behavior that is represented in the model shown below.

The model is written as a UML state machine. The state ChooseADate is the initial state (for this widget) when the user interface shows only the text field, from which the user action called StartTyping leads to state TypingDate where the user would repeatedly do Typing to compose the date; if the date is Valid the user could completeTyping it and return to ChooseADate. Alternatively, the user could Open the date picker and then ChooseADay(d) (where d is the parameter holding the actual day of the month), or he or she could OpenMenuData (by clicking on the month name) and from then directly choose a year or month, or close it.

UML state machine model of the dynamics of the state picker.

Notice that in such a model no reference at all is made to the details of the user interface, such as link names, locators as XPATH expressions or CSS selectors. In fact these details are represented separately in configuration files that associate states and transitions of the model to these locators.

Writing the TH is then only a matter of creating classes (if we use an object oriented language for representing the TH, for example Java) and their operations for each of the states that describe the SUT. While this can be done manually, we do that through a tool we developed called GENTEC (Generator of TEst Cases).

Once the TH is ready, what remains to be done is writing the actual test cases. But this is now easy, thanks to the high level of abstraction provided by the TH. In fact the test engineer could now refer to states and transitions, and resulting test cases would be free from the problems mentioned above. Each test case could be framed as a jUnit test, for example, so that it can be run by one of the many test engines (as an Ant task, within IDEs such as Eclipse, within Continuous Integration tools such as Jenkins).

For example, to test that when an incorrect date is typed an appropriate error message appears and the date is not accepted, one could write:

sut.initialize();
sut.action(“StartTyping”);// move focus on text field
sut.action(“Typing”, “22/05/9999”);// input an incorrect year
sut.action(“CompleteTyping”);// press enter
sut.stateIs(“TypingDate”); // check that the resulting state is …
sut.check(“errorMessageContains”, “incorrect year”);

Alternatively, for a correct date the script could be:

sut.initialize();
sut.action(“StartTyping”);// move focus on text field
sut.action(“Typing”, “22/05/2014”);// input a correct year
sut.action(“CompleteTyping”);// press enter
sut.stateIs(“ChooseADate”); // check that the resulting state is …
sut.check(“NoErrorMessage”);

These are test cases that check correctness of functionality of the SUT. Other test cases can be written that test whether appropriate accessibility features are in place. For example:

sut.initialize();
sut.stateIs(“ChooseADate”);
sut.check(“tableHasTH”, “ext-gen276”);// id of the table

where the last check() is defined in terms of the following code fragment (automatically assembled by GENTEC from appropriate configuration specifications):

// check that the table contains TH tags
String selectorKey = “//table/tbody/th”;
By selector = By.xpath(selectorKey);
List eltList = sut.findElements(selector);
assertNotNull(String.format(“Unable to find TH for table %s”, tableId), eltList;
sut.log(“checked TH %s”, name);// add a log entry

A similar check can be written to make sure that TH elements have the appropriate scope attributes:

// check that the table contains TH tags with scope attributes
String selectorKey = “//table/tbody/th[@scope]”;
By selector = By.xpath(selectorKey);
List eltList = sut.findElements(selector);
assertNotNull(String.format(“Unable to find TH with scope attribute for table %s”, tableId), eltList;
sut.log(“checked TH[@scope] %s”, name);// add a log entry

Notice that the test engineer does not have to mingle with details of the user interface, which are kept separate from the TH code and from actual test cases (and expressed as XML clauses).

To sum up we believe that the benefits of following this “deep modeling” approach are that:

The models are independent from the testing platform, and could be reused to test for example web apps through Selenium Webdriver (as we have done here) or through QTestLib or Android Test Framework.
The models are independent from implementation details of the user interface, and the resulting TH could be reused many times. Should the behavior of the user interface change, then we should edit the models, adapt some of the test cases and the configuration clauses, and regenerate the TH.
Models are formulated using a standard language, UML and UML state diagrams.
Because of the interfaces provided by the TH, test cases are also independent from details of the user interface, providing us with stable and potentially solid code, easily reusable. Test cases are also parametric with data that can be provided in tabular form, as in Cucumber.
Accessibility test cases can be defined as any other functional test, and therefore can be run as any other test case by test engines.
Accessibility test cases can be as specific as needed and refer to deep aspects of the user interface, therefore reducing the number of false positives that usually are associated to automated tests of accessibility.

IDS SrL Part. IVA 02077400301

Privacy Policy – Cookie Policy

Via Linussio 51, 33100 – Udine Italy

Phone: +39 0432 629717

contact@designcoaching.net

MODEL-DRIVEN TESTING FOR ACCESSIBILITY

Follow us