THE MOST IMPORTANT TESTING QUESTION: WHY

In a previous post I introduced the notion of sustainable testing. I suggested that a sustainable testing process needs to support the business, to support developers, to be fast and frequent, and has to lead to an adequate Return on Investment.

A testing process, no matter how structured it is, is the basis for organizing the work and the artifacts and for monitoring the process to make sure that it actually works. The degree of structure of the process will vary, also significantly: from projects aiming at creating a simple e-commerce application to projects that have to build complex radio telescopes.

But in all cases the process has to match the context of the project (kind of software to be produced, its quality profile, the people working on it and those who will be buying or using it, the environments in which it will be run, the project constraints such as time and budget). As a consequence the testing process has to be customized, in such a way to preserve its sustainability.

The Why, What and How of testing

In setting up a sustainable testing process, there are three overarching questions that need to be answered.

The first question is “Why“: why should we spend time and money for testing? After all, the software that we are building either will help making more money because customers can do new things, or it will help saving some money because customers/users enjoy higher productivity.
This is not the case for testing: it only costs us time and money. Therefore we need to find out why we need testing.
Then, the second important question is how to structure the testing process itself. What are the systems/subsystems that we want to test? why those and not other ones? What types of tests (regression, functional, usability, etc) need to be put in place? Are these things appropriate for achieving the goals identified by the previous answers?
Finally, the third question is related to whether we are doing it right. Some interesting questions are how we identify defects, what is their life cycle, how do we measure the “intensity” of testing, how are tests designed and executed.

In this post I’d like to elaborate on how to make sure that a testing process can support the business, that is the “why” question. We’ll see that once this is established, then this provide a framework where the other decisions can be made more easily. I will discuss that in a future post.

Setting up a testing process that supports the business

How precisely testing can support the business is very tightly linked to the nature of the business and of the organization running it.

There are in fact many possible business objectives that testing can influence.

For example, one objective could be to increase trust that customers have with respect to our organization. This is of course affected also by the quality of the releases of our product.
Or perhaps, to increase credibility that our firm projects onto prospect clients. This can be affected by the quality of the demos and proof-of-concepts that we produce.
Another possible business objective is to increase productivity of our customers’ employees, which is affected by the reliability, utility and usability of the product.
Yet another objective might be to reduce the after-sale costs. This too is affected by usability of the product.

Notice that the most general question we are answering here is “Why quality matters to us?” Very often when I ask the “why” question, many of our customers stare at me with a puzzled look, because they have never thought of testing from that point of view. Yet, it is the most important one to answer.

Testing therefore can be viewed as service for the business. But precisely how can this service be useful for the business?

Finding bugs, which is often the most obvious answer, is just one option.

In fact testing can also inform us on the quality status of the system under test. This is a view of testing as a quality monitoring activity. This monitoring is likely to be driven by requirements and quality risks. One might be interested in knowing, for the current candidate release, what is the quality status of a particular requirement, for example those that are referred to by a marketing campaign. Traceability of tests to requirements is especially important in this context, in order to show to what extent a certain requirements was tested and what was the outcome. The outcome can be reported in terms of how many of those tests passed, failed or ended up in an error. Finally, coverage information is also useful: it would be good to know how many total tests could be written (with respect to some predefined and agreed notion of coverage criterion), and therefore how much was covered by the last test run. This quality monitoring could be used when deciding if to release or not, or to estimate the effort required in customer support after releasing the system.
Testing can help us verify compliance of the product with respect to existing standards, or with respect to a set of specified requirements, such as a list of critical functionalities that our product needs to perform in order to be deployable. For example, critical functionalities can be specified as use cases that are deemed by the customers to be essential.
Testing can be used also for monitoring the health status of an application, which also called “smoke testing” or “synthetic monitoring”. This is especially useful to test complex distributed systems, such as those based on a microservice architecture. The idea is to periodically run some automated focal usage scenarios on the system in production and to monitor response times and log messages. The aim is to detect large variation from the average values, and use this as an indicator that some component in the architecture is not working as it should anymore. Although not being a proper testing activity, this is often done by reusing some of the already existing test cases that are developed for end-to-end system tests.
Testing can provide a living documentation, that is documentation of the product and its behavior that is specific and guaranteed to be reliable. This is done by implementing automated tests in a language that can be understood also by non technical people (like business analysts) and that are void of technical aspects (like details about the interface of the SUT that is used, such as the ID of a button or the actual parameters of an API endpoint). In this way tests are THE documentation, which is kept updated by the testing process itself: if a test fails, either the system under test is fixed and the documentation does not need to change or the test is fixed, and therefore the documentation gets automatically updated.
Testing can help in validating requirements. In fact, testing activities can also help in clarifying user stories when during product backlog refinement sessions. For example, a tester could help identifying alternative paths of execution and additional acceptance criteria.
Testing can be used to validate fitness for use of the product. This can be done by exploratory testing, usability tests, user-acceptance tests and security tests, which are the best candidates to assess the degree to which the product is suitable to be used as intended.

The testing policy

All this is often specified in a short document or wiki page called “Test policy“.

In a properly defined testing policy one should find a crisp description of business objectives that are expected to be affected by testing, and corresponding ways in which testing can support them. For example, the business objective of increasing customers’ trust can be effectively supported by testing goals such as

monitoring quality so to enable us to make informed decisions on whether to release or not;
finding as many bugs as possible regarding a certain group of critical functionalities;
preventing regression bugs as much as possible.

Alignement of testing goals to business objectives is of paramount importance and characterizes a good policy. It is the final link that enables us to answer the “why are we testing?” question.

The biggest benefit of a testing policy is that it helps us making important decisions, such as “do we need to allocate more resouces to testing?”.

For example, some of our clients made the decision to invest in implementing a suite of automated end-to-end tests through the 121T service in order to check compliance of the system with respect to a set of critical functionalities. At the same time the 121T service is used to provide living documentation that can be relied upon to compare different versions of their application, to support development of other applications, and to support the Help Desk.

In another case, test cases developed through 121T service that originally were aimed at system testing, are now also used for smoke testing.

Other clients decided that to achieve their objectives they needed to improve the skill levels on topics like test design techniques and exploratory testing.

Conclusion

Defining a testing policy on the basis described in this post can be done for every kind of software development process where quality matters. Regardless on how it is carried out, whether through an agile approach or a more structured development process, a testing policy that answers the “Why” question is needed.

So, what is your testing policy?

In a future post I will discuss how to tackle the second important question: what kind of testing do we need to achieve the testing goals defined in a policy.

IDS SrL Part. IVA 02077400301

Privacy Policy – Cookie Policy

Via Linussio 51, 33100 – Udine Italy

Phone: +39 0432 629717

contact@designcoaching.net

THE MOST IMPORTANT TESTING QUESTION: WHY

The Why, What and How of testing

Setting up a testing process that supports the business

The testing policy

Conclusion

Follow us