Regression is a major topic in Agile testing. Its importance is major because in Agile we build our product in an iterative way, and it is essential to ensure that the addition of a feature has not negatively impacted the existing. To evaluate this impact, we have a tool: regression.
This frequent regression is the source of many changes in practices. I am thinking in particular of:
- The need for automation: to save time on execution
- Exploratory testing: to test more in less time for non-reused tests, but also to fight against the
- Pesticide paradox
- DevOps and continuous testing: to make the modification process more fluid
- Shift right: and production testing to compensate for less manual test coverage
- BDD: to have tests that can integrate regression even before the development of the
In short, there is no sustainable Agile product without a good regression campaign. But how do we ensure that this campaign is “good”? What should a regression campaign contain? To answer this, I invite you first to look at the role of a regression campaign?
Why do we need a regression campaign ?
A regression campaign has many roles. Its main purpose is to detect major regressions on the product under construction. It is obviously impossible to detect all regressions (exhaustive testing is impossible) or even to test a new feature as thoroughly as an already implemented feature for budget and time reasons.
The concept of “major regression” is therefore everything, and I propose to return to it quickly:
- First, “regressions” must be detected, i.e. the behavior must be the same between the previous and the new version. There is, on paper, no need to detect anomalies already present. In 2019, I attended a tutorial on AI and the Oracle problem for my first JFTL. The presenter considered that regression was simple to design because we had an oracle: production! And that, in the end, it was a “version control“. I really like this view because it limits the purpose of regression and allows you to focus on the essentials.
- The regression must be “major“, i.e. judged to have sufficient impact to cause real discomfort for our users or on the use of the product… which often means, in a practical way, detecting anomalies that we want to correct rather than sending them to production. With these points in mind, it is easier to imagine what this type of campaign should look like.
The regression campaign content
For a regression to be major (and here it is highly context dependent) it must have a significant impact or occur frequently enough to cause real discomfort. It therefore seems particularly important, according to the second point, that this regression accurately represents the uses. A regression on a scenario that is never used by users and has no non-functional impact is potentially of little value. This is the subject of Bruno Legeard’s fine article on the testing of uses defined thanks to usage logs. In this article he is using as example Gravity which answer totally to that point of view. That lead us to this question: is a “major regression” on a path that is not used, really major ?
There is no easy answer because it depends of the context, however we can say that a regression must therefore propose frequent user paths… but also that the content of only frequent user paths is not enough! Even if these tests can/should present the great majority of the tests of a regression campaign, it is essential to go further with the regression… and this is why Bruno indicates that the gravity tool does not totally lighten the workload on the regression but 80%… Which is already huge because it can help to drastically reduce the cost of maintenance! Besides, it also can help to initiate a regression campaign if there is no current one or if it is deprecated. You can create a free Gravity account to try by yourself :
What about the 20% left?
The remaining 20% of the effort is dedicated to what is often referred to as “non-passing” cases or non-functional tests such as performance and security. Indeed, security tests are necessary to avoid intrusion, data theft or identity theft… but attacks are not as frequent as the flow of ordinary users. Similarly, a drop in performance will not change the results of the usability tests, but it will greatly deteriorate the experience of our users. A regression campaign must therefore propose complementary tests to the uses and these tests must be in line with the different risks identified… these risks are often linked to non-functional issues.
The regression campaign is a major element of quality during the development of a product in Agile. It is therefore necessary to have a “good” regression campaign in order to have, over time, a quality product. To achieve this, it is obviously necessary to have tests that are close to what is used in order to avoid the illusion of no errors, but unfortunately this is not enough; it is also necessary to integrate more specific tests that are often linked to non-functional issues (security, performance, robustness, etc.)
Find this article in French in the blog of “La Taverne du Testeur“