In a constantly changing society, companies must be able to improve the efficiency and quality of their solutions, in order to make them ever more efficient and effective. The software delivery cycle must be smooth, continuous and fast in order to meet market expectations and requirements. Implementing automated tests to ensure the best quality and to detect the maximum number of anomalies in the shortest possible time seems obvious.
why automate?
the benefits of automation.
The implementation of automated tests improves software quality, increases test coverage while testing vital functionalities on a regular basis, or even continuously. The key benefit of test automation is also productivity. This approach makes it possible to test more frequently and to report operating anomalies as soon as possible. This rapid detection of anomalies allows development teams to greatly reduce the time required to correct them. This means an increase in the number of tests performed and their frequency of execution, which will contribute to improving the quality and reliability of the product.
Another added value of test automation is that it can contribute to reducing the time to market, i.e. delivering more frequently, putting more and more functionality into production and tending towards a virtually zero rate of malfunctions in production. Which, of course, contributes to customer satisfaction. By discovering an anomaly as soon as possible, the risk incurred during production is limited.
Freed from this recurring task, functional testers can concentrate their efforts on activities with greater added value (exploratory tests, BDD/ATDD, etc.). The benefits of automating functional tests are therefore on several levels:
- Application reliability and customer satisfaction
- Increased productivity
- Reduction in time to market
- Availability of testers for higher added value activities
constraints related to automation.
In order to control the reliability of automated tests, test execution must take place within a stable and controlled framework. The effectiveness of the tests is closely linked to the stability of the environment in which they are run. Another area of focus is the reliability and relevance of test data. The reliability of this data is necessary to guarantee the conformity of the test results but also their management over time. Before embarking on the automation project, it is essential to assess the technical feasibility of the functional scope to be automated.
hidden costs.
Beyond these constraints, there are also some hidden costs that need to be considered, mainly related to the complexity of the test cases, the volume of test cases and their associated maintainability, as well as the frequency of test case execution. The relevance of the choice of test cases to be automated, linked to the length of time the tests are run, also has an indirect impact. Another hidden cost is in the choice of the controller, in conjunction with the profile of the testers who will develop the automated tests. If functional profiles are envisaged, the choice of a tool that is too technical will be a hindrance to the development of skills and to broader automation.
But then, is there any real benefit to automating functional testing? Should as many test cases as possible be automated with a significant cost for the maintenance of functional automated tests? The answer is a double yes! But for all the reasons mentioned above, an appropriate automated testing strategy must be designed that takes these risks and constraints into account.
what is the right strategy for automated testing?
approach.
It is important to have a testing strategy in place that is aligned with the project priorities and that creates value at each iteration of the application lifecycle. The test automation strategy for functional testing should therefore be an integral part of the test strategy and should allow for in-depth testing of the critical requirements and paths of an application. Test automation should not be seen as a side project, but part of the project and therefore of the overall test strategy. It must be shared and validated by all those involved in the project (business, developers and testers).
A recommended approach is to start from the business and user vision in order to better understand the critical paths of the application and choose the priorities in the testing effort, whether manual or automated. This approach allows for ambition while prioritising automated testing activities and quickly creating value and trust. This allows for a gradual increase in skills, while gradually raising automation objectives.
It is also essential to correlate this strategy with ISTQB best practice and the distribution of tests in the test pyramid, as opposed to the ice-cream cone view of all manual testing.
To do this, the test strategy must be shared and co-constructed with the project stakeholders in order to be as effective and relevant as possible. The priorities and the risks are therefore shared and known by all those involved in the project. The approach to test automation can therefore be illustrated as follows:
To do this, it is essential to make collaboration between the different parties involved in a project simple and effective, in order to facilitate and accelerate decision-making. Visual management and in particular business modelling is a preliminary step, which allows collaboration between businesses, developers and testers. While providing a macro vision of business applications, on the one hand and, on the other hand, critical application paths that rethink the user experience. This tool establishes a common language and documentation that is alive and working.
Test-oriented modelling follows in a second stage, to deepen the business modelling. It allows collaboration between the same parties involved and enhances living documentation. It is during this modelling phase that the project team views and refines its testing strategy. It can serve the current vision of the project (test coverage, strategy, reference communication) or its target vision (impact and evolution on tests, identification of gaps, needs in environments and data).
During the scripting stage, the automation specialists rely on the modelling phases in order to have a clear vision. Typically, automation focuses first on core functionality, vital end-to-end paths and passing cases, and then on error cases, edge cases and important functionality.
why modelling.
Modelling is a collaborative method, which enables business processes to be represented visually and quickly using a common vocabulary. When different parties work together on a project, barriers arise between them, which slow down the progress of the project. Indeed, developers do not use the same vocabulary as testers, who do not use the same as product owners, who do not use the same terms as other professions. Modelling allows all the parties involved to have a global view of the project and to visualise all the interactions between the different internal and external systems. It also allows newcomers to learn quickly, thanks to the simplicity of reading and understanding the project. Finally, it offers advantages and significant time savings to the various parties involved, which we detail below.
To answer this problem, the use of the Business Process Model Notation method (BPMN, ISO/IEC 19510 standard) facilitates the sharing of information in a fast, fluid and understandable way, because each party involved has the same level of visual information. This results in significant time savings.
best and bad practices
bad practice 1 - want to automate everything.
One of the beliefs about automation is to think that everything can be automated and that everything should be automated. This practice risks generating unreliability over time and very high maintenance costs and will lead to a loss of trust in the reliability of automated tests over time.
Tip: target the tests and test paths to be automated based on modelling.
bad practice 2 - automate as for manual testing
Another belief is to want to automate in the same way as manual testing, with long functional tests and a lot of checks at each test stage. One of the consequences of this belief is that maintenance over time is very difficult and automated tests are systematically abandoned.
Tip: Create test cases on a feature with a minimum of validation objectives, or on the behaviour of a user journey with minimum checks sufficient to validate the test case.
bad practice 3 - test everything with the ui.
As a tester, the end-to-end view starting from the UI can suggest that automation should mainly be done with the UI. However, in many project contexts, this means that the test cases are too complex to automate or, where possible, the test scripts are complex and unstable due to the regular changes to the screens and the addition of new pages.
Tip: In this case, focus on API test cases with a lower level automated testing approach, inspired by the test pyramid, if possible, and include a minimum of automated UI testing only on critical cases, if possible.
bad practice 4 - automation, a special project.
In order to accelerate automation, the idea would be to set up an autonomous “Test Automation” project in parallel with the team concerned, in order to have a better transformation. In this case, two risks emerge: the first is to obtain highly technical automated test cases that are not in line with the business values, and difficult to understand and maintain by the test team once the “auto test” project has been completed and passed on to the test team. The second risk is that too many automated tests are produced, particularly on the end-to-end side (see no. 1).
Tip: The automated testing strategy should be included in the test strategy and supported by those involved in the project. It should guide the development of automated tests. Based on the approach described above, test automation becomes efficient and is fully integrated into an overall test strategy.
conclusion.
All too often, projects embark fully on automating their functional tests without first developing a strategy for automated functional testing and without prioritising the actions and identifying the test cases that will provide the most value. Thinking they are saving time, project teams end up wasting a lot of it by having to deal one by one with problems that could have been avoided. Visual management, in particular the modelling of business paths, is a powerful and effective tool which makes it possible to provide a good understanding of the project to all those involved, as well as a good visibility of the modifications and developments of the project and the resulting impacts. This allows you to adapt your test strategy, to make the right decisions in a collaborative way and to refine the tests to be automated which should be kept, maintained or set aside. This greatly facilitates the maintainability of the automated and non-automated test repository.