Automated testing vs. manual testing

5 April 2019
Testing
AUSY is taking part in the French Software Testing Day on 9 April. During the event we will be offering feedback and advice on automatic testing and automation solutions, as well as the investment costs required for this new working method.

The changes to development methods brought about by agility and, more recently, DevOps prompts us to question testing processes. It is now necessary to test more i.e. in line with development iterations and test better i.e. to take into account the larger scope.

 

Automating tests makes it possible to limit execution costs and risks. Automation also reduces lead times thanks to faster execution and parallel testing.

DevOps and agility best practices also reduce the time needed to express the requirement (by the Business) and its subsequent operation. Therefore, automating tests and reducing their lead times contributes to an overall reduction in the delivery time. Moreover, these tests can be conducted outside business hours.

The overall application quality of the tests is improved thanks to a wider and better adapted functional coverage, tests at each delivery and a reduction in human error caused by repetitive tasks and poor interpretation. Testers can ‘switch off’ following endless rounds of testing but fortunately this can be avoided with automation. Testers’ time can be reallocated to high value-added tests (end-to-end tests, boundary tests, complex business tests, etc.).

All this generates gains for the project as a whole, especially as some automated scripts can be used by other teams during several other phases of the project. For example, development and integration teams can reuse automated scripts in integration tests, vitals tests and load tests. Production/pre-production teams can reuse them for their performance tests, load tests and validation tests. The more a script is reused the better the return on investment (ROI).

 

Test automation solution

AUSY recommends the RobotFramework solution for automating tests.

automatisation_test

 

This powerful and comprehensive solution, thanks to its ability to integrate third-party libraries, boasts an unrestricted development interface i.e. you can develop tests using several different tools (RIDE, Eclipse, etc.). Multi-platforms, RobotFramework has an architecture that can handle several different technologies in the same test (keywords, Python, Java, etc.). Moreover, its large community can help trouble shoot technical problems making it a well known and well used tool.

This automation solution can be easily integrated into testing tools such as Squash and JIRA. Automated test campaign reports and reviews (regression tests) can be automatically integrated into the global test repository.

 

Automation strategy

The benefit of automating a test is higher if it is repeated often i.e. when it is run for multiple deliveries, and in different environments with different databases.

Before automating a test, it is necessary to check the technical feasibility of the functional scope to be tested. The tester evaluates this during the test scoping and design phase. Reasons not to automate a test case can be linked to the business complexity of the case, specific development requirements or the pertinence of the functionality to be tested in production (volumetric, frequency, etc.).

Based on our feedback in terms of test automation, we would like to highlight the importance of test automation prerequisites.

The first point is to look at (prior to testing) the stability of the test environment. With a view to improving the efficiency of automated testing, a stable environment dedicated to each campaign will guarantee the integrity of the results. A “production copy” environment (pre-production) is increasingly being recommended for regression testing.

The second point refers to the reliability and stability of the testing data when constructing the test population. Reliable test data is necessary to guarantee the conformity of the test results obtained during regression test campaigns. A special database for automated tests guarantees the non-corruption of data by manual tests.

It is important to comply with the prerequisites for application developments:

  • Development teams must be aware of the automated tests carried out on the application they are developing.
  • Application coding best practices must be generalised and applied to everything.
  • The daily sharing of information between developers and testers is also essential to analyse the impact of application modifications on automation scripts.

 

AUSY also implements best practices on its automation projects e.g. mastering manual test processes before automating. These manual tests shed light on potential functional, technical or data issues.

AUSY also recommends having a full team to work on automation projects. An automation engineer will participate in the impact studies, develop and maintain scripts, run the automated test campaigns and ensure the technical functioning of the automation solution and test environments. The test pilot will organise, plan and track the progress of automation projects. He/she must have an in-depth knowledge of the challenges surrounding test automation, as well as making the most of efficiency indicators. Finally, functional and business experts can assist the test pilot in analysing the results and helping the automation engineer with the proper execution of the automated tests.

 

With a view to improving their level of maturity in this area, AUSY teams regularly learn about new test automation methods and tools. And because AUSY teams are constantly monitoring new technologies, as well as actively sharing information in this area, they contribute to improving the skills of all the teams involved in automation projects (customers, etc.).

 

Cost comparison between manual tests and automated tests

A key aspect to take into account when we want to estimate the cost of automatic tests is the implementation of an automation solution, its infrastructure, training costs and the initial implementation cost.

How is it possible to design a high value-added product while also reducing the time to market?

The costs covered in this calculation include direct and indirect costs. Direct costs include the cost of licences, automation tools, infrastructure, training the team, as well as specific costs for the automation engineer, the possible creation of a test framework, the creation of datasets and scripts, and their attendant maintenance. Indirect costs include undetected anomalies and managing false positives.

 

Below is an example of the calculation of costs for automating 100 test cases in a stable test scope:

Tableau comparatif de coûts J/H

  1. Calculating the cost of a manual test
  • Test design

100 test cases (20 simple cases, 60 average cases and 20 complex cases)

According to graphs based on feedback from AUSY projects, the cost of the design = 48 d/h

  • Conducting manual tests

Using the same assumptions, a test campaign run manually works out at 31 d/h

  • Test maintenance

The cost of updating the test design is estimated at 1 d/h per test campaign

  1. Calculating the cost of an automatic test
  • Test design

This involves creating automated scripts. Using the same assumptions as for manual tests, the cost of creating automated scripts is estimated at 168 d/h

  • Conducting automatic tests

The first automatic test campaign is evaluated at 12 d/h.

A test campaign executed automatically works out at 7 d/h as of the second execution

  • Maintaining automated scripts

The cost of updating automated scripts is estimated at 5 d/h per test campaign

 

Courbes de coûts tests manuels vs tests autosBased on these assumptions and calculations (but without taking into account installation and configuration costs), we estimate that we can achieve a cost equal to the cost of a manual test at the level of the 6th automated test campaign.

AUSY evaluates (based on feedback) the productivity gain at the end of the 12th campaign as 30%. If we carry out one test campaign per month, we obtain 30% gains at the end of the year.

 

In conclusion, in a world where projects are becoming increasingly flexible and agile, it is necessary to see automating tests in a stable environment as an investment that will result in better quality and a quicker time to market.

 

Don’t hesitate to check out our web page on: Software Tests.

Let's have a chat about your projects.

contact