Measuring Testing Times
Testing improvements
What Is A Slow Test?
There is no simple way to define a slow test, since it depends on the type of test, the complexity of the project, what are you testing, and many factors. But you can define time ranges to work on and know if a test is slow compared to the whole (or total time).
There are multiple types of tests but in mobile, there are 3 main ones: unit tests, integration tests, and snapshots tests.
Unit tests
Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the “unit”) meets its design and behaves as intended. https://en.wikipedia.org/wiki/Unit_testing
These tests are the fastest because they only check the logic of a part of the code, dependencies are mocked, they don’t need UI, etc.
If we have a small project in which the tests last half a second each, we might not give it relevant, but when the number of tests increases because the project is larger and more complex, this time value is not scalable, so we could say that a unit test that lasts more than half a second is slow.
I do not mean that half a second is an optimal time for a unit test, on the contrary, it is still a long time, but it serves as a concept to define the slowness of a test.
Integration tests
Integration testing is the phase in software testing in which individual software modules are combined and tested as a group. https://en.wikipedia.org/wiki/Integration_testing
These are the longest time consuming tests as they verify if the flow of the app works correctly. This flow can be of variable duration since it will depend on what you want to test, so it is not possible to define an optimal time range.
Snapshot tests
These are similar to unit tests but instead of checking the logic, they verify if the visuals are matching the reference images of that view. These tests are also fast as the unit tests as it is a unit test that loads a view. In this case, we could also define a time range for the execution of these tests.
How to identify slow tests?
In this article, we are going to analyse the unit and snapshot test times we have in the XING app in the iOS project.
To know how to make such an analysis, let’s use the Pareto rule: approximately 80% of the consequences come from 20% of the causes.
We are going to apply this rule to the results obtained in the xcresult file when launching the tests of our project. This file shows us a lot of information, but we are going to keep the information of the targets launched in the tests, the test classes of each target, the tests of each class, and how long each test has taken.
With this information of the times and ordering it from those that take the longest to those that take the least, we can make a graph with the Pareto rule:
In both graphs, we can see that the tests that take the longest, represent more than 50% of the total time spent running all the tests even if they are a minimum number of tests. Thanks to this graph we have been able to identify which are the slowest and now we can focus on improving the times of these tests and thus spend less time launching them locally or in the CI.
Analysing the snapshot tests a little more in detail, we have seen that the total duration of the execution of the tests is approximately 1 minute, and looking at the sum of the tests that last the longest we see that they are 33 seconds, more than half the time, so if we improve those 6 tests, we could reduce by half the execution times of the capture tests.
As explained above, the duration of integration tests depends on what is being tested. But if we find tests that last much longer than the average of the others, we could analyse if smaller tests can be done to test in parts of this large flow.
Conclusion
The ideal test execution time representation would be a diagonal line, which would mean that all tests take the same time to run and there is no time accumulated by slow tests. Since the ideal never exists, it would be best to get a line as close to that linear behaviour as possible.
