Autotests are important. But approaches and quality can differ dramatically. We came to a project that already had autotests. We were able to improve coverage and speed up the passing of tests without a fundamental code rewriting. Here is our story.
A few words about the project
Although we cannot reveal the details of the project due to the NDA, the task looked like this in general terms. We got involved in developing a fintech API service that interacted with the database, returning the necessary financial objects (prices, tariffs, etc.). Our task was to test mobile clients for this service — both web and native mobile applications.
Test automation on this project developed gradually, along with the increasing complexity of the service. This is probably how “end-to-end tests” appeared, which we found in the project legacy. Most of them were no longer working by that time, because the service had changed, and there was no one to support the tests. The only Automator left the project long before our appearance.
Even those tests that seemed to correspond to the functionality sometimes failed due to confusion with versions or inaccessibility of external resources. A separate infrastructure was used for testing — test benches, where the necessary versions were deployed. Different teams had access to infrastructure, and they were not always coordinated well. Lack of coordination resulted in failures of APIs used in testing. After that, even working tests stopped working. The tests no longer showed the workability of the service itself, but rather showed problems of the test infrastructure.
How we approached chaos
It would seem that it is necessary to abandon all the old developments and build testing anew in this situation. We have chosen another way. The structure of testing itself was retained, focusing on solving specific problems — slow test passage, instability, insufficient test case coverage. For each of them, there was a solution.
Refactoring
First of all, we partially reworked the code of old tests, relying on more modern design patterns.
Part of the legacy code had to be removed — it was too difficult to maintain. In another code part, we caught all the weaknesses. We replaced sleep functions with normal waiters, brought preparation for all tests to the global setup through test runner annotations, etc. A lot of small steps allowed us to reduce the passage of the average end-to-end test from 3–4 to 1–2 minutes.
Atomic Approach
To speed up the creation of new tests and simplify the maintenance of old ones, we have moved away from end-to-end cases.
I have nothing against end-to-end testing, but when you need to check one specific screen (or even some of the information on it), going through all the stages, starting with user authorization, is too expensive. Imagine that we are testing an online store, and we only need to check a receipt that will be sent to a customer after purchasing a specific product. Instead of fetching only one screen from the system, we would log in by login and password, select an item, confirm a purchase, etc. In this approach, we perform steps unrelated to a specific testing task. But every action takes time. Even with all the optimization done, running an end-to-end test is 2 minutes while checking a particular screen is only 10 seconds. Therefore, where it was possible, we switched to such “atomic” checks, referring only to the screen that interests us within the test case.
Along the way, just to compare screens, we implemented snapshot-testing, which allows you to check most of the UI. Having tests and application code in one repository, we can use the methods of this application in tests, i.e., raise any screens that are needed in the process. Thus, we can find errors when comparing test screenshots with reference ones.
Now we have about 300 snapshot tests, and their number is gradually growing since this approach can significantly reduce the time for checking the final production version. This entire suite of tests runs automatically when a pull request is opened and runs 40 minutes. The developers quickly get feedback on issues in the current branch.
Of course, some end-to-end tests have survived. They are indispensable wherever large business scenarios are required to be verified, but it makes sense to run them when all the details have already been verified.
Mocking
To eliminate the influence of an unstable test bench on the result of our tests, we launched a mock server. We have chosen Okhttp mockwebserver. We have an article on this topic.
As a result, the share of tests that occasionally fails due to external causes has significantly decreased.
Kotlin DSL
In parallel, we made the tests more readable.
Those who do UI testing know how difficult it is to analyze a bunch of locators in a long test log (especially when it is end-to-end tests). They are easy to navigate when you have been on the project for two years, and even in the middle of the night can remember which one. But if you have just come, it is a separate big task to understand what is going on. To prevent new people from having these problems, we decided to switch to Kotlin DSL. It has a simple and understandable structure. If earlier tests consisted of a set of the same low-level calls — clicks, text input, scrolling, now all this has turned into something more “business” — something like a BDD approach. Everything is visible and understandable.
In my opinion, we have made a certain reserve for the future. This project has once already faced the departure of the single automation engineer. This departure did not end in the best way. Tests simply ceased to be supported, since the entry threshold turned out to be too high. It took a lot of time and a specific skill to understand such code. We remade the tests in such a way that it will be possible to quickly transfer people from other projects or manual testing to automation at any time. Almost anyone can write the simplest tests on Kotlin DSL. The Automators can avoid low-level code, and exploit people from the functional team to write new simple tests quickly. They have enough knowledge of business logic, and the project will only benefit from the fact that the functional team will be more involved in writing autotests. Kotlin DSL allows you to describe test cases precisely how they would like to see all the checks, leaving the low-level implementation of methods outside of their work.
In general, all this made it possible to increase the coverage with autotests faster. Earlier it took 16–20 hours to implement a new test suite, then with a new approach, depending on the complexity of the tests, it takes from 4 to 12 hours (and the labor costs for support have been reduced from 16–24 to 8–12 hours per week).
Article author: Ruslan Abdulin
PS. Subscribe to our social networks: Twitter, Telegram, FB to learn about our publications and Maxilect news.
