Robot Framework vs. Pytest
I am an active supporter of the Robot Framework. It can be used to solve almost any test automation problem, especially when development is in Python. However, our company also uses Pytest on various projects. I had to learn this tool from A to Z. Now, I’m ready to make a full comparison between Pytest and Robot Framework, of course, my subjective opinion.
The core of testing tools
Pytest is an implementation of the xUnit framework for Python. Suppose you have worked with jUnit or nUnit all your life (the two most common representatives of this family for Java and .NET, respectively), then in Pytest. In that case, you will find the same standards — you will quickly understand what is happening, you will follow the usual approaches in writing unit tests. xUnit frameworks are add-ons over a programming language convenient for developers to use. These frameworks are quite common and widespread.
It is necessary to use a log generation tool and Pytest(also with many xUnit frameworks). Allure has traditionally been used in this role. Probably, today it is the standard for autotest logs that business wants to see. However, you need to understand that one community develops Pytest, and another develops Allure. The vectors of these teams may be different at some point.
Unlike Pytest, Robot Framework is a domain specific language (DSL). It’s not Python, although it’s based on it. In Python, you can write anything you want for the Robot Framework. All the features I’m talking about: integrations, settings, etc. are available immediately.
Robot Framework is not that convenient for developers, as it requires a deeper dive. This is a different approach to autotests, like Cucumber for Java. The Robot Framework independently generates detailed reports (we will talk about this later), i.e., it also contains a layer for generating reports. The product is developed by one community — indivisibly and harmoniously.
By the way, the community is open and quite responsive. There is an open channel in Slack, where anyone can ask questions about the Robot Framework and get an answer. I sometimes write there. I even contribute to Robot Framework sometimes.
Pros of the Robot Framework
No unnecessary naming restrictions
In Pytest, a test case must always start with a “test.” It seems logical to me when the test method is used. Otherwise, naming restrictions don’t help the process.
I suppose that these limitations in Pytest came from a desire to separate what is related to test cases and what concerns common methods called from tests. Since Pytest is designed for unit testing, the application’s functioning and test methods can be in the same class, so there can be confusion.
However, in my opinion, the problem here is precisely the chosen approach. In the 21st century, we have to put some restrictions on test cases’ names. Absurd.
Robot Framework is not in any way designed for unit tests. It has a keywords section, and the tests themselves can be called in any way. It seems to be more convenient. Moreover, in the Robot Framework, any entities, including keywords, can be called in Russian(for instance). Having chosen the names correctly, the test’s text can be turned into a story (“Open the site, enter your login, make sure that the result is …”). This allows you to write short comments about what is happening. Any development contributor or manager who has never dug into the depths of testing can open this test and understand it. It saves time.
Sometimes happens that a set of test cases (test suite) requires the same type of settings. For instance, there is a method that prepares some data, creates entities, and stores their identifiers for tests. The Robot Framework provides separate suite setup settings that apply to all tests in a suite. There is a different “suite teardown” that runs after all the tests in the suite and similar settings for each test case (test setup and test teardown). It is convenient, logical, and does not require the invention of the wheel.
The xUnit frameworks have annotations that override the suite setup, while Pytest has fixtures with scope =” class”.
In Pytest, test cases can be just methods (and then they have no suite setup — i.e., no single preparation). If we need uniform preparation, we can wrap these methods in a class. But if we make a fixture with scope =” class” for this class (ie, try to implement suite setup), we will get a separate instance of the class for each test so that the data from the suite setup will not get into the test cases. The individual instances are probably created assuming that data from different test cases should not interfere with each other. Thus, setting up an environment for running tests is much more complicated than in the Robot Framework, where a suite setup is provided a priori.
Usually, this issue in Pytest has to be bypassed by creating a separate fixture where the data is loaded. This fixture is subsequently pulled into the tests. Another way is to use Python tools by creating a static field for the class that will be common to all its instances (self .__ class __. test_id = 2). But in my opinion, you should not access the fields of a class through an underscore from outside.
As I noted above, Allure is most often used to generate logs in Pytest. It is beautiful and fashionable. But due to the feature described above, Allure does not understand how it is related to the test. The report does not include actions from the suite setup. We have to work around this by writing handlers. From my point of view, this is inconvenient.
By the way, other xUnit frameworks have this problem too.
The Robot Framework logs are as detailed as possible. The Robot logs everything you do. This log makes it much easier to catch floating errors that cannot be easily reproduced. You know exactly where and what value the variable had, what API was called. Often you don’t even need to restart the test to understand what’s going on. For Pytest, in such complex cases, you have to develop tools that help generate a log, like in the Robot Framework.
In the Robot Framework, there is no need to invent complicated constructs. Some expressions save you from unnecessary code.
I like to write keyword wrappers and wrap other keywords in them. For example, a keyword takes an ID from the API response. You can pass keywords that initiate different APIs (if the company has coding standards, then the API responses will be similar — the ID field will most likely be in all).
For example, you can write: “Id from entity creation.” Here “Id from” is a wrapper keyword, “entity creation” is a keyword that calls the API and gives the entire response. Alternatively, you can write “Identifier from table creation,” where “table creation” is another keyword.
In the header of the log, Robot Framework shows statistics for each tag. If you set your tags right from the start, you only need to look at these statistics to see the problem without bothering yourself with what’s going on inside. Tags also can be bound to bug IDs in Jira or in any other tracker that you use. I recently had a case when on one of the supported tests, which I forgot already, old forgotten bugs tags “worked”. It took several seconds to find them, instead of minutes and hours.
There is tagging in Pytest, but it is not logged. If you need a log, you do it yourself. Therefore, as far as I know, almost no one does this.
By the way, Allure does not provide the ability to display statistics on tags. Probably, if such an opportunity appeared in it, in my eyes, it would bring the Pytest + Allure bundle closer to the Robot Framework in functionality.
Pros of the Pytest
There are only two arguments for Pytest on my list. But from a supporter of another instrument, they should sound solid.
Unfortunately, you cannot pause execution in the Robot Framework to see the values of variables. This fact can be considered as the Pytest advantage.
However, in my opinion, this is not necessary. Everything that could be viewed in this way is available in the logs.
Pytest Parametric Tests
I found something that the Robot Framework does not have. Thanks to parameterization, Pytest can create test cases on the fly. In Robot Framework, ten tests will always be ten tests, no more, no less. In Pytest, in some cases, a single parametric test can cover a huge number of cases.
Suppose we have an API with two parameters, each of which can take several values (for example, one takes seven values, the other 10). In accordance with the theory of testing, in this case, it is necessary to select several cases that more or less uniformly cover the grid of 70 “intersections” (pairwise method). I’ve used the product method from the itertools module that prepared 70 data combinations, and then made the exhaustive testing. When another option appears in the initial data, I just need to add a line to one of the initial lists.
This cannot be done in the Robot Framework. There you have to write a test pattern that takes two values and then make 70 calls. As new values appear, the number of calls will have to increase.
The author of the article: Vladimir Vasyaev