Why Robot Framework?

Recently I’ve entered the new project. There was no testing, either manual or automatic, but there was a need in it. The customer has given me almost a carte blanche. Requirements for the testing toolkit was my own choice(except that it was Python). In this article, I will explain why the Robot Framework(RF) was the best choice in this situation. There will be some examples of tests created especially for the article which illustrates the case.

Robot Framework advantages

I think I can be considered a fan of the Robot Framework. It seems to me that almost everything can be done with it. In my opinion, pytest does not have such capabilities, since it is limited by the capabilities of Python. You just write Python code and state that you expect a certain value in a certain variable. It’s simple but not flexible. Robot Framework allows more. Here are four pros of the Robot Framework.

Flexibility

Robot Framework is written in Python so it can do anything Python can. It has a huge collection of libraries created by the community for it, which greatly expands the range of problems to be solved. With the help of the RF, you can easily write your libraries and integrate it with almost anything. It all will work right away. No wheel inventions are required. Moreover, the RF allows you to write parametric tests (templates), which greatly accelerates the work of the automation tool.

Unusual languages in tests

One of the basic points is that keywords (that’s how we call methods in RF) can be written in any language. As a result, the localized test texts become more understandable(yet, far from the literary language, ). First of all, this is needed by people who deal with business, not programming. They can independently open the tests and see what exactly happens there: “Select a random element from …”, etc.

Tagging

Robot Framework exploits tags. Not only you can run tests with a specific tag, but also analyze reports with their help. The log file contains statistics on tags, which can help you to interpret the situation and take needed actions.

I advise you to design your tag structure in correspondence with the project architecture. I have used a tree folder structure for the project itself. An example of this structure is shown in figure 1. There are 4 basic levels:

• 1-level. Folder of the project. No tags;
• 2-level. Folder of tested functionality. Tag the name of functionality;
• 3-level. Folder of test type. Tag the test type: smoke, CRUD, integration,…
• 4-level. Specific tests. Tag the test name or any informative tag.

Fig. 1. Project structure example.

Each of the folders can contain an “__init__.robot” file. There we can enable the specific tag(or several tags) for all nested folders and scripts. This can be done with code like this:

*** Settings ***
Force Tags Smoke

The rest of the tags are introduced in “*****tests.robot” files inside the test definitions. You can check tag syntax in the examples below.

It is also convenient to tag the type of return value of tested methods. It can also give additional information during the log analysis.

I have also bound autotests to Jira using tags. When the bug closes, the test from red turns green, we still have a history of what exactly has changed. Many months later, if the test turns red again, we can see what steps were taken to fix the problem last time, and what reasons stay behind the bug.

I have added another special tag for non-critical bugs. Gitlab does not allow us to do anything with the assembly if at least one test crashes. Thus, we can’t make a release if there are bugs. But if we have low priority and insignificant bugs? I selected a tag that allows RF not to reject the entire assembly if specific tests fail.

Excellent Log

Log analysis is a crucial part of testing. The Robot Framework records absolutely everything that happens during the tests run. It goes so far, that I had to write a special wrapper that hides the login and password from the database connection log.

Such detailing helps to understand the origin of the bug. Was it a tester or developer bug? It’s not always obvious. The errors of developers and testers are distributed equally.

In other tools (pytest) you also can create such detailed logs. But then you have to code them by yourself. It’s not always easy to design an informative and effective log by yourself.

Practice

All examples are designed specifically for the article to illustrate the ideas presented above. If you want to experiment with these examples yourself, they are all in our repository: https://github.com/maxilect/robot-framework-examples-en.

Example of the simplest test

Let’s start with the simplest test on the Robot Framework.

In the example below, a session with a link to the accessible resource is created (in our case, https://en.wikipedia.org/wiki). By invoking the Get request for root (/), we check the status code 200.

*** Settings ***
Documentation Example of smoke-autotest.
Library RequestsLibrary
*** Variables ***
${base_url} https://en.wikipedia.org/wiki
${url} /
*** Test Cases ***
Check accessibility Wiki
Create session conn ${base_url} disable_warnings=1
${response} Get request conn ${url}
Delete all sessions
Should be equal ${response.status_code} ${200}

Other language test

We can use other languages to create RF methods. Here is the same example as above.

*** Settings ***
Documentation Example of smoke-autotest (Russian).
Library RequestsLibrary
*** Variables ***
${base_url} https://en.wikipedia.org/wiki
${url} /
*** Test Cases ***
Проверить доступность Wiki
Create session conn ${base_url} disable_warnings=1
${response} Get request conn ${url}
Delete all sessions
Should be equal ${response.status_code} ${200}

Templates (parametric) tests

Robot Framework allows you to create test patterns. If we transfer the test from the previous example to the template view, we can easily scale it by adding calls with the necessary arguments. For example, let’s check the answer 200 for pages with biographies of scientists.

*** Settings ***
Documentation Example of smoke-autotest with template.
… Tests with templates are scaled easily.
Library RequestsLibrary
Test Setup Create connection
Test Teardown Close all connections
Test Template Smoke-test
*** Variables ***
${base_url} https://en.wikipedia.org/wiki
*** Test Cases ***
Check accessibility of Isaac Newton page
/Isaac_Newton
Check accessibility of Albert Einstein page
/Albert_Einstein
Check accessibility of Stephen Hawking page
/Stephen_Hawking
*** Keywords ***
Create connection
Create session conn ${base_url} disable_warnings=1
Close all connections
Delete all sessions
Smoke-test
[Arguments] ${url}
${response} Get request conn ${url}
Should be equal ${response.status_code} ${200}

Tagging and logging

We continue to improve our simple test.

Having a template as a single entry point, we can easily add verification of the scientist’s year of birth. This way we confirm that the loaded page displays the correct data.

Besides, in the example below, I wrapped the existing keyword in Russian names, just for the sake of practice.

*** Settings ***
Documentation Example of smoke-autotest with template.
… Tests with templates are scaled easily.
… Tagging added including non-critical tags
… Russian language keywords
Library RequestsLibrary
Test Setup Создать соединение
Test Teardown Закрыть все соединения
Test Template Smoke-тест
*** Variables ***
${base_url} https://en.wikipedia.org/wiki
*** Test Cases ***
Check accessibility of Isaac Newton page
[Tags] Newton
/Isaac_Newton 1642
Check accessibility of Albert Einstein page
[Tags] Einstein
/Albert_Einstein 1879
Check accessibility of Stephen Hawking page
[Tags] Hawking
/Stephen_Hawking 1942
Check accessibility of non-existent page (to display error)
[Tags] Numbers
/123456789 1899
*** Keywords ***
Создать соединение
Create session conn ${base_url} disable_warnings=1
Закрыть все соединения
Delete all sessions
Smoke-тест
[Arguments] ${url} ${expected_word}
${response} Get request conn ${url}
Should be equal ${response.status_code} ${200}
… msg=Response code != 200 ОК during GET ${url}.
Check word on page ${response.text} ${expected_word}
Check word on page
[Arguments] ${text} ${expected_word}
Should contain ${text} ${expected_word} msg=Can’t find the word ${expected_word}!

Pay attention to [Tags]. You can add tags here, which, as I wrote above, will help to assess problems at the report level. Similarly to the Force Tags in the __init __. Robot file (see the example in our repository: https://github.com/maxilect/robot-framework-examples) allows you to set tags for all tests in the directory, including nested ones . If the tagging is performed correctly, without even reading the names of the tests themselves and without crawling into their logic, we can fairly accurately assume that it does not work in the test project.

Look at the report on the launch of these tests. For clarity, I added a test that will result in a fail. Report statistics are its most important part.

In our example, the tests with the “numbers” tag did not pass completely (we have 1 out of 1, but in real life there will be, for example, 17 out of 20). Thus, It can be assumed that the problem is in this page.

Tags help to run tests selectively. To run all tests with a specific tag in the launch line, you must specify:

 — include <tag>

Even logical operations with tags are supported:

- include <tag> AND <tag>

For example, if you want to run only the Smoke test for tests with the numbers tag, you should add:

 — include smoke AND numbers

Non-critical tests

Let’s do the trick that simplifies the work greatly.

You can define one or more bugs as “non-critical” with tags. A test with such a tag will still show errors (if any), but in the end, these errors will not be interpreted as “critical”. I use this option when some minor bugs are taken into account, entered into the bug tracker, but have not yet been fixed. Without it, the autotests will not allow you to assemble the project, and this is not always convenient.

Check accessibility of non-existent page (to display non-critical errors)
[Tags] Letters Known
/abcdefghi 1799

At startup, we define the added tag as “non-critical” using the key:

 — noncritical Known

Keyword in python

With the following example I will illustrate how to create your own library. Create a new keyword “Generate an array of numbers”.

from random import randint
from typing import List
from robot.api.deco import keyword
class ArrayGeneratorLibrary:
ROBOT_LIBRARY_SCOPE = ‘GLOBAL’
@keyword(“Generate numbers array”)
def generate_array(self, length: int, minimal: int, maximal: int) -> List[int]:
result = []
for i in range(int(length)):
result.append(randint(int(minimal), int(maximal)))
return result

Connect the library to the test:

Library libraries.ArrayGeneratorLibrary

and use it:

Example of python lib usage.
${array} Generate numbers array ${5} ${2} ${8}
Log to console ${array}

Do not forget that if you have a nested structure, you should separate it with dots, as in the example.

Eszee little trick: numbers passed through the ${} are treated as int, and not as a string!

Built-in arguments

Another nice thing is built-in arguments, which are passed not at the end of the call, as usual, but directly in its body. To illustrate, we write a wrapper for the array generator created above that allows using the built-in arguments:

Generate ${n} numbers, from ${from} to ${to}
${result} Generate numbers array ${n} ${from} ${to}
[Return] ${result}

Now you can use it:

Method wrapper with nested parameters.
${array} Generate 5 numbers, from 2 to 8
Log to console ${array}

Substitution of the method name part, python insertion and loops

The next trick that I really like about the Robot Framework is the substitution of the name part. Suppose we have two methods: one selects even numbers, the other odd.

Find even numbers in list
[Arguments] ${list}
${evens} Evaluate [i for i in $list if i % 2 == 0]
[Return] ${evens}
Find odd numbers in list
[Arguments] ${list}
${odds} Evaluate [i for i in $list if i % 2 != 0]
[Return] ${odds}

The keyword Evaluate used above allows you to execute a line of python code inline. Please note, if you do not want to replace a piece of the string with the contents of the variable, namely, pass a link to it, then you must specify the variable name immediately after the $ sign without curly braces!

You can call both methods by changing the different part of their name:

Pass the part of the method name as parameter. Loop.
${types} Create list even odd
${array} Generate 5 numbers, from 12 to 28
FOR ${type} IN @{types}
${numbers} Run keyword Find ${type} numbers in list ${array}
log to console ${numbers}
END

Method Decorators

Yes, the Robot Framework allows you to write a decorator for other keywords!

To illustrate this feature, we write a decorator that selects negative numbers in the response of any method that returns a list.

Find negatives in the response by calling
[Arguments] ${keyword} @{args} &{kwargs}
${list} Run keyword ${keyword} @{args} &{kwargs}
${negs} Evaluate [i for i in $list if i < 0]
[Return] ${negs}

Note:

• @ {args} are all unnamed arguments;
• & {kwargs} are all named arguments.

Having this bunch, you can redirect them, actually creating a decorator. You can use it this way:

Decorator example
${negs} Find negatives in the response by calling Generate numbers array 10 -5 5
log to console ${negs}

In the examples, I showed the main Robot Framework features that make work easier. But its convenient features are not limited to this list.

The author of the article: Vladimir Vasyaev, 13.5 year in test automation.