OpenApiTools is a set of libraries centered around the OpenAPI Specification

OpenApiLibGen for Robot Framework

OpenApiLibGen is a command-line tool that can be used to generate a Robot Framework library based on an OpenAPI document.

In a (virtual) environment where robotframework-openapitools is installed the generate-library shell command is availabe:


$ generate-library -h
usage: openapi_libgen [-h] [-s SOURCE] [-d DESTINATION] [-n NAME] [--recursion-limit RECURSION_LIMIT]
                      [--recursion-default RECURSION_DEFAULT]

The OpenApiTools library generator

options:
  -h, --help            show this help message and exit
  -s SOURCE, --source SOURCE
  -d DESTINATION, --destination DESTINATION
  -n NAME, --name NAME
  --recursion-limit RECURSION_LIMIT
  --recursion-default RECURSION_DEFAULT

Inspired by roboswag. Thank you Bartlomiej and Mateusz for your work!

Generation of your library is an interactive process with a few steps:


$ generate-library
Please provide a source for the generation:
$ http://127.0.0.1:8000/openapi.json
Please provide a path to where the library will be generated:
$ ./generated
Please provide a name for the library [default: FastAPI]:
$ My Generated Library
generated/MyGeneratedLibrary/__init__.py created
generated/MyGeneratedLibrary/my_generated_library.py created

Tip: The name for the library that you provide (or that's taken from the OpenAPI document) will be transformed into a Python-legal module name and a Python-legal class name. This means special characters like - will be removed or replaced. If you used spaces in the desired name, they will be removed from the Library (class) name and the next character will be capitalized while the spaces will be replaced with an underscore in the module name as can be seen in the example above.


Note: There's currently 2 environment variables that are taken into account by the generator:
    USE_SUMMARY_AS_KEYWORD_NAME can result in nicer keyword names (but: uniqueness is not guaranteed, so you might need to rename some of the generated keywords manually)
    EXPAND_BODY_ARGUMENTS changes how body arguments are provided to keywords (either a single body dict or key-value pairs for the body parameters)

If the location where the library is located is in your Python search path, you'll be able to use it like a regular Robot Framework library (in fact, it's a drop-in replacement for OpenApiLibCore). The generated library has a keyword for every endpoint in the OpenAPI document used to generated it. In addition, all the keywords from OpenApiLibCore are available.

Keyword documentation for OpenApiLibCore can be found here.

The generated library can be used as shown below:


*** Settings ***
Library         MyGeneratedLibrary
...                 source=${ORIGIN}/openapi.json
...                 origin=${ORIGIN}
...                 base_path=${EMPTY}
...                 mappings_path=${ROOT}/tests/user_implemented/custom_user_mappings.py
Variables       ${ROOT}/tests/variables.py


*** Variables ***
${ORIGIN}=      http://localhost:8000


*** Test Cases ***
Test Generated Keywords: Get Employees
    ${response}=    Get Employees
    Should Be Equal As Integers    ${response.status_code}    200

Test Generated Keywords: Post Employee
    VAR    &{body}    name=Robin the Robot
    ${response}=    Post Employee    body=${body}
    # ${response}=    Post Employee    name=Robin the Robot
    Should Be Equal As Integers    ${response.status_code}    201
    Should Be Equal    ${response.json()}[name]    Robin the Robot

Test Generated Keywords: Get Employee
    ${response}=    Get Employee
    Should Be Equal As Integers    ${response.status_code}    200

Test Generated Keywords: Patch Employee
    ${employee_id}=    Get Valid Id For Path    path=/employees/{employee_id}
    VAR    &{body}    date_of_birth=2021-12-31
    ${response}=    Patch Employee    employee_id=${employee_id}    body=${body}
    # ${response}=    Patch Employee    employee_id=${employee_id}    date_of_birth=2021-12-31
    Should Be Equal As Integers    ${response.status_code}    403

Test Generated Keywords: Post WageGroup
    VAR    &{body}    hourly_rate=99.99
    ${response}=    Post Wagegroup    body=${body}
    # ${response}=    Post Wagegroup    hourly_rate=99.99
    Should Be Equal As Integers    ${response.status_code}    201
    Should Be Equal As Numbers    ${response.json()}[hourly-rate]    99.99

Test Generated Keywords: Get Energy Label
    ${response}=    Get Energy Label
    ...    zipcode=1111AA
    ...    home_number=10
    ...    extension=too long to be acceptable
    ...    validate_against_schema=${FALSE}
    Should Be Equal As Integers    ${response.status_code}    422

    VAR    @{omit}    extension
    ${response}=    Get Energy Label
    ...    zipcode=1111AA
    ...    home_number=10
    ...    extension=too long to be acceptable
    ...    omit_parameters=${omit}
    Should Be Equal As Integers    ${response.status_code}    200

OpenApiDriver for Robot Framework

OpenApiDriver is an extension of the Robot Framework DataDriver library that allows for generation and execution of test cases based on the information in an OpenAPI document. For more information about the DataDriver library, see https://github.com/Snooz82/robotframework-datadriver.

Keyword documentation for OpenApiDriver can be found here.

Introduction

My RoboCon 2022 talk about OpenApiDriver and OpenApiLibCore can be found here. For more information about Robot Framework, see http://robotframework.org.

Installation

If you already have Python >= 3.10 with pip installed, you can simply run:

pip install --upgrade robotframework-openapitools

OpenAPI (aka Swagger)

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs, see https://www.openapis.org/ The OpenApiTools libraries implement a number of Robot Framework keywords that make it easy to interact with an OpenAPI implementation by using the information in the OpenAPI document (previously: Swagger file), for examply by automatic generation of valid values for requests based on the schema information in the document.

Note: The OpenApiTools libraries are designed for APIs based on the OAS v3. The libraries have not been tested for APIs based on the OAS v2.

Getting started

Before trying to use the keywords exposed by any of the libraries from OpenApiTools on the target API, it's recommended to first ensure that the OpenAPI document for the API is valid under the OpenAPI Specification. This can be done using the command line interface of the prance package that is installed as a prerequisite for OpenApiTools. Both a local openapi.json or openapi.yaml file or one hosted by the API server can be checked using the prance validate shell command:


(.venv) prance validate --backend=openapi-spec-validator http://localhost:8000/openapi.json

Processing "http://localhost:8000/openapi.json"...
 -> Resolving external references.
Validates OK as OpenAPI 3.0.2!

(.venv) prance validate --backend=openapi-spec-validator /tests/files/petstore_openapi.yaml

Processing "/tests/files/petstore_openapi.yaml"...
 -> Resolving external references.
Validates OK as OpenAPI 3.0.2!

You'll have to change the url or file reference to the location of the openapi document for your API.

Note: Although recursion is technically allowed under the OAS, tool support is limited and changing the OAS to not use recursion is recommended.

Support for parsing OpenAPI documents with recursion in them is limited. See the recursion_limit and recursion_default library parameters for the available options. If the OpenAPI document passes this validation, the next step is trying to do a test run with a minimal test suite. The examples below can be used with source, origin and path altered to fit your situation.


*** Settings ***
Library            OpenApiDriver
...                    source=http://localhost:8000/openapi.json
...                    origin=http://localhost:8000
Test Template      Validate Using Test Endpoint Keyword

*** Test Cases ***
Test Endpoint for ${method} on ${path} where ${status_code} is expected

*** Keywords ***
Validate Using Test Endpoint Keyword
    [Arguments]    ${path}    ${method}    ${status_code}
    Test Endpoint
    ...    path=${path}    method=${method}    status_code=${status_code}


*** Settings ***
Library            OpenApiLibCore
...                    source=http://localhost:8000/openapi.json
...                    origin=http://localhost:8000

*** Test Cases ***
Getting Started
    ${url}=    Get Valid Url    path=/employees/{employee_id}

Running the above tests for the first time may result in an error / failed test. You should look at the Robot Framework log.html to determine the reasons for the failing tests. Depending on the reasons for the failures, different solutions are possible. The OpenApiTools libraries also support handling of relations between resources within the scope of the API being validated as well as handling dependencies on resources outside the scope of the API. In addition there is support for handling restrictions on the values of parameters and properties. This is supported by the mappings_path library parameter. Details about the mappings_path parameter usage can be found under the Advanced Use tab.

Limitations

There are currently a number of limitations to supported API structures, supported data types and properties. The following list details the most important ones. - Only JSON request and response bodies are supported. - No support for per-path authorization levels (only simple 401 / 403 validation). For all open issues please visit the OpenApiTools GitHub page.

OpenApiLibCore for Robot Framework

The OpenApiLibCore library is a utility library that is meant to simplify creation of other Robot Framework libraries for API testing based on the information in an OpenAPI document. Both OpenApiDriver and libraries generated using OpenApiLibGen rely on OpenApiLibCore and its keywords for their functionality. This document explains how to use the OpenApiLibCore library.

Keyword documentation for OpenApiLibCore can be found here.

Introduction

My RoboCon 2022 talk about OpenApiDriver and OpenApiLibCore can be found here. For more information about Robot Framework, see http://robotframework.org.

Installation

If you already have Python >= 3.10 with pip installed, you can simply run:

pip install --upgrade robotframework-openapitools

OpenAPI (aka Swagger)

Note: The OpenApiTools libraries are designed for APIs based on the OAS v3. The libraries have not been tested for APIs based on the OAS v2.

Getting started


(.venv) prance validate --backend=openapi-spec-validator http://localhost:8000/openapi.json

Processing "http://localhost:8000/openapi.json"...
 -> Resolving external references.
Validates OK as OpenAPI 3.0.2!

(.venv) prance validate --backend=openapi-spec-validator /tests/files/petstore_openapi.yaml

Processing "/tests/files/petstore_openapi.yaml"...
 -> Resolving external references.
Validates OK as OpenAPI 3.0.2!

You'll have to change the url or file reference to the location of the openapi document for your API.

Note: Although recursion is technically allowed under the OAS, tool support is limited and changing the OAS to not use recursion is recommended.


*** Settings ***
Library            OpenApiDriver
...                    source=http://localhost:8000/openapi.json
...                    origin=http://localhost:8000
Test Template      Validate Using Test Endpoint Keyword

*** Test Cases ***
Test Endpoint for ${method} on ${path} where ${status_code} is expected

*** Keywords ***
Validate Using Test Endpoint Keyword
    [Arguments]    ${path}    ${method}    ${status_code}
    Test Endpoint
    ...    path=${path}    method=${method}    status_code=${status_code}


*** Settings ***
Library            OpenApiLibCore
...                    source=http://localhost:8000/openapi.json
...                    origin=http://localhost:8000

*** Test Cases ***
Getting Started
    ${url}=    Get Valid Url    path=/employees/{employee_id}

Limitations

Advanced use scenario's: using the mappings_path

Introduction

When working with APIs, there are often relations between resources or constraints on values. The property on one resource may refer to the id of another resource. The value for a certain property may have to be unique within a certain scope. Perhaps an url contains parameters that must match values that are defined outside the API itself. These types of relations and limitations cannot be described / modeled within the openapi document. To support automatic validation of API endpoints where such relations apply, OpenApiLibCore supports the usage of a custom mappings file.

Taking a custom mappings file into use

To take a custom mappings file into use, the absolute path to it has to be passed to OpenApiLibCore as the mappings_path parameter:


*** Settings ***
Library             OpenApiLibCore
...                     source=http://localhost:8000/openapi.json
...                     origin=http://localhost:8000
...                     mappings_path=${ROOT}/tests/custom_user_mappings.py
...

Note: An absolute path is required. In the example above, ${ROOT} is a global variable that holds the absolute path to the repository root.

The custom mappings file

Just like custom Robot Framework libraries, the mappings file has to be implemented in Python. Since this Python file is imported by the OpenApiLibCore, it has to follow a fixed format (more technically, implement a certain interface). The (almost) bare minimum implementation of a mappings.py file looks like this:


from OpenApiLibCore import (
    IGNORE,
    Dto,
    IdDependency,
    IdReference,
    PathPropertiesConstraint,
    PropertyValueConstraint,
    UniquePropertyValueConstraint,
)


ID_MAPPING = {
    "/myspecialpath", "special_thing_id",
}


class MyDtoThatDoesNothing(Dto):
    @staticmethod
    def get_relations():
        relations = []
        return relations

    @staticmethod
    def get_path_relations():
        relations = []
        return relations

    @staticmethod
    def get_parameter_relations():
        relations = []
        return relations


DTO_MAPPING = {
    ("/myspecialpath", "post"): MyDtoThatDoesNothing
}


PATH_MAPPING = {
    "/mypathwithexternalid/{external_id}": MyDtoThatDoesNothing
}

There are 5 main parts in this mappings file:

The import section. Here the classes needed to implement custom mappings are imported. This section can just be copied without changes.
The ID_MAPPING "constant" definition / assignment.
The section defining the mapping Dtos. More on this later.
The DTO_MAPPING "constant" definition / assignment.
The PATH_MAPPING "constant" definition / assignment.

The ID_MAPPING, DTO_MAPPING and PATH_MAPPING

When a custom mappings file is used, the OpenApiLibCore will attempt to import it and then import DTO_MAPPING, PATH_MAPPING and ID_MAPPING from it. For this reason, the exact same name must be used in a custom mappings file (capitilization matters).

The ID_MAPPING

The ID_MAPPING is a dictionary with a string as its key and either a string or a tuple of string and a callable as its value. The callable must take a string as its argument and return a string. The ID_MAPPING is used to specify what the unique identifier property of a resource at the given path is, if different from the default_id_property_name (see library parameters). In some situations, the identifier of the resource is not url-safe (e.g. containing a /). To support this type of resource identifier, a transformer can be provided:


def my_transformer(identifier_name: str) -> str:
    return identifier_name.replace("/", "_")


ID_MAPPING = {
    "/myspecialpath": ("special_thing_id", my_transformer),
}

The DTO_MAPPING

The DTO_MAPPING is a dictionary with a tuple as its key and a mappings Dto as its value. The tuple must be in the form ("path_from_the_paths_section", "method_supported_by_the_path"). The path_from_the_paths_section must be exactly as found in the openapi document. The method_supported_by_the_path must be one of the methods supported by the path and must be in lowercase.

The PATH_MAPPING

The PATH_MAPPING is a dictionary with a "path_from_the_paths_section" as its key and a mappings Dto as its value. The path_from_the_paths_section must be exactly as found in the openapi document.

Dto mapping classes

As can be seen from the import section above, a number of classes are available to deal with relations between resources and / or constraints on properties. Each of these classes is designed to handle a relation or constraint commonly seen in REST APIs. To explain the different mapping classes, we'll use the following example:

Imagine we have an API path /employees where we can create (post) a new Employee resource. The Employee has a number of required properties; name, employee_number, wagegroup_id, and date_of_birth. There is also the the /wagegroups path where a Wagegroup resource can be created. This Wagegroup also has a number of required properties: name and hourly rate.

`IdReference`

The value for this propery must the the id of another resource

To add an Employee, a wagegroup_id is required, the id of a Wagegroup resource that is already present in the system. Since a typical REST API generates this id for a new resource and returns that id as part of the post response, the required wagegroup_id can be obtained by posting a new Wagegroup. This relation can be implemented as follows:


class EmployeeDto(Dto):
    @staticmethod
    def get_relations():
        relations = [
            IdDependency(
                property_name="wagegroup_id",
                get_path="/wagegroups",
                error_code=451,
            ),
        ]
        return relations

DTO_MAPPING = {
    ("/employees", "post"): EmployeeDto
}

Notice that the get_path of the IdDependency is not named post_path as one might expect. This is deliberate for two reasons: 1. The purpose is getting an id 2. If the post operation is not supported on the provided path, a get operation is performed instead. It is assumed that such a get will yield a list of resources and that each of these resources has an id that is valid for the desired post operation. Also note the error_code of the IdDependency. If a post is attempted with a value for the wagegroup_id that does not exist, the API should return an error_code response. This error_code should be described as one of the responses in the openapi document for the post operation of the /employees path.

`IdDependency`

This resource may not be DELETED if another resource refers to it

If an Employee has been added to the system, this Employee refers to the id of a Wagegroup for its required employee_number property. Now let's say there is also the /wagegroups/${wagegroup_id} path that supports the delete operation. If the Wagegroup refered to the Employee would be deleted, the Employee would be left with an invalid reference for one of its required properties. To prevent this, an API typically returns an error_code when such a delete operation is attempted on a resource that is refered to in this fashion. This error_code should be described as one of the responses in the openapi document for the delete operation of the /wagegroups/${wagegroup_id} path. To verify that the specified error_code indeed occurs when attempting to delete the Wagegroup, we can implement the following dependency:


class WagegroupDto(Dto):
    @staticmethod
    def get_relations():
        relations = [
            IdReference(
                property_name="wagegroup_id",
                post_path="/employees",
                error_code=406,
            ),
        ]
        return relations

DTO_MAPPING = {
    ("/wagegroups/{wagegroup_id}", "delete"): WagegroupDto
}

`UniquePropertyValueConstraint`

The value of this property must be unique within its scope

In a lot of systems, there is data that should be unique; an employee number, the email address of an employee, the domain name for the employee, etc. Often those values are automatically generated based on other data, but for some data, an "available value" must be chosen by hand. In our example, the required employee_number must be chosen from the "free" numbers. When a number is chosen that is already in use, the API should return the error_code specified in the openapi document for the operation (typically post, put and patch) on the endpoint. To verify that the specified error_code occurs when attempting to post an Employee with an employee_number that is already in use, we can implement the following dependency:


class EmployeeDto(Dto):
    @staticmethod
    def get_relations():
        relations = [
            UniquePropertyValueConstraint(
                property_name="employee_number",
                value=42,
                error_code=422,
            ),
        ]
        return relations

DTO_MAPPING = {
    ("/employees", "post"): EmployeeDto,
    ("/employees/${employee_id}", "put"): EmployeeDto,
    ("/employees/${employee_id}", "patch"): EmployeeDto,
}

Note how this example reuses the EmployeeDto to model the uniqueness constraint for all the operations (post, put and patch) that all relate to the same employee_number.

`PropertyValueConstraint`

Use one of these values for this property

The OpenApiLibCore uses the type information in the openapi document to generate random data of the correct type to perform the operations that need it. While this works in many situations (e.g. a random string for a name), there can be additional restrictions to a value that cannot be specified in an openapi document. In our example, the date_of_birth must be a string in a specific format, e.g. 1995-03-27. This type of constraint can be modeled as follows:


class EmployeeDto(Dto):
    @staticmethod
    def get_relations():
        relations = [
            PropertyValueConstraint(
                property_name="date_of_birth",
                values=["1995-03-27", "1980-10-02"],
                error_code=422,
            ),
        ]
        return relations

DTO_MAPPING = {
    ("/employees", "post"): EmployeeDto,
    ("/employees/${employee_id}", "put"): EmployeeDto,
    ("/employees/${employee_id}", "patch"): EmployeeDto,
}

Now in addition, there could also be the restriction that the Employee must be 18 years or older. To support additional restrictions like these, the PropertyValueConstraint supports two additional properties: error_value and invalid_value_error_code:


class EmployeeDto(Dto):
    @staticmethod
    def get_relations():
        relations = [
            PropertyValueConstraint(
                property_name="date_of_birth",
                values=["1995-03-27", "1980-10-02"],
                error_code=422,
                invalid_value="2020-02-20",
                invalid_value_error_code=403,
            ),
        ]
        return relations

DTO_MAPPING = {
    ("/employees", "post"): EmployeeDto,
    ("/employees/${employee_id}", "put"): EmployeeDto,
    ("/employees/${employee_id}", "patch"): EmployeeDto,
}

So now if an incorrectly formatted date is send a 422 response is expected, but when 2020-02-20 is send the expected repsonse is 403. In some API implementations, there may be a property that will always return a specific error code if it's value is not valid. This means that sending e.g. an invalid type of value will not result in the default error code for the API (typically 422 or 400). This situation can be handled by use of the special IGNORE value (see below for other uses):


class EmployeeDto(Dto):
    @staticmethod
    def get_relations():
        relations = [
            PropertyValueConstraint(
                property_name="date_of_birth",
                values=["1995-03-27", "1980-10-02"],
                error_code=403,
                invalid_value=IGNORE,
                invalid_value_error_code=422,
            ),
        ]
        return relations

DTO_MAPPING = {
    ("/employees", "post"): EmployeeDto,
    ("/employees/${employee_id}", "put"): EmployeeDto,
    ("/employees/${employee_id}", "patch"): EmployeeDto,
}

Note that while this configuration will prevent failing test cases generated by OpenApiDriver, it does not explicitly check for business logic errors anymore (younger than 18 in this example).

`PathPropertiesConstraint`

Just use this for the path

Note: The PathPropertiesConstraint is only applicable to the get_path_relations in a Dto and only the PATH_MAPPING uses the get_path_relations.

To be able to automatically perform endpoint validations, the OpenApiLibCore has to construct the url for the resource from the path as found in the openapi document. Often, such a path contains a reference to a resource id, e.g. "/employees/${employee_id}". When such an id is needed, the OpenApiLibCore tries to obtain a valid id by taking these steps: 1. Attempt a post on the "parent path" and extract the id from the response. In our example: perform a post request on the /employees path and get the id from the response. 2. If 1. fails, perform a get request on the /employees path. It is assumed that this will return a list of Employee objects with an id. One item from the returned list is picked at rondom and its id is used. This mechanism relies on the standard REST structure and patterns. Unfortunately, this structure / pattern does not apply to every endpoint; not every path parameter refers to a resource id. Imagine we want to extend the API from our example with an endpoint that returns all the Employees that have their birthday at a given date: "/birthdays/${month}/${date}". It should be clear that the OpenApiLibCore won't be able to acquire a valid month and date. The PathPropertiesConstraint can be used in this case:


class BirthdaysDto(Dto):
    @staticmethod
    def get_path_relations():
        relations = [
            PathPropertiesConstraint(path="/birthdays/03/27"),
        ]
        return relations

PATH_MAPPING = {
    "/birthdays/{month}/{date}": BirthdaysDto
}

`IGNORE`

Never send this query parameter as part of a request

Some optional query parameters have a range of valid values that depend on one or more path parameters. Since path parameters are part of an url, they cannot be optional or empty so to extend the path parameters with optional parameters, query parameters can be used. To illustrate this, let's imagine an API where the energy label for a building can be requested: "/energylabel/${zipcode}/${home_number}". Some addresses however have an address extension, e.g. 1234AB 42 ^2.C. The extension may not be limited to a fixed pattern / range and if an address has an extension, in many cases the address without an extension part is invalid. To prevent OpenApiLibCore from generating invalid combinations of path and query parameters in this type of endpoint, the IGNORE special value can be used to ensure the related query parameter is never send in a request.


class EnergyLabelDto(Dto):
    @staticmethod
    def get_parameter_relations():
        relations = [
            PropertyValueConstraint(
                property_name="address_extension",
                values=[IGNORE],
                error_code=422,
            ),
        ]
        return relations

    @staticmethod
    def get_relations(:
        relations = [
            PathPropertiesConstraint(path="/energy_label/1111AA/10"),
        ]
        return relations

DTO_MAPPING = {
    ("/energy_label/{zipcode}/{home_number}", "get"): EnergyLabelDto
}

Note that in this example, the get_parameter_relations() method is implemented. This method works mostly the same as the get_relations() method but applies to headers and query parameters.

Type annotations

An additional import to support type annotations is also available: Relation. A fully typed example can be found here.