Asynchronous Test Framework #HighPerformance #EasyToLearn #FastToCode #AsyncTests


Keywords
test, unittest, restapi, testframework, asyncio, async, asynchronous, testingframework, rest, api, python, python3, testing, unittesting, automation, automationtest, automationtesting, restapitest, restapitesting, restapiunittest, restapiunittesting, restapiautomation, restapiautomationtest, restapiautomationtesting, apitest, apitesting, apiunittest, apiunittesting, apiautomation, apiautomationtest, apiautomationtesting
License
MIT
Install
pip install test-rest-api==0.0.0.0.31

Documentation

test_rest_api

TEST REST API

Create fast modern asynchronous tests for REST API testing

#FAST #EASY #ASYNC #RESTAPI #TESTING #AUTOMATION

Features

  • Asyncronus programming (Powered by asyncio)
  • Auto discovery of test modules and functions
  • Supports both asyncronus & syncronuous tests
  • High speed test executions using async functions
  • Html test reporting with custom logs & summary dashboards
  • Create complex flows using parameterization & correlation
  • Group similar tests using hashtags eg: #smoke
  • Supports CI/CD test automation integrations
  • Designed to be easy to use & learn

Installation

If you already have Python with pip installed, you can simply run:

pip install test_rest_api

Usage

1. Basic usage


Syntax

python -m test_rest_api -t "<Test folder/file path>"

  • Tests are executed from the command line using the test_rest_api module directly
  • The basic usage is by giving a file path or directory path as an argument
  • -t stands for Test suite path
  • We can organise folders, sub folders and python files in any custom structure
  • test_rest_api will autodetect python files and folders as Test suites

2. Set report path


Syntax

python -m test_rest_api -t "<Test folder/file path>" -r "Result folder path"

  • In the above example, html test report is saved under the same test folder path
  • We can save the final report to our custom path by providing -r followed by path
  • -r stands for Report path
  • test_rest_api creates beautiful rich test report with summary dashboards
  • We can also add our custom logs to each individual tests in the test report

3. Set .env path


Syntax

python -m test_rest_api -t "<Test folder/file path>" -r "Result folder path" -e ".env file path"
  • We can set variables with values, example Domain, Username, Password etc. in .env file
  • test_rest_api will auto fetch all these values and save under Global Variables as constants
  • Global Variables can be accessed in tests for parametrization
  • One example can be dynamic url creation. Domain can be parameterised in url creation
  • This will also help developers to run the same tests in different environments by creating separate .env files
  • We can set the environment variables by providing -e followed by path
  • -e stands for Environment path

4. Set hashtags


Syntax

python -m test_rest_api -t "<Test folder/file path>" -r "Result folder path" -e ".env file path" -h #SMOKE#SANITY
  • Group your test by providing tags in test creation
  • This helps to execute a group of testcases, example smoke testing, regression testing etc.
  • test_rest_api will fetch all tests with the provided tags in the command line and executes them
  • We can set the hashtags for execution by providing -h followed by tags, example #SMOKE#SANITY
  • -h stands for Hashtags
  • When no tags are provided, all the tests are executed
  • When we provide a single tag, only those testcases with that tag are executed
  • Some tests like login should run always for all tags irrespective of the value
  • In this case it's not practical to add all tags to the login test function
  • Adding all tags will be hard to maintain when we introduce new custom tags
  • To tackle this issue, we can use inbuilt #ALL tag
  • #ALL tagged tests will always be executed. It will not be skipped
  • Tests like login, logout etc. are perfect candidates for #ALL

5. Project structure


.
├── api                           # Store all your rest api files
│   ├── auth                      # Custom structure for subfolders & files
│   │   ├── login.py              # Python file to store rest_api code
│   │   │   ├── def user_login()  # Python function to create rest_api
│   │   │   ├── def admin_login()
│   │   │   └── ...    
│   │   └── ...
│   └── ...
├── testsuite                     # Root testsuite folder
│   ├── auth                      # Custom structure for subfolders & files
│   │   ├── login.py              # Python file as testsuite to store tests
│   │   │   ├── async def t001()  # Python async function as testcases
│   │   │   ├── async def t002()
│   │   │   └── ...    
│   │   └── ...
│   └── ...
└── ...
  • 2 sub folders api & testsuite
  • Any custom names can be used instead of api & testsuite
  • api should store all the api creation python files
  • testsuite should store all the test creation python files
  • Any custom structure with multiple sub folders can be used to organise the files
  • Separating api from tests will avoid code duplication for rest_api creation
  • Root testsuite folder path will be used in command line execution value for -t

Examples

1. My first test


from test_rest_api import test


@test()
async def my_first_test():
    assert 4 == 5
  • Create an async python function with any custom name
  • Decorate the function using @test()
  • Async python functions decoratd with @test() will be auto-detected as testcase
  • Normal python functions decoratd with @test() will not be considered as testcase

Congrats !

You have successfully created your first test (my_first_test)

Now let's execute it from command line . . .

  • Create a python virtual environment
  • Install test_rest_api using command pip install test_rest_api
  • Create new python file (eg: my_first_testsuite.py) and paste the above code
  • Execute the test from the command line using the test_rest_api module directly

python -m test_rest_api -t "<Test folder/file path>" -r "Result folder path"

2. Configure my test


from test_rest_api import test


@test(name='Custom Name', desc='Description', enabled=True, tags=['SMOKE', 'ABC'], is_async=True, execution_order='1')
async def my_second_test():
    assert 4 == 5
  • In our first example, we used @test() decorator with empty parameters

  • In this example, we are using parameters to configure our test function

  • This can be considered as adding settings to your test function

  • name

    • Mandatory: False
    • Data Type: str
    • Expected: Custom name for your test
    • Default: Function name
  • desc

    • Mandatory: False
    • Data Type: str
    • Expected: Test description
    • Default: Empty string
  • enabled

    • Mandatory: False
    • Data Type: bool
    • Expected: True or False. Provide False, to disable the test
    • Default: True
  • tags

    • Mandatory: False
    • Data Type: list
    • Expected: List of tags. Group your tests with custom tags
    • Default: Empty list
  • is_async

    • Mandatory: False
    • Data Type: bool
    • Expected: True or False. Provide False, to run tests sequentially
    • Default: True
  • execution_order

    • Mandatory: False
    • Data Type: str
    • Expected: Custom text for ordering. This will work only when is_async = False
    • Default: 'zzzzz'

3. Replace inbuilt assert


from test_rest_api import test, Assert


@test()
async def my_first_logger():
    # assert 4 == 5
    Assert.equal(4, 5)
  • Assert is same as inbuilt python assert statement
  • Using test rest api Assert class improves logging
  • Assertions done using Assert will be automatically logged in final html report
  • It is recommended to use Assert instead of inbuilt assert statement for all your tests

4. My first log


from test_rest_api import test


@test()
async def my_first_logger():
    print('My 1st log')
    print('My 2nd log')
    assert 4 == 5
  • Python inbuilt print() function is used to add custom messages to the final html test report
  • Add print() functions inside your test function
  • Add any number of log messages without any limit
  • It is recommended to add logs for all your tests
  • Note: print() statements will not be printed to console

5. Set global variables value


from test_rest_api import test, GlobalVariables


@test()
async def set_global_variables_value():
    GlobalVariables.set(name='token', value='token from response')
  • Global variables are used to save & retrieve values in runtime
  • This will help in parameterization and correlation of tests
  • Save Token from login api response in your first test
  • Retrieve this Token in all your upcoming tests for authentication
  • Here we will learn how to set the value in GlobalVariables

GlobalVariables.set(name='token', value='token from response')

  • name
    • Mandatory: True
    • Data Type: str
    • Expected: Custom name for your global variable
  • value
    • Mandatory: True
    • Data Type: any
    • Expected: Any python data type can be stored as global variables value

6. Set global variables value as constant


from test_rest_api import test, GlobalVariables


@test()
async def set_global_variables_value_as_constant():
    GlobalVariables.set(name='pi', value=3.14, is_constant=True)
  • In the above example we learned to set value to global variables
  • Multiple set calls are possible for the same variable
  • Token variable can be set again in a different test function
  • To avoid this, we can set global variables as a constant value
  • Use is_consant optional parameter & make it True

GlobalVariables.set(name='pi', value=3.14, is_constant=True)

  • name
    • Mandatory: True
    • Data Type: str
    • Expected: Custom name for your global variable
  • value
    • Mandatory: True
    • Data Type: any
    • Expected: Any python data type can be stored as global variables value
  • is_constant
    • Mandatory: False
    • Data Type: bool
    • Expected: True or False. Provide True, to create constants
    • Default: False

7. Get global variables value


from test_rest_api import test, GlobalVariables


@test()
async def get_global_variables_value():
    token: str = GlobalVariables.get(name='token')
    pi: float = GlobalVariables.get(name='pi')
  • In the above examples we learned to set varying and constant value to global variables
  • Now it's time to retrieve them in our tests
  • Please make sure to use valid variable names while retrieving the values
  • If the variable name is not present in global variables, test will terminate with error

GlobalVariables.get(name='token')

  • name
    • Mandatory: True
    • Data Type: str
    • Expected: Valid name of any saved global variable

8. My first bug


from test_rest_api import test, Bug


@test()
async def my_first_bug():
    Bug()
  • Bug is used to raise issues in tests
  • Add custom checks in your tests to validate rest api response
  • If actual result is not the expected result, just call Bug()
  • This will terminate the current test function execution
  • Bug details can be viewed in final html test report

9. Configure my bug


from test_rest_api import test, Bug, Logger


@test()
async def my_second_bug():
    logger = Logger()
    logger.log('step 1')
    logger.log('step 2')
    logger.log('Consider all steps are logged')
    Bug(message="msg", priority=Bug.PRIORITY.BLOCKER, actual_result="", expected_result="", steps_to_reproduce=logger)
  • In our first bug example, we used Bug() with empty attributes
  • In this example, we are using attributes to configure our Bug
  • This can be considered as adding more info to your bug in the final html test report
  • We are adding custom logging in this example to show how logger instance is useful in Bug creation
  • Logger instance variable can be passed to steps_to_reproduce attribute during Bug creation
Bug(message='', priority='', actual_result='', expected_result='', steps_to_reproduce='')
  • message
    • Mandatory: False
    • Data Type: str
    • Expected: Custom message for your bug
    • Default: Empty string
  • priority
    • Mandatory: False
    • Data Type: str
    • Expected: Priority for this bug. Supported list: Bug.PRIORITY.[ items ]
    • Default: Empty string
  • actual_result
    • Mandatory: False
    • Data Type: str
    • Expected: Provide the actual result
    • Default: Empty string
  • expected_result
    • Mandatory: False
    • Data Type: str
    • Expected: Provide the expected result
    • Default: Empty list
  • steps_to_reproduce
    • Mandatory: False
    • Data Type: str
    • Expected: Logger instance can be used to auto-populate this field
    • Default: Empty string

10. My first rest api


from test_rest_api import test, RestApi


@test()
async def my_first_rest_api():
    rest_api = RestApi(url='https://dummyjson.com/products/1')
  • RestApi is used create rest api instance in tests
  • Here we have created a basic rest api with just the url information
  • This example is only about creating rest api, no send action is performed here
  • We will use this instance variable for sending the request in upcoming examples

11. Configure my rest api


from test_rest_api import test, RestApi


@test()
async def configure_my_rest_api():
    rest_api = RestApi(url='my url', parameters={'p1': 'v1', 'p2': 'val2'}, headers={'h1': 'v1', 'h2': 'val1'}, body={})
  • In the above example, we have only used url attribute for rest api creation

  • Other attributes for creation are parameters, headers and body

  • This example shows the syntax for adding these optional attributes

  • url

    • Mandatory: True
    • Data Type: str
    • Expected: Rest api url
  • parameters

    • Mandatory: False
    • Data Type: dict
    • Expected: Key value pairs of parameter name & value
    • Default: {}
  • headers

    • Mandatory: False
    • Data Type: dict
    • Expected: Key value pairs of header name & value
    • Default: {}
  • body

    • Mandatory: False
    • Data Type: dict
    • Expected: Provide the json request payload
    • Default: {}

12. Reuse my rest api


from test_rest_api import RestApi, GlobalVariables


def login_api(username: str, password: str):
    domain = GlobalVariables.get('DOMAIN')
    url = f'https://{domain}/login'
    body = {'username': username, 'password': password}
    return RestApi(url=url, body=body)
  • A rest api will be used for multiple tests
  • Creation of rest api inside test async functions will result in code duplication
  • Duplicate code makes your program lengthy and decreases your code quality
  • Updating & maintaining this rest api creations in multiple tests will be difficult
  • New changes to rest api, will result in changing the same code multiple times
  • To avoid duplication, we can use a separate folder for rest api files
  • Use python functions to create a rest api which will avoid code duplication
  • You can call a function 100 times instead of writing it 100 times
  • In this example we have created a simple login api
  • All the dynamic values for rest api creation can be passed as function parameters
  • This helps in calling the same api with different inputs
  • Return the RestApi instance which can be used in test functions for sending

13. Send my rest api


from test_rest_api import test, RestApi


@test()
async def send_my_rest_api():
    rest_api = RestApi(url='https://dummyjson.com/products/1')
    response1 = await rest_api.get()
    response2 = await rest_api.send(method='get')
    response3 = await rest_api.send(method=rest_api.METHODS.GET)
  • In the above examples, we learned to create a rest api
  • Now it's time to send them using http methods
  • Supported http methods are GET, POST, PUT, PATCH, DELETE, OPTIONS & HEAD
  • Here we are sending the rest_api using GET http method
  • All the responses (1, 2 & 3) will have the same result
  • Because they perform the same functionality with different syntax
  • Similarly, other http methods can be used, with your desired syntax

14. Rest api response


from test_rest_api import test, RestApi


@test()
async def send_my_rest_api():
    rest_api = RestApi(url='https://dummyjson.com/products/1')
    response = await rest_api.get()

    status_code = response.status_code
    body = response.body
    headers = response.headers
    content_type = response.content_type
    obj = response.obj
  • We have covered rest api creation & sending part

  • Now lets see more about the response object

  • Mostly all checks will be performed using response object data

  • Response object contains the below details

  • response.status_code

    • Data Type: int
    • Value: Response status code
  • response.body

    • Data Type: dict
    • Value: Response body
  • response.headers

    • Data Type: dict
    • Value: Response headers
  • response.content_type

    • Data Type: str
    • Value: Response content type
  • response.obj

    • Data Type: aiohttp.ClientResponse
    • Value: Python aiohttp ClientResponse object

15. Demo with all the above features


.env file
---------

DOMAIN=dummyjson.com
from test_rest_api import test, RestApi, Bug, Logger, GlobalVariables


@test(name='Demo', desc='All features')
async def demo():
    logger = Logger()
    logger.log('Starting the test')

    logger.log('Creating rest api')
    domain = GlobalVariables.get('DOMAIN')
    rest_api = RestApi(url=f"https://{domain}/products/1")

    logger.log('Sending rest api')
    response = await rest_api.get()

    logger.log(f'Response code = {response.status_code}')
    logger.log(f'Response body = {response.body}')

    logger.log('Validate response code')
    if response.status_code != 200:
        Bug(message='Invalid status code', priority=Bug.PRIORITY.MINOR, actual_result=response.status_code,
            expected_result='200', steps_to_reproduce=logger)

    logger.log('Save price details to global variables')
    title = response.body.get('title', '')
    price = response.body.get('price', '')
    GlobalVariables.set(name=title, value=price)

    logger.log('Successfully completed the test')

Reports

1. My first report


  • Single basic test with PASS status

Click here to view the html report

Console output

2023-04-08 15:40:22,712: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 15:40:22,712: Starting test setup
2023-04-08 15:40:22,712: Auto detecting test suites
2023-04-08 15:40:22,712: Total test suites: 1
2023-04-08 15:40:22,712: Auto detecting tests
2023-04-08 15:40:22,713: Total synchronous tests: 0
2023-04-08 15:40:22,714: Total asynchronous tests: 1
2023-04-08 15:40:22,714: Total tests: 1
2023-04-08 15:40:22,714: Created aiohttp client session
2023-04-08 15:40:22,714: Completed test setup
2023-04-08 15:40:22,714: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 15:40:22,714: PASS    my_first_test (root file) [1]
2023-04-08 15:40:22,715: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             PASS
                        Tests:              1
                        Start:              2023-04-08 15-40-22
                        End:                2023-04-08 15-40-22
                        Duration:           0.000956233 seconds
                        Tags:               []
                        
                        PASS:               1
                        FAIL:               0
                        ERROR:              0
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

2. Async tests report


  • 5 async tests
  • Each tests takes 1 second to complete

Click here to view the html report

Console output

2023-04-08 15:52:05,062: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 15:52:05,062: Starting test setup
2023-04-08 15:52:05,062: Auto detecting test suites
2023-04-08 15:52:05,062: Total test suites: 1
2023-04-08 15:52:05,062: Auto detecting tests
2023-04-08 15:52:05,063: Total synchronous tests: 0
2023-04-08 15:52:05,063: Total asynchronous tests: 5
2023-04-08 15:52:05,063: Total tests: 5
2023-04-08 15:52:05,063: Created aiohttp client session
2023-04-08 15:52:05,063: Completed test setup
2023-04-08 15:52:05,063: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 15:52:06,064: PASS    testcase_1 (root file) [1]
2023-04-08 15:52:06,064: PASS    testcase_2 (root file) [2]
2023-04-08 15:52:06,065: PASS    testcase_3 (root file) [3]
2023-04-08 15:52:06,065: PASS    testcase_4 (root file) [4]
2023-04-08 15:52:06,065: PASS    testcase_5 (root file) [5]
2023-04-08 15:52:06,066: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             PASS
                        Tests:              5
                        Start:              2023-04-08 15-52-05
                        End:                2023-04-08 15-52-06
                        Duration:           1.002069748 seconds
                        Tags:               []
                        
                        PASS:               5
                        FAIL:               0
                        ERROR:              0
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

3. Sync tests report


  • 5 sync tests
  • Each tests takes 1 second to complete

Click here to view the html report

Console output

2023-04-08 15:56:19,128: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 15:56:19,129: Starting test setup
2023-04-08 15:56:19,129: Auto detecting test suites
2023-04-08 15:56:19,129: Total test suites: 1
2023-04-08 15:56:19,129: Auto detecting tests
2023-04-08 15:56:19,129: Total synchronous tests: 5
2023-04-08 15:56:19,129: Total asynchronous tests: 0
2023-04-08 15:56:19,129: Total tests: 5
2023-04-08 15:56:19,129: Created aiohttp client session
2023-04-08 15:56:19,129: Completed test setup
2023-04-08 15:56:19,130: 
                          =======================================================
                        || ................  S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 15:56:20,131: PASS    testcase_1 (root file) [1]
2023-04-08 15:56:21,132: PASS    testcase_2 (root file) [2]
2023-04-08 15:56:22,132: PASS    testcase_3 (root file) [3]
2023-04-08 15:56:23,134: PASS    testcase_4 (root file) [4]
2023-04-08 15:56:24,135: PASS    testcase_5 (root file) [5]
2023-04-08 15:56:24,137: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             PASS
                        Tests:              5
                        Start:              2023-04-08 15-56-19
                        End:                2023-04-08 15-56-24
                        Duration:           5.006235552 seconds
                        Tags:               []
                        
                        PASS:               5
                        FAIL:               0
                        ERROR:              0
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

4. Sync & Async report


  • 5 sync & async tests each
  • Total 10 tests
  • Each tests takes 1 second to complete

Click here to view the html report

Console output

2023-04-08 15:59:38,170: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 15:59:38,170: Starting test setup
2023-04-08 15:59:38,170: Auto detecting test suites
2023-04-08 15:59:38,170: Total test suites: 1
2023-04-08 15:59:38,170: Auto detecting tests
2023-04-08 15:59:38,172: Total synchronous tests: 5
2023-04-08 15:59:38,172: Total asynchronous tests: 5
2023-04-08 15:59:38,172: Total tests: 10
2023-04-08 15:59:38,173: Created aiohttp client session
2023-04-08 15:59:38,173: Completed test setup
2023-04-08 15:59:38,173: 
                          =======================================================
                        || ................  S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 15:59:39,174: PASS    testcase_1 (root file) [1]
2023-04-08 15:59:40,176: PASS    testcase_2 (root file) [2]
2023-04-08 15:59:41,177: PASS    testcase_3 (root file) [3]
2023-04-08 15:59:42,178: PASS    testcase_4 (root file) [4]
2023-04-08 15:59:43,180: PASS    testcase_5 (root file) [5]
2023-04-08 15:59:43,180: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 15:59:44,182: PASS    testcase_async_1 (root file) [6]
2023-04-08 15:59:44,182: PASS    testcase_async_2 (root file) [7]
2023-04-08 15:59:44,183: PASS    testcase_async_3 (root file) [8]
2023-04-08 15:59:44,183: PASS    testcase_async_4 (root file) [9]
2023-04-08 15:59:44,183: PASS    testcase_async_5 (root file) [10]
2023-04-08 15:59:44,185: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             PASS
                        Tests:              10
                        Start:              2023-04-08 15-59-38
                        End:                2023-04-08 15-59-44
                        Duration:           6.01064299 seconds
                        Tags:               []
                        
                        PASS:               10
                        FAIL:               0
                        ERROR:              0
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

5. Multi status report


  • 5 async tests with different status values
  • Status list: PASS, FAIL, ERROR, DISABLE & SKIP

Click here to view the html report

Console output

2023-04-08 16:17:44,328: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 16:17:44,329: Starting test setup
2023-04-08 16:17:44,329: Auto detecting test suites
2023-04-08 16:17:44,329: Total test suites: 1
2023-04-08 16:17:44,329: Auto detecting tests
2023-04-08 16:17:44,330: Total synchronous tests: 0
2023-04-08 16:17:44,330: Total asynchronous tests: 5
2023-04-08 16:17:44,330: Total tests: 5
2023-04-08 16:17:44,330: Created aiohttp client session
2023-04-08 16:17:44,330: Completed test setup
2023-04-08 16:17:44,330: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 16:17:44,330: ERROR   t_error (root file) [2]
2023-04-08 16:17:44,331: FAIL    t_fail (root file) [3]
2023-04-08 16:17:44,331: PASS    t_pass (root file) [4]
2023-04-08 16:17:44,332: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              5
                        Start:              2023-04-08 16-17-44
                        End:                2023-04-08 16-17-44
                        Duration:           0.001355524 seconds
                        Tags:               ['SMOKE']
                        
                        PASS:               1
                        FAIL:               1
                        ERROR:              1
                        DISABLE:            1
                        SKIP:               1
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              1
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         1

6. Multi bug report

  • 5 async tests with different bug priority values
  • Priority list: LOW, MINOR, MAJOR, CRITICAL, BLOCKER

Click here to view the html report

Console output

2023-04-08 16:30:21,400: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 16:30:21,400: Starting test setup
2023-04-08 16:30:21,400: Auto detecting test suites
2023-04-08 16:30:21,400: Total test suites: 1
2023-04-08 16:30:21,400: Auto detecting tests
2023-04-08 16:30:21,401: Total synchronous tests: 0
2023-04-08 16:30:21,401: Total asynchronous tests: 5
2023-04-08 16:30:21,401: Total tests: 5
2023-04-08 16:30:21,402: Created aiohttp client session
2023-04-08 16:30:21,402: Completed test setup
2023-04-08 16:30:21,402: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 16:30:21,402: FAIL    t_blocker (root file) [1]
2023-04-08 16:30:21,404: FAIL    t_critical (root file) [2]
2023-04-08 16:30:21,404: FAIL    t_low (root file) [3]
2023-04-08 16:30:21,404: FAIL    t_major (root file) [4]
2023-04-08 16:30:21,404: FAIL    t_minor (root file) [5]
2023-04-08 16:30:21,406: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              5
                        Start:              2023-04-08 16-30-21
                        End:                2023-04-08 16-30-21
                        Duration:           0.003122595 seconds
                        Tags:               []
                        
                        PASS:               0
                        FAIL:               5
                        ERROR:              0
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                1
                        MINOR:              1
                        MAJOR:              1
                        CRITICAL:           1
                        BLOCKER:            1
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

7. Rest api errors


  • Developers can make errors while coding
  • General error exception messages will increase the time to find and fix these issues
  • test_rest_api provides short & precise error's with exact traceback info
  • Here we are purposefully making errors in RestApi functions

Click here to view the html report

Console output

2023-04-08 19:41:19,099: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 19:41:19,099: Starting test setup
2023-04-08 19:41:19,099: Auto detecting test suites
2023-04-08 19:41:19,099: Total test suites: 1
2023-04-08 19:41:19,099: Auto detecting tests
2023-04-08 19:41:19,099: Total synchronous tests: 0
2023-04-08 19:41:19,099: Total asynchronous tests: 12
2023-04-08 19:41:19,099: Total tests: 12
2023-04-08 19:41:19,099: Created aiohttp client session
2023-04-08 19:41:19,099: Completed test setup
2023-04-08 19:41:19,099: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 19:41:19,100: ERROR   t1 (root file) [1]
2023-04-08 19:41:19,101: ERROR   t10 (root file) [2]
2023-04-08 19:41:19,110: ERROR   t2 (root file) [5]
2023-04-08 19:41:19,110: ERROR   t3 (root file) [6]
2023-04-08 19:41:19,110: ERROR   t4 (root file) [7]
2023-04-08 19:41:19,110: ERROR   t5 (root file) [8]
2023-04-08 19:41:19,110: ERROR   t6 (root file) [9]
2023-04-08 19:41:19,111: ERROR   t7 (root file) [10]
2023-04-08 19:41:19,111: ERROR   t8 (root file) [11]
2023-04-08 19:41:19,111: ERROR   t9 (root file) [12]
2023-04-08 19:41:19,115: ERROR   t11 (root file) [3]
2023-04-08 19:41:19,285: ERROR   t12 (root file) [4]
2023-04-08 19:41:19,288: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              12
                        Start:              2023-04-08 19-41-19
                        End:                2023-04-08 19-41-19
                        Duration:           0.18714125 seconds
                        Tags:               []
                        
                        PASS:               0
                        FAIL:               0
                        ERROR:              12
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           12
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

8. Global variables errors


  • Developers can make errors while coding
  • General error exception messages will increase the time to find and fix these issues
  • test_rest_api provides short & precise error's with exact traceback info
  • Here we are purposefully making errors in GlobalVariables functions

Click here to view the html report

Console output

2023-04-08 18:38:05,825: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 18:38:05,825: Starting test setup
2023-04-08 18:38:05,825: Auto detecting test suites
2023-04-08 18:38:05,826: Total test suites: 1
2023-04-08 18:38:05,826: Auto detecting tests
2023-04-08 18:38:05,827: Total synchronous tests: 0
2023-04-08 18:38:05,827: Total asynchronous tests: 13
2023-04-08 18:38:05,827: Total tests: 13
2023-04-08 18:38:05,827: Created aiohttp client session
2023-04-08 18:38:05,827: Completed test setup
2023-04-08 18:38:05,827: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 18:38:05,827: ERROR   t_1 (root file) [1]
2023-04-08 18:38:05,828: ERROR   t_10 (root file) [2]
2023-04-08 18:38:05,828: ERROR   t_11 (root file) [3]
2023-04-08 18:38:05,828: ERROR   t_12 (root file) [4]
2023-04-08 18:38:05,829: ERROR   t_13 (root file) [5]
2023-04-08 18:38:05,829: ERROR   t_2 (root file) [6]
2023-04-08 18:38:05,829: ERROR   t_3 (root file) [7]
2023-04-08 18:38:05,829: ERROR   t_4 (root file) [8]
2023-04-08 18:38:05,829: ERROR   t_5 (root file) [9]
2023-04-08 18:38:05,830: ERROR   t_6 (root file) [10]
2023-04-08 18:38:05,830: ERROR   t_7 (root file) [11]
2023-04-08 18:38:05,830: ERROR   t_8 (root file) [12]
2023-04-08 18:38:05,830: ERROR   t_9 (root file) [13]
2023-04-08 18:38:05,831: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              13
                        Start:              2023-04-08 18-38-05
                        End:                2023-04-08 18-38-05
                        Duration:           0.003679225 seconds
                        Tags:               []
                        
                        PASS:               0
                        FAIL:               0
                        ERROR:              13
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   13
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         0

9. Bug errors


  • Developers can make errors while coding
  • General error exception messages will increase the time to find and fix these issues
  • test_rest_api provides short & precise error's with exact traceback info
  • Here we are purposefully making errors in Bug functions

Click here to view the html report

Console output

2023-04-08 19:55:51,870: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 19:55:51,871: Starting test setup
2023-04-08 19:55:51,871: Auto detecting test suites
2023-04-08 19:55:51,871: Total test suites: 1
2023-04-08 19:55:51,871: Auto detecting tests
2023-04-08 19:55:51,872: Total synchronous tests: 0
2023-04-08 19:55:51,872: Total asynchronous tests: 7
2023-04-08 19:55:51,872: Total tests: 7
2023-04-08 19:55:51,872: Created aiohttp client session
2023-04-08 19:55:51,872: Completed test setup
2023-04-08 19:55:51,872: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 19:55:51,872: ERROR   t1 (root file) [1]
2023-04-08 19:55:51,874: ERROR   t2 (root file) [2]
2023-04-08 19:55:51,875: ERROR   t3 (root file) [3]
2023-04-08 19:55:51,875: ERROR   t4 (root file) [4]
2023-04-08 19:55:51,875: ERROR   t5 (root file) [5]
2023-04-08 19:55:51,875: ERROR   t6 (root file) [6]
2023-04-08 19:55:51,876: ERROR   t7 (root file) [7]
2023-04-08 19:55:51,877: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              7
                        Start:              2023-04-08 19-55-51
                        End:                2023-04-08 19-55-51
                        Duration:           0.003555115 seconds
                        Tags:               []
                        
                        PASS:               0
                        FAIL:               0
                        ERROR:              7
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                7
                        LOGGER:             0
                        UNEXPECTED:         0

10. Logger errors


  • Developers can make errors while coding
  • General error exception messages will increase the time to find and fix these issues
  • test_rest_api provides short & precise error's with exact traceback info
  • Here we are purposefully making errors in Logger functions

Click here to view the html report

Console output

2023-04-08 21:54:56,023: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 21:54:56,023: Starting test setup
2023-04-08 21:54:56,023: Auto detecting test suites
2023-04-08 21:54:56,023: Total test suites: 1
2023-04-08 21:54:56,023: Auto detecting tests
2023-04-08 21:54:56,023: Total synchronous tests: 0
2023-04-08 21:54:56,023: Total asynchronous tests: 7
2023-04-08 21:54:56,023: Total tests: 7
2023-04-08 21:54:56,023: Created aiohttp client session
2023-04-08 21:54:56,023: Completed test setup
2023-04-08 21:54:56,023: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 21:54:56,023: ERROR   t1 (root file) [1]
2023-04-08 21:54:56,025: ERROR   t2 (root file) [2]
2023-04-08 21:54:56,025: ERROR   t3 (root file) [3]
2023-04-08 21:54:56,026: ERROR   t4 (root file) [4]
2023-04-08 21:54:56,026: ERROR   t5 (root file) [5]
2023-04-08 21:54:56,026: ERROR   t6 (root file) [6]
2023-04-08 21:54:56,026: ERROR   t7 (root file) [7]
2023-04-08 21:54:56,027: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              7
                        Start:              2023-04-08 21-54-56
                        End:                2023-04-08 21-54-56
                        Duration:           0.003578533 seconds
                        Tags:               []
                        
                        PASS:               0
                        FAIL:               0
                        ERROR:              7
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             7
                        UNEXPECTED:         0

11. Unexpected errors


Click here to view the html report

  • Developers can make errors while coding
  • General error exception messages will increase the time to find and fix these issues
  • test_rest_api provides short & precise error's with exact traceback info
  • Here we are purposefully making unexpected errors in tests

Console output

2023-04-08 21:58:04,529: 
                          =======================================================
                        || ..................................................... ||
                        || ..................................................... ||
                        || ...........  T E S T - R E S T - A P I   ............ ||
                        || ..................................................... ||
                        || ..................................................... ||
                          =======================================================
                        
2023-04-08 21:58:04,529: Starting test setup
2023-04-08 21:58:04,530: Auto detecting test suites
2023-04-08 21:58:04,530: Total test suites: 1
2023-04-08 21:58:04,530: Auto detecting tests
2023-04-08 21:58:04,530: Total synchronous tests: 0
2023-04-08 21:58:04,530: Total asynchronous tests: 6
2023-04-08 21:58:04,530: Total tests: 6
2023-04-08 21:58:04,531: Created aiohttp client session
2023-04-08 21:58:04,531: Completed test setup
2023-04-08 21:58:04,531: 
                          =======================================================
                        || ............... A S Y N C - T E S T S ............... ||
                          =======================================================
2023-04-08 21:58:04,531: ERROR   t1 (root file) [1]
2023-04-08 21:58:04,532: ERROR   t2 (root file) [2]
2023-04-08 21:58:04,533: ERROR   t3 (root file) [3]
2023-04-08 21:58:04,534: ERROR   t4 (root file) [4]
2023-04-08 21:58:04,534: ERROR   t5 (root file) [5]
2023-04-08 21:58:04,534: ERROR   t6 (root file) [6]
2023-04-08 21:58:04,535: 
                          =======================================================
                        || ..................................................... ||
                        || ............  T E S T - S U M M A R Y   ............. ||
                        || ..................................................... ||
                          =======================================================
                        
                        Status:             FAIL
                        Tests:              6
                        Start:              2023-04-08 21-58-04
                        End:                2023-04-08 21-58-04
                        Duration:           0.003570664 seconds
                        Tags:               []
                        
                        PASS:               0
                        FAIL:               0
                        ERROR:              6
                        DISABLE:            0
                        SKIP:               0
                        
                        LOW:                0
                        MINOR:              0
                        MAJOR:              0
                        CRITICAL:           0
                        BLOCKER:            0
                        
                        REST API:           0
                        GLOBAL VARIABLES:   0
                        BUG:                0
                        LOGGER:             0
                        UNEXPECTED:         6

Author

License

Copyright Troy M Jose, 2023.

Distributed under the terms of the MIT license, test_rest_api is free and open source software.