pyviztest
A Visual Test Automation Library in Python. Works with Playwright and Selenium. Integrable with Allure Report.
Why Project pyviztest?
Forewords
In Python there was no robust and integrable Visual Test Automation library/module. Thankfully, Kumar Aaditya created a pytest fixture for Playwright using the power of pixelmatch library for Visual Testing, and Symon Storozhenko enhanced it to create pytest-playwright-visual. I really loved both, so I tried to realign the same concept by developing a library and added a few more functionality to make the use of it more flexible, and also compatible with Playwright and Selenium. All thanks go to both of them for being the harbinger of the visual testing libraries in python. I merely followed the path and improvized.
Features
So, previously created pytest-playwright-visual had the power to be summoned anywhere inside the tests which had it as a parameter, and to
-
- Create Golden Snapshot (The reference image of the UI which needs to be compared with during test execution) of the AUT(Applicayion under test).
-
- Update existing Golden Snapshot.
-
- Take snapshot in runtime and compare it with the Golden Snapshot. If both do not match, it creates an image where the difference is highlighted.
In addition to these features, pyviztest is capable of doing a few more things!
-
- Supports Windows, Linux and MacOS systems.
-
- Supports Playwright and Selenium. You just need to provide the Page and WebDriver instances.
-
- Supports custom snapshot directories.
-
- Supports option to Create & Update snapshots(particular and all) without failing the tests.
-
- Supports option to add multiple snapshot validation in a single test case, and if one of them fails it will still validate others.
-
- Supports integration with Allure Report.
-
- Supports auto-naming of the snapshots with respect to the test suite and test case name.
-
- Provides api to return Golden, Actual and Difference Snapshots in bytes, to easily integrate with any custom report.
-
- Supports comparing images with different in sizes without failing the test abruptly.
-
- Supports comparing snapshots of multiple web elements together or the full page.
Upcoming Next:
-
- Support for Appium.
-
- Support for element specific identification.
-
- A fixture along with the usual library.
How To Use It:
Installing
Simple and easy to install because it's just another python library! Install using below pip command:
$ pip install pyviztestConfiguration
- Required: Import VisualTest from pyviztest.viztest
from pyviztest.viztest import VisualTest- Required: The object of VisualTest is the driving factor of the visual tests.
visualtest = VisualTest(driverpage=driver)Note You may directly create an object of VisualTest class for each test separately and proceed with that object, or you can create a pytest fixture for dependency injection to initialize the VisualTest instance from inside that class. Using a separate class/fixture to implement dependency injection is always recommended. The constructor of VisualTest class can take 5 arguments i.e.
snapshot_path, [MANDATORY]driverpage: WebDriver/Page,updatesnapshot,savefailuresnapondisk&allurereport. The usage of them are elaborated in below code snippet comments.
class VisualTest(
snapshot_path: str = '', #The path where all the snapshots will reside. If not provided, it will consider the parent directory of the test class as the snapshot path by default.
driverpage: Any = '', #You MUST to provide either the driver object (if you are using Selenium for your tests) or page fixture/object (if you are using Playwright)
updatesnapshot: bool = False, # If True, it will update all the existing golden snapshots or to create new golden snapshots for the current session of Visualtest class.
savefailuresnapondisk: bool = True, #If True, all the failure snapshots will be saved inside the failure directories under snapshot_path as image files in PNG format.
allurereport: bool = False #If True, allure report will be generated with Golden snapshots and failure snapshots.
)- Required: Invoke the setpaths() function for each of the tests.
visualtest.setpaths()Note The setpaths function can take 2 optional arguments i.e.
updatesnapshotandnumberofclassesaftertestclass. The usage of them are elaborated in below code snippet comments.
(method) def setpaths(
updatesnapshot: bool = False, # Same as the updatesnapshot mentioned above. It provides the flexibility to update snapshots for a particular test case. If not given, it would take the default value during creation of VisualTest class object.
numberofclassesaftertestclass: int = 0 # This parameter defines the number of classes between the test methods and where the visualtest_web() method is invoked.
) -> NoneWarning
numberofclassesaftertestclassparameter helps you to fetch the testcase name in order to generate the snapshot's name by default if you are not providing any specific name. By default the value of it is 0, that means it assumes that the visualtest_web() method is being invoked from inside the test mthod directly without any further abstraction. If you add N number of abstractions, then you should provide the value of the parameters as N. See point no. 4 for more info.
visualtest.setpaths(numberofclassesaftertestclass=N)- Now that the configuration is done, you have to invoke
visualtest_web()method wherever the webpage snapshots need to be validated.
assert vt.visualtest_web(stepname="validateloginpage")Note This method takes a minimum of 7 arguments i.e.
stepname,threshold,fail_fast,updatesnapshot,fullpage,snapshot_of_locatorsandexclude_locators. The usage of them are elaborated in below code snippet comments.
(method) def visualtest_web(
*,
stepname: str = '', # Optional, but it is recommended to provide the value to distinguish snapshots
threshold: float = 0.1, # Optional, sets the threshold for the comparing the snapshots 0 to 1
fail_fast: bool = False, # Optional, if you'd like to fail the comparison even if 1 pixel doesn't match, make it True
updatesnapshot: bool = False, # Same as the updatesnapshot mentioned above. It provides the flexibility to update snapshots for a particular test step. If not given, it would take the default value during creation of VisualTest class object or setpaths().
fullpage: bool = True, # Optional, to take snapshot of the full page or not
snapshot_of_locators: list = [], # Optional, you may send set of locators or webelements to take snapshots of only those elements instead of a full page snapshot
exclude_locators: list = [] # Optional, you may send set of locators to mask them in the snapshots, so during the comparison they will be excluded.
) -> boolWarning
fullpageandexclude_locatorsare only applicable for Playwright tests. They do not work for Selenium tests. Note Thisvisualtest_web()method returns a boolean value as per below table:
| updatesnapshot(VisualTest()) | updatesnapshot(setpaths()) | updatesnapshot(visualtest_web()) | Snapshots Match? | visualtest_web() |
|---|---|---|---|---|
| True | X | X | X | True |
| False | True | X | X | True |
| False | False | True | X | True |
| False | False | False | True | True |
| False | False | False | False | False |
Warning
Do not Change the value of
numberofclassesaftertestclassif you don't understand the below concept well.As mentioned above already,
numberofclassesaftertestclassparameter helps you to fetch the testcase name in order to generate the snapshot's name by default if you are not providing any specific name. Let's look at an example:
Project|
|---src|
| |--testsuitename|
| |---test_playwright.py|
| | |---test_1()
| | |---test_2()
| |
| |---test_selenium.py|
| |---test_1()
| |---test_2()
|---VisualTestResults In case of the folder structure depicted above, if the OS is Linux,
snapshot_pathisVisualTestResults, and thevisualtest_web()method is invoked from thetest_1()andtest_2()methods, then the name of the Golden snapshot created fortest_1()intest_playwright.pywill belinux_test_1.pngand it will reside inside VisualTestResults/Golden_Snapshots/test_playwright/test_1/linux_test_1.png. So after the execution ifupdatesnapshot= True, the folder structure will be like below:
Project|
|---src|
| |--testsuitename|
| |---test_playwright.py|
| | |---test_1()
| | |---test_2()
| |
| |---test_selenium.py|
| |---test_1()
| |---test_2()
|---VisualTestResults|
|---Golden_Snapshots|
|---test_playwright|
|---test_1|
| |---linux_test_1.png
|
|---test_2|
|---linux_test_2.png Now, let's look at a different folder structure below:
Project|
|---src|
| |--testsuitename|
| |---test_playwright.py|
| | |---test_1()|
| | | |--validate_launchpage()|
| | | |--validate_loginbutton()
| | |---test_2()|
| | |--validate_loginpage()|
| | |--validate_homepage()
| |
| |---test_selenium.py|
| |---test_1()
| |---test_2()
|---VisualTestResults Here, the
visualtest_web()method is invoked fromvalidate_loginbutton()andvalidate_homepage()methods instead of thetest_1()andtest_2()methods. then the name of the Golden snapshot created forvalidate_loginbutton()will belinux_validate_loginbutton.pngand it will reside inside VisualTestResults/Golden_Snapshots/validate_launchpage/validate_loginbutton/linux_validate_loginbutton.png. Now clearly, if we want the name of the test folders to be of the test method's name, then we should initializenumberofclassesaftertestclass= 2. (2 because there are 2 invoked methods between the test methodtest_1()andvalidate_loginbutton()).
visualtest.setpaths(numberofclassesaftertestclass=2)Example Code
Playwright:
from playwright.sync_api import Page
import pytest
from pyviztest.viztest import VisualTest
@pytest.fixture(scope="function", autouse=True)
def before_each(page:Page):
page.goto(url=url)
yield
def test_playwright_updatesnapshot(page:Page) -> None:
# Create an object of VisualTest class
# with parameters ->
# @snapshotpath where all the Golden and other snapshots will be stored
# @driverpage where you have to provide the page fixture of Playwright
# @updatesnapshot a boolean value, make it True if you'd like to update all the snapshots
# @savefailuresnapondisk a boolean value, make it True if you'd like to save all the snapshots as image files
# @allurereport boolean value, make it True if you'd like to generate allure report with the snapshots
vt = VisualTest(snapshot_path=visualtestresults, driverpage=page, updatesnapshot=False,
savefailuresnapondisk=False, allurereport=True)
# setpaths() function must to be invoked before summoning any visualtest method
# @updatesnapshot parameter will override the default updatesnapshot value set during
# VisualTest object creation
vt.setpaths(updatesnapshot=True)
page.wait_for_load_state(state='networkidle')
page.wait_for_selector(id_header_productstore)
assert page_title in page.url
# For visual testing for any Web app you need to invoke visualtest_web() function
assert vt.visualtest_web(stepname="launchpage")Selenium:
from playwright.sync_api import Page
import pytest
from pyviztest.viztest import VisualTest
@pytest.fixture (scope="function", autouse=True)
def browser():
driver = webdriver.Chrome()
driver.get(url=url)
yield driver
driver.quit()
def test_selenium_updatesnapshot(browser) -> None:
# Create an object of VisualTest class
# with parameters ->
# @snapshotpath where all the Golden and other snapshots will be stored
# @driverpage where you have to provide the page fixture of Playwright
# @updatesnapshot a boolean value, make it True if you'd like to update all the snapshots
# @savefailuresnapondisk a boolean value, make it True if you'd like to save all the snapshots as image files
# @allurereport boolean value, make it True if you'd like to generate allure report with the snapshots
vt = VisualTest(snapshot_path=visualtestresults, driverpage=browser, updatesnapshot=False,
savefailuresnapondisk=False, allurereport=True)
# setpaths() function must to be invoked before summoning any visualtest method
# @updatesnapshot parameter will override the default updatesnapshot value set during
# VisualTest object creation
vt.setpaths(updatesnapshot=True)
headerelement = browser.find_element(By.XPATH, xpath_header_productstore)
assert page_title in browser.current_url
assert headerelement.is_displayed()
# For visual testing for any Web app you need to invoke visualtest_web() function
assert vt.visualtest_web(stepname="launchpage")Execution
First Time:
When the Golden Snapshots (the reference snapshots with which the execution snapshots will be compared) are not existing, then you have to create them first. So make the value of updatesnapshot as True and call the pytest command in command line interface.
$ pytestAfter the execution, you will see that the Golden Snaphots will be generated in your project directory where you have set your snapshot_path.
Second Time onwards/For snapshot validation:
Make the value of updatesnapshot as False and call the pytest command in command line interface.
$ pytestAfter the execution, if the value of savefailuresnapondisk is True, you will see that the Failure Snaphots will be generated in your project directory where you have set your snapshot_path. If no test case fails, no Failure Snapshot would be generated. If the value of savefailuresnapondisk is False, Snapshot files will not be generated.
Now, if allurereport = True, then you can generate allure report but you have to run below command:
$ pytest --alluredir=allure-resultsfollowed by
$ allure serve .\allure-results\Then you will be able to see the generated allure reports with pass and failure reports with mismatch snapshots.
To Update Golden Snapshots:
If you want to update the Golden Snapshots of all the validations in a single VisualTest() session, then make the value of updatesnapshot as True while creating a VisualTest() object. You'd have the option to do the same for setpaths() and visualtest_web methods as well. Then call the pytest command in command line interface.
$ pytestAfter the execution, the Golden Snapshots of the targeted test cases will be replaced with new snapshots.
