Data Quality Framework
Weiser is a data quality framework designed to help you ensure the integrity and accuracy of your data. It provides a set of tools and checks to validate your data and detect anomalies. It also includes a dashboard to visualize the results of the checks.
To install Weiser, use the following command:
pip install weiser-aiConnections are defined at the datasources section in the config file see: examples/example.yaml.
Run checks in verbose mode:
weiser run examples/example.yaml -v
Compile checks only in verbose mode:
weiser compile examples/example.yaml -vcd weiser-ui
pip install -r requirements.txt
streamlit run app.py
Simple count check defintion
- name: test row_count
dataset: orders
type: row_count
condition: gt
threshold: 0Custom sql definition
- name: test numeric
dataset: orders
type: numeric
measure: sum(budgeted_amount::numeric::float)
condition: gt
threshold: 0Target multiple datasets with the same check definition
- name: test row_count
dataset: [orders, vendors]
type: row_count
condition: gt
threshold: 0Check individual group by values in a check
- name: test row_count groupby
dataset: vendors
type: row_count
dimensions:
- tenant_id
condition: gt
threshold: 0Time aggregation check with granularity
- name: test numeric gt sum yearly
dataset: orders
type: sum
measure: budgeted_amount::numeric::float
condition: gt
threshold: 0
time_dimension:
name: _updated_at
granularity: yearCustom SQL expression for dataset and filter usage
- name: test numeric completed
dataset: >
SELECT * FROM orders o LEFT JOIN orders_status os ON o.order_id = os.order_id
type: numeric
measure: sum(budgeted_amount::numeric::float)
condition: gt
threshold: 0
filter: status = 'FULFILLED'Missing values check
- name: customer data quality
dataset: orders
type: not_empty
dimensions: ["customer_id", "product_id", "order_date"]
condition: le
# Allow up to 5 NULL values per dimension
threshold: 5
filter: "status = 'active'"Anomaly detection check
- name: test anomaly
# anomaly test should always target metrics metadata dataset
dataset: metrics
type: anomaly
# References Orders row count.
check_id: c5cee10898e30edd1c0dde3f24966b4c47890fcf247e5b630c2c156f7ac7ba22
condition: between
# long tails of normal distribution for Z-score.
threshold: [-3.5, 3.5]We welcome contributions!
This project is licensed under the Apache 2.0 License. See the LICENSE file for more details.
