Data Preprocessing Engine

A simple library that wraps common data processing tasks into an easy to use preprocessing engine. The library currently supports transformation of csv files loaded into Pandas dataframe.

Getting Started

In order to get started simply fetch the package using the following pip command:

pip install datapreprocessor

Prerequisites

DataPreprocessor is built on top of existing Python libraries and therefore requires them in order to work properly.

Here's the list of dependencies:

• numpy==1.13.3
• pandas==0.20.3
• python-dateutil==2.6.1
• pytz==2017.2
• scikit-learn==0.19.0
• scipy==0.19.1
• six==1.11.0
• sklearn==0.0

Usage

Here you can find examples of using DataPreprocessor for processing a dummy dataset.

First step is defining the metadata for the dataset columns. The metadata for the given dataset would be as following:

[FEATURE_TYPES]
Country = CATEGORIC
Age = NUMERIC
Salary = NUMERIC
Purchased = CATEGORIC

Handling missing data

One of the most common preprocessing tasks in machine learning is handling the missing data. DataProcessor handles missing numeric data by allowing you to exchange missing values with either MEAN or MEDIAN value of numeric columns.

import pandas as pd

from datapreprocessor.engine import PreprocessingEngine
from datapreprocessor.metadata import DatasetMetadataLoader
from datapreprocessor.missingdata import ImputeStrategy

# Load metadata
metadata = DatasetMetadataLoader("metadata.ini")
# Load the dataset
dataset = pd.read_csv('Data.csv')

# Process the dataset 
processedDataframe = PreprocessingEngine(dataset, metadata) \
    .imputeNumericDataUsing(ImputeStrategy.MEAN) \
    .getProcessedDataframe()

Executing the above code would result in a new dataframe that has the missing values for age and salary filled with their means as shown on next table.

• Handling missing categoric data such as Country is supported only by eliminating the rows with missing data using method

filteredDataframe = PreprocessingEngine(processedDataframe, metadata) \
          .filterMissingCategoricData()
          .getProcessedDataframe()

Encoding categoric columns

As shown in the following code, setting columnsToTransform to "Country" would eliminate that column and replace it with three columns: Germany, France and Spain.

encodedDataframe = PreprocessingEngine(processedDataframe, metadata) \
            .encodeCategoricData(columnsToTransform="Country") \
            .getProcessedDataframe()

The dataframe after encoding looks like shown in the next table.

Scaling the data

Scaling the dataframe can be performed using scaleDataset() method which accepts any sklearn scaler. In the following example, min-max scaling was used as strategy.

from sklearn.preprocessing import MinMaxScaler

 scaledDataframe = PreprocessingEngine(encodedDataframe, metadata) \
            .scaleDataset(MinMaxScaler()) \
            .getProcessedDataframe()

Moving columns

There are times when we want to move some columns to the rightmost column of the dataset. This can be done using shiftRight() method in PreprocessingEngine.

For example in the dataframe above we could move the Purchased column to right by executing the following code.

shiftedDataframe = PreprocessingEngine(encodedDataframe, metadata) \
            .shiftRight("Purchased") \
            .getProcessedDataframe()

The shiftedDataFrame would look like shown in the next table.

Transforming the dataset in one go

The transformation operations can be executed in one go just by calling them one after the other.

processedDataframe = PreprocessingEngine(dataset, metadata) \
            .imputeNumericDataUsing(ImputeStrategy.MEAN) \
            .filterMissingCategoricData() \
            .encodeCategoricData(columnsToTransform="Country") \
            .scaleDataset(MinMaxScaler()) \
            .getProcessedDataframe()

Running the tests

In order to run the tests clone this repository and execute following commands:

pip install -r requirements.txt
cd tests
python RunAllUnitTests.py

Authors

• Igor Jovic - jovicigor