missingdata

missing data visualization and imputation

Goals

To provide an easy to use yet thorough assessment of missing values in one's dataset:

• in addition to the blackholes plot bellow,
• show the variable-to-variable, subject-to-subject co-missingness, and
• quantify the TYPE of missingness etc

Note

To easily manage your data with missing values etc, I strongly recommend you to move away from CSV files and start managing your data in self-contained flexible data structures like pyradigm, as your data, as well your needs, will only get bigger & more complicated e.g. with mixed-types, missing values and large number of groups.

These would be great contributions if you have time.

Features

• visualization
• imputation (coming!)
• other handling

blackholes plot

State

• Software is beta and under dev
• Contributions most welcome.

Installation

pip install -U missingdata

Usage

Let's say you have all the data in a pandas DataFrame, where subject IDs are in a 'sub_ids' column and variable names are in a 'var_names' column, and they belong to groups identified by sub_class and var_group, you can use the following code produce the blackholes plot:

from missingdata import blackholes

blackholes(data_frame,
           label_rows_with='sub_ids', label_cols_with='var_names',
           group_rows_by=sub_class, group_cols_by=var_group)

If you were interested in seeing subjects/variables with least amount of missing data, you can control miss perc window with filter_spec_samples and/or filter_spec_variables by passing a tuple of two floats e.g. (0, 0.1) which will filter away those with more than 10% of missing data.

from missingdata import blackholes

blackholes(data_frame,
           label_rows_with='sub_ids', label_cols_with='var_names',
           filter_spec_samples=(0, 0.1))

The other parameters for the function are self-explanatory.

Please open an issue if you find something confusing, or have feedback to improve, or identify a bug. Thanks.

Citation

If you find this package useful, I'd greatly appreciate if cite this package via:

Pradeep Reddy Raamana, (2019), "missingdata python library" (Version v0.1). Zenodo. http://doi.org/10.5281/zenodo.3352336
DOI: 10.5281/zenodo.3352336