How to use pandas_cub

The README.ipynb notebook will serve as the documentation and usage guide to pandas_cub.

Installation

pip install pandas-cub

What is pandas_cub?

pandas_cub is a simple data analysis library that emulates the functionality of the pandas library. The library is not meant for serious work. It was built as an assignment for one of Ted Petrou's Python classes. If you would like to complete the assignment on your own, visit this repository. There are about 40 steps and 100 tests that you must pass in order to rebuild the library. It is a good challenge and teaches you the fundamentals of how to build your own data analysis library.

pandas_cub functionality

pandas_cub has limited functionality but is still capable of a wide variety of data analysis tasks.

• Subset selection with the brackets
• Arithmetic and comparison operators (+, -, <, !=, etc...)
• Aggregation of columns with most of the common functions (min, max, mean, median, etc...)
• Grouping via pivot tables
• String-only methods for columns containing strings
• Reading in simple comma-separated value files
• Several other methods

pandas_cub DataFrame

pandas_cub has a single main object, the DataFrame, to hold all of the data. The DataFrame is capable of holding 4 data types - booleans, integers, floats, and strings. All data is stored in NumPy arrays. panda_cub DataFrames have no index (as in pandas). The columns must be strings.

Missing value representation

Boolean and integer columns will have no missing value representation. The NumPy NaN is used for float columns and the Python None is used for string columns.

Code Examples

pandas_cub syntax is very similar to pandas, but implements much fewer methods. The below examples will cover just about all of the API.

Reading data with read_csv

pandas_cub consists of a single function, read_csv, that has a single parameter, the location of the file you would like to read in as a DataFrame. This function can only handle simple CSV's and the delimiter must be a comma. A sample employee dataset is provided in the data directory. Notice that the visual output of the DataFrame is nearly identical to that of a pandas DataFrame. The head method returns the first 5 rows by default.

import pandas_cub as pdc

df = pdc.read_csv('data/employee.csv')
df.head()

DataFrame properties

The shape property returns a tuple of the number of rows and columns

df.shape

(1535, 4)

The len function returns just the number of rows.

len(df)

1535

The dtypes property returns a DataFrame of the column names and their respective data type.

df.dtypes

The columns property returns a list of the columns.

df.columns

['dept', 'race', 'gender', 'salary']

Set new columns by assigning the columns property to a list.

df.columns = ['department', 'race', 'gender', 'salary']
df.head()

The values property returns a single numpy array of all the data.

df.values

array([['Houston Police Department-HPD', 'White', 'Male', 45279],
       ['Houston Fire Department (HFD)', 'White', 'Male', 63166],
       ['Houston Police Department-HPD', 'Black', 'Male', 66614],
       ...,
       ['Houston Police Department-HPD', 'White', 'Male', 43443],
       ['Houston Police Department-HPD', 'Asian', 'Male', 55461],
       ['Houston Fire Department (HFD)', 'Hispanic', 'Male', 51194]],
      dtype=object)

Subset selection

Subset selection is handled with the brackets. To select a single column, place that column name in the brackets.

df['race'].head()

Select multiple columns with a list of strings.

df[['race', 'salary']].head()

Simultaneously select rows and columns by passing the brackets the row selection followed by the column selection separated by a comma. Here we use integers for rows and strings for columns.

rows = [10, 50, 100]
cols = ['salary', 'race']
df[rows, cols]

You can use integers for the columns as well.

rows = [10, 50, 100]
cols = [2, 0]
df[rows, cols]

You can use a single integer and not just a list.

df[99, 3]

Or a single string for the columns

df[99, 'salary']

You can use a slice for the rows

df[20:100:10, ['race', 'gender']]

You can also slice the columns with either integers or strings

df[20:100:10, :2]

df[20:100:10, 'department':'gender']

You can do boolean selection if you pass the brackets a one-column boolean DataFrame.

filt = df['salary'] > 100000
filt.head()

df[filt].head()

df[filt, ['race', 'salary']].head()

Assigning Columns

You can only assign an entire new column or overwrite an old one. You cannot assign a subset of the data. You can assign a new column with a single value like this:

df['bonus'] = 1000
df.head()

You can assign with a numpy array the same length as a column.

import numpy as np
df['bonus'] = np.random.randint(100, 5000, len(df))
df.head()

You can assign a new column with a one column DataFrame.

df['salary'] + df['bonus']

df['total salary'] = df['salary'] + df['bonus']
df.head()

Arithmetic and comparison operators

df1 = df[['salary', 'bonus']] * 5
df1.head()

df1 = df[['salary', 'bonus']] > 100000
df1.head()

df1 = df['race'] == 'White'
df1.head()

Aggregation

Most of the common aggregation methods are available. They only work down the columns and not across the rows.

df.min()

Columns that the aggregation does not work are dropped.

df.mean()

df.argmax()

df['salary'].argmin()

Check if all salaries are greater than 20000

df1 = df['salary'] > 20000
df1.all()

Count the number of non-missing values

df.count()

Get number of unique values.

df.nunique()

Non-Aggregating Methods

These are methods that do not return a single value.

Get the unique values of each column. The unique method returns a list of DataFrames containing the unique values for each column.

dfs = df.unique()

dfs[0]

dfs[1]

dfs[2]

Rename columns with a dictionary.

df.rename({'department':'dept', 'bonus':'BONUS'}).head()

Drop columns with a string or list of strings.

df.drop('race').head()

df.drop(['race', 'gender']).head()

Non-aggregating methods that keep all columns

The next several methods are non-aggregating methods that return a DataFrame with the same exact shape as the original. They only work on boolean, integer and float columns and ignore string columns.

Absolute value

df.abs().head()

Cumulative min, max, and sum

df.cummax().head()

Clip values to be within a range.

df.clip(40000, 60000).head()

Round numeric columns

df.round(-3).head()

Copy the DataFrame

df.copy().head()

Take the nth difference.

df.diff(2).head(10)

Find the nth percentage change.

df.pct_change(2).head(10)

Sort the DataFrame by one or more columns

df.sort_values('salary').head()

Sort descending

df.sort_values('salary', asc=False).head()

Sort by multiple columns

df.sort_values(['race', 'salary']).head()

Randomly sample the DataFrame

df.sample(n=3)

Randomly sample a fraction

df.sample(frac=.005)

Sample with replacement

df.sample(n=10000, replace=True).head()

String-only methods

Use the str accessor to call methods available just to string columns. Pass the name of the string column as the first parameter for all these methods.

df.str.count('department', 'P').head()

df.str.lower('department').head()

df.str.find('department', 'Houston').head()

Grouping

pandas_cub provides the value_counts method for simple frequency counting of unique values and pivot_table for grouping and aggregating.

The value_counts method returns a list of DataFrames, one for each column.

dfs = df[['department', 'race', 'gender']].value_counts()

dfs[0]

dfs[1]

dfs[2]

If your DataFrame has one column, a DataFrame and not a list is returned. You can also return the relative frequency by setting the normalize parameter to True.

df['race'].value_counts(normalize=True)

The pivot_table method allows to group by one or two columns and aggregate values from another column. Let's find the average salary for each race and gender. All parameters must be strings.

df.pivot_table(rows='race', columns='gender', values='salary', aggfunc='mean')

If you don't provide values or aggfunc then by default it will return frequency (a contingency table).

df.pivot_table(rows='race', columns='gender')

You can group by just a single column.

df.pivot_table(rows='department', values='salary', aggfunc='mean')

df.pivot_table(columns='department', values='salary', aggfunc='mean')