By Kevin Hou
3 minute read
This blog post contains a number of useful code snippets, functions, and classes that will help with machine learning in Jupyter Notebooks. Specific usage instructions as well as dependencies
Choose a random sample of rows from a large dataframe. This is very useful when trying to reduce your training set for debugging purposes.
1indices = np.random.randint(10, size=2) 2smallerDF = df[indices,:] 3
It's very easy to extract or remove columns from a pandas dataframe using their built in indexing actions.
1# Extract these keys into their own data frame 2preserveKeys = ['x', 'y'] 3smallDF = df[preserveKeys] 4 5# Create a new dataframe without certain columns 6newDF = df.drop(columns=['z']) 7
This is helpful when converting a dataframe into a boolean dataframe in which a 1 indicates the presence of a value and a null indicates there was no value. It can turn any dataframe into a sort of "checkbox" which is helpful for certain types of data processing like collaborative filtering where the actual value doesn't matter.
1booleanDF = copy.deepcopy(df) # Deep copy so don't modify other DF 2 3# Convert 'np.nan' to 0's and everything else to 1's 4booleanDF = booleanDF.notnull().astype('int') 5 6# Replace all 0's with 'np.nan' 7booleanDF = booleanDF.replace(0, np.nan) 8
1# Merge 2 dataframes with the same rows (ie. add new columns) 2finalDF = pd.concat([df1, df2], sort=True) 3 4# Merge 2 dataframes with the same columns (ie. add more rows) 5finalDF = pd.concat([df1, df2], sort=True, axis=0) 6 7# Merge 2 dataframes by row and add new columns when appropriate 8finalDF = pd.concat([df1, df2], sort=True, axis=0, ignore_index=True) 9
It's often difficult to deal with abstract, seemingly black-box machine learning algorithms. What can help alleviate some of this stress is knowing what your data really looks like. Here's a few examples that will help you understand what's going on in your dataset.
1df.describe() 2 3df.head(5) # First 5 rows 4
Running df.describe() will print a table of all columns and their respective counts (how many non-null values in the column), mean, std (standard deviation), min, 25%, 50%, 75%, and max. Sometimes the row name doesn't get included. This can be fixed by passing in the argument "include='all'" as follows: df.describe(include='all').
Running df.head(n) will print the first n rows of your dataset and can give you a good understanding of the form of your data. While the describe() function is good at showing you basic distributions, the head() function will show you what your data actually looks like quickly and easily.