<jupyter_start><jupyter_text>Minimally Sufficient Pandas - All sources referenced from Ted Petrou Github repository<jupyter_code>import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/lalitgarg12/python-exercises/master/sample_data.csv',index_col=0)
df<jupyter_output><empty_output><jupyter_text>**Selection with the brackets**Placing a column name in the brackets appended to a DataFrame selects a single column of a DataFrame as a Series.<jupyter_code>df['state']<jupyter_output><empty_output><jupyter_text>**Selection with dot notation**Alternatively, you may select a single column with dot notation. Simply, place the name of the column after the dot operator. The output is the exact same as above.<jupyter_code>df.state<jupyter_output><empty_output><jupyter_text>**Issues with the dot notation**There are three issues with using dot notation. It doesn’t work in the following situations:- When there are spaces in the column name- When the column name is the same as a DataFrame method- When the column name is a variable<jupyter_code>df.favorite food<jupyter_output><empty_output><jupyter_text>You can only use the brackets to select columns with spaces.<jupyter_code>df['favorite food']<jupyter_output><empty_output><jupyter_text>**The column name is the same as a DataFrame method**When a column name and a DataFrame method collide, Pandas will always reference the method and not the column name. For instance, the column namecount is a method and will be referenced when using dot notation. This actually doesn’t produce an error as Python allows you to reference methods without calling them. Let’s reference this method now.<jupyter_code>df.count<jupyter_output><empty_output><jupyter_text>Regardless, it’s clear that using dot notation did not select a single column of the DataFrame as a Series. Again, you must use the brackets when selecting a column with the same name as a DataFrame method.<jupyter_code>df['count']<jupyter_output><empty_output><jupyter_text>**Guidance: Use the brackets for selecting a column of data**The dot notation provides no additional functionality over the brackets and does not work in all situations. Therefore, I never use it. Its single advantage is three fewer keystrokes. **Performance comparison iloc vs iat vs NumPy**Let’s compare the perfomance of selecting a single cell with iloc, iat and a NumPy array. Here we create a NumPy array with 100k rows and 5 columns containing random data. We then create a DataFrame out of it and make the selections.<jupyter_code>import numpy as np
a = np.random.rand(10 ** 5, 5)
df1 = pd.DataFrame(a)
row = 50000
col = 3
%timeit df1.iloc[row, col]
%timeit df1.iat[row, col]
%timeit a[row, col]<jupyter_output>The slowest run took 70.95 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 176 ns per loop<jupyter_text>**Guidance:** Use **NumPy** arrays if your application relies on performance for selecting a single cell of data and **not at or iat**. **Method Duplication**There are multiple methods in Pandas that do the exact same thing. Whenever two methods share the same exact underlying functionality, we say that they are aliases of each other. Having duplication in a library is completely unnecessary, pollutes the namespace and forces analysts to remember one more bit of information about a library.This next section covers several instances of duplication along with other instances of methods that are very similar to one another. **read_csv vs read_table duplication**<jupyter_code>college = pd.read_csv('https://raw.githubusercontent.com/lalitgarg12/python-exercises/master/college.csv')
college.head()
college2 = pd.read_table('https://raw.githubusercontent.com/lalitgarg12/python-exercises/master/college.csv', delimiter=',')
college.equals(college2)<jupyter_output><empty_output><jupyter_text>**Guidance: Only use read_csv to read in delimitted text files** **isna vs isnull and notna vs notnull**The isna and isnull methods both determine whether each value in the DataFrame is missing or not. The result will always be a DataFrame (or Series) of all boolean values.<jupyter_code>college_isna = college.isna()
college_isnull = college.isnull()
college_isna.equals(college_isnull)<jupyter_output><empty_output><jupyter_text>**Guidance: Use only isna and notna**<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07228915662650602, 'fraction_numerical': 0.009547624460104568, 'mean_word_length': 6.166123778501628, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '381284', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1296, 'n_tokens_neox': 1236, 'n_words': 524, 'path': 'Minimal_Sufficient_Pandas.ipynb', 'repo_name': 'lalitgarg12/Minimally-Sufficient-Pandas', 'size': '4399'}