<jupyter_start><jupyter_text>This tutorial will walk you through the essentials of how to index & filter data with Pandas. Think of it as a greatly condensed, opinionated, version of [the official indexing documentation.](http://pandas.pydata.org/pandas-docs/stable/indexing.html).We'll start by loading Pandas and the data:<jupyter_code>import pandas as pd
df = pd.read_csv('parks.csv', index_col=['Park Code'])
df.head(3)<jupyter_output><empty_output><jupyter_text>Indexing: Single RowsThe simplest way to access a row is to pass the row number to the `.iloc` method. Note that first row is zero, just like list indexes.<jupyter_code>df.iloc[2]<jupyter_output><empty_output><jupyter_text>The other main approach is to pass a value from your dataframe's index to the `.loc` method:<jupyter_code>df.loc['BADL']<jupyter_output><empty_output><jupyter_text>Indexing: Multiple RowsIf we need multiple rows, we can pass in multiple index values. Note that this changes the order of the results!<jupyter_code>df.loc[['BADL', 'ARCH', 'ACAD']]
df.iloc[[2, 1, 0]]<jupyter_output><empty_output><jupyter_text>Slicing the dataframe just as if it were a list also works.<jupyter_code>df[:3]
df[3:6]<jupyter_output><empty_output><jupyter_text>Indexing: ColumnsWe can access a subset of the columns in a dataframe by placing the list of columns in brackets like so:Note - Park Code is the index<jupyter_code>df['State'].head(3)<jupyter_output><empty_output><jupyter_text>You can also access a single column as if it were an attribute of the dataframe, but only if the name has no spaces, uses only basic characters, and doesn't share a name with a dataframe method. So, `df.State` works:<jupyter_code>df.State.head(3)
df.columns<jupyter_output><empty_output><jupyter_text>> but `df.Park Code` will fail as there's a space in the name:<jupyter_code>df.Park Code<jupyter_output><empty_output><jupyter_text>We can only access the 'Park Code' column by passing its name as a string in brackets, like `df['Park Code']`. I recommend either always using that approach or always converting your column names into a valid format as soon as you read in the data so that you don't have to mix the two methods. It's just a bit tidier.1) It's a good practice to clean your column names to prevent this sort of error. I'll use a very short cleaning function here since the names don't have any odd characters. By convention, the names should also be converted to **lower case**. Pandas is case sensitive, so future calls to all of the columns will need to be updated.<jupyter_code>df.columns = [col.replace(' ', '_').lower() for col in df.columns]
print(df.columns)<jupyter_output>Index(['park_name', 'state', 'acres', 'latitude', 'longitude'], dtype='object')<jupyter_text>Indexing: Columns and RowsIf we need to subset by both columns and rows, you can stack the commands we've already learned.<jupyter_code>df[['state', 'acres']][:3]<jupyter_output><empty_output><jupyter_text>Indexing: Scalar ValuesAs you may have noticed, everything we've tried so far returns a small dataframe or series. If you need a single value, simply pass in a single column and index value.<jupyter_code>val = df.state.iloc[2]
type(val)<jupyter_output><empty_output><jupyter_text>Note that you will get a different return type if you pass a single value in a list.<jupyter_code>val2 = df.state.iloc[[2]]
val2
type(val2)<jupyter_output><empty_output><jupyter_text>Selecting a Subset of the Data The main method for subsetting data in Pandas is called [boolean indexing](http://pandas.pydata.org/pandas-docs/stable/indexing.htmlboolean-indexing). First, let's take a look at what pandas does when we ask it to evaluate a boolean:<jupyter_code>(df.state == 'UT').head(3)<jupyter_output><empty_output><jupyter_text>We get a series of the results of the boolean. Passing that series into a dataframe gives us the subset of the dataframe where the boolean evaluates to `True`.<jupyter_code>df[df.state == 'UT']<jupyter_output><empty_output><jupyter_text>Some of the logical operators are different:- `~` replaces `not`- `|` replaces `or`- `&` replaces `and`If you have multiple arguments they'll need to be wrapped in parentheses. For example:<jupyter_code>df[(df.latitude > 60) | (df.acres > 10**6)].head(3)<jupyter_output><empty_output><jupyter_text>You can also use more complicated expressions, including lambdas.<jupyter_code>df[df['park_name'].str.split().apply(lambda x: len(x) == 3)].head(3)<jupyter_output><empty_output><jupyter_text>Let's break this down<jupyter_code>df1 = df['park_name'].str.split()
df1.head()
# Filter out rows that have exactly 3 elements in the list.
df2 = df1.apply(lambda x: len(x) == 3)
df2.head()<jupyter_output><empty_output><jupyter_text>Now apply this boolean mask to our dataframe<jupyter_code>df[df2].head()<jupyter_output><empty_output><jupyter_text>Notice how BIBE is missing .. It was masked out.  Key Companion Methods: `isin` and `isnull`These methods make it much easier and faster to perform some very common tasks. Suppose we wanted to find all parks on the West coast. `isin` makes that simple:<jupyter_code>df[df.state.isin(['WA', 'OR', 'CA'])].head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10546648638207456, 'fraction_numerical': 0.006567510141008306, 'mean_word_length': 6.323903818953324, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '608658', 'license': 'MIT', 'n_tokens_mistral': 1703, 'n_tokens_neox': 1636, 'n_words': 588, 'path': 'AccessingDataWithPandas.ipynb', 'repo_name': 'nachobot/DataScience', 'size': '5177'}