<jupyter_start><jupyter_text>Miscellaneous Subset SelectionIn this chapter, a few more methods for subset selection are described. The methods used in this chapter do not add any additional functionality to pandas, but are covered for completeness.I personally do not use the methods described in this chapter and suggest that also avoid them. They are all valid syntax and some pandas users do actually use them so you may find them valuable. Selecting a column with dot notationThe best way to select a single column from a DataFrame as a Series is by placing the name of the column within *just the brackets*. There's actually an alternative way to select a single column of data and that is with dot notation. Let's read in the the `sample_data2.csv` dataset.<jupyter_code>import pandas as pd
df = pd.read_csv('../data/sample_data2.csv')
df<jupyter_output><empty_output><jupyter_text>Place the name of the column directly after the name dot as if it were an attribute.<jupyter_code>df.name.head()<jupyter_output><empty_output><jupyter_text>This produces an identical result as using *just the brackets*.<jupyter_code>df['name'].head()<jupyter_output><empty_output><jupyter_text>Avoid dot notation - use just the bracketsAlthough this method for column selection requires less syntax and is used by many pandas users, it has many downsides. The following is a partial list of the functionality that is impossible using dot notation.* Select column names with spaces* Select column names that have the same name as methods* Select columns with variablesExamples of each scenario will now be covered. Using dot notation does not allow you to select columns with spaces. Selecting the column `average score` raises a syntax error.<jupyter_code>df.average score<jupyter_output><empty_output><jupyter_text>The only way to select this column is with *just the brackets*.<jupyter_code>df['average score']<jupyter_output><empty_output><jupyter_text>Dot notation is unable to select columns that are the same name as methods. For instance, `max` is a method that all DataFrames have. In this particular DataFrame, it also the name of the column. Attempting to select it via dot notation will access the method.<jupyter_code>df.max<jupyter_output><empty_output><jupyter_text>Again, the only way to select this column is with *just the brackets*.<jupyter_code>df['max']<jupyter_output><empty_output><jupyter_text>Dot notation is unable to select a column using a variable name. Let's say we assign the variable `col` to the string 'name' which is the name of the first column. Attempting to select it via dot notation raises an error.<jupyter_code>col = 'name'
df.col<jupyter_output><empty_output><jupyter_text>Once again, use *just the brackets*.<jupyter_code>df[col]<jupyter_output><empty_output><jupyter_text>Slicing rows with just the bracketsSo far, we have covered three ways to select subsets of data with *just the brackets*.  You can use a single string, a list of strings, or a boolean Series. Let's quickly review those ways right now using the bikes dataset.<jupyter_code>bikes = pd.read_csv('../data/bikes.csv')<jupyter_output><empty_output><jupyter_text>A single string<jupyter_code>bikes['tripduration'].head(3)<jupyter_output><empty_output><jupyter_text>A list of strings<jupyter_code>cols = ['trip_id', 'tripduration']
bikes[cols].head(3)<jupyter_output><empty_output><jupyter_text>A boolean SeriesThe previous two examples selected columns. Boolean Series select rows.<jupyter_code>filt = bikes['tripduration'] > 5000
bikes[filt].head(3)<jupyter_output><empty_output><jupyter_text>Using a sliceIt is possible to use slice notation within just the brackets. For example, the following selects the rows beginning at location 2 up to location 10 with a step size of 3. You can even use slice notation when the index is strings.<jupyter_code>bikes[2:10:3]<jupyter_output><empty_output><jupyter_text>I do not recommend using slicing with *just the brackets*Although slicing with *just the brackets* seems simple, I do not recommend using it. This is because it is ambiguous and can make selections either by integer location or by label. I always prefer explicit, unambiguous methods. Both `loc` and `iloc` are unambiguous and explicit. Meaning that even without knowing anything about the DataFrame, you would be able to explain exactly how the selection will take place. If you do want to slice the rows, then use `loc` if you are using labels or `iloc` if you are using integer location, but do not use *just the brackets*. Selecting a single cell with `at` and `iat`pandas provides two more rarely seen indexers, `at`, and `iat`. These indexers are analogous to `loc` and `iloc` respectively, but only select a single cell of a DataFrame. Since they only select a single cell, you must pass both a row and column selection as either a label (`loc`) or an integer location (`iloc`). Let's see an example of each.<jupyter_code>bikes.at[40, 'temperature']
bikes.iat[-30, 5]<jupyter_output><empty_output><jupyter_text>The current index labels for `bikes` is integers which is why the number 40 was used above. It is the label for a row, but also happens to be an integer. What's the purpose of these indexers?All usages of `at` and `iat` may be replaced with `loc` and `iloc` and would produce the exact same results. These `at` and `iat` indexers are optimized to select a single cell of data and therefore provide slightly better performance than `loc` or `iloc`. Let's verify this below.<jupyter_code>bikes.loc[40, 'temperature']
bikes.iloc[-30, 5]<jupyter_output><empty_output><jupyter_text>I never use these indexersPersonally, I never use these specialty indexers as the performance advantage for a single selection is minor. It would require a case where single element selections were happening in great numbers to see any significant improvement and doing so is rare in data analysis. Much bigger performance improvement using numpy directlyIf you truly wanted a large performance improvement for single-cell selection, you would select directly from numpy arrays and not a pandas DataFrame. Below, the data is extracted into the underlying numpy array with the `values` attribute. We then time the performance for selecting with numpy and also with `iat` and `iloc` on a DataFrame. On my machine, `iat` shows a 30-40% improvement over `iloc`, but selecting with numpy is about 50x as fast. There is no comparison here, if you care about performance for selecting a single cell of data, use numpy.<jupyter_code>values = bikes.values
%timeit -n 5 values[-30, 5]
%timeit -n 5 bikes.iat[-30, 5]
%timeit -n 5 bikes.iloc[-30, 5]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0671966401679916, 'fraction_numerical': 0.007049647517624119, 'mean_word_length': 5.681362725450902, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '279316', 'license': 'MIT', 'n_tokens_mistral': 1892, 'n_tokens_neox': 1781, 'n_words': 895, 'path': 'jupyter_notebooks/pandas/mastering_data_analysis/02. Selecting Subsets of Data/09. Miscellaneous Subset Selection.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '6667'}