<jupyter_start><jupyter_text>Hierarchical Indexes on DataFrames and Series<jupyter_code># np.random.seed(12345)
# pd.options.display.max_rows = 20
# plt.rc('figure', figsize=(10, 6))<jupyter_output><empty_output><jupyter_text>Hierarchical Indexing* We briefly discussed hierarchical indexing in the context of `groupby`  * - Hierarchical indexing allows you to have multiple index levels on an axis- Despite the name (indexing), it applies to both indexes and columns<jupyter_code>prog_languages = pd.DataFrame({ "Course":["Python", "Rust", "Python", "Rust"], 
              "Nb_participants": [10, 30, 27, 18]})
prog_languages.groupby("Course").sum()
              
data = pd.Series(np.random.randn(9),
                 index=[['a', 'a', 'a', 'b', 'b', 'c', 'c', 'd', 'd'],
                        [1, 2, 3, 1, 3, 1, 2, 2, 3]])
data
data.index
data[('b', 1)]

data['b', 1]
data['b':'c']

data.loc[['b', 'd']]
data.loc[: , 2]<jupyter_output><empty_output><jupyter_text>Stacked vs. Unstacked Data* Hierarchical indexes represent how the data is often collected * Ex. if variables in a hospital are measured and recorded in different files```python    File 1    patient_ABC LDL 112    patient_ABC HDL 48    patient_CCX  LDL 112    patient_VDM  LDL 112    patient_ABC  VO2 112    patient_CCZ  RER 48    ...```* This format is preferred  when not all the variables are measured in all the patients  * I.e., using columns for the variables would result in a large number of empty cells  * You can convert back and forth between a data frame and a hierarchical index with `stack()` and `unstack()`<jupyter_code>x = data.unstack()
x
x.stack()<jupyter_output><empty_output><jupyter_text>Hierarchical indexes on rows and columns* In a `DataFrame`, both axes can have a hierarchical index.<jupyter_code>data = pd.DataFrame(np.arange(12).reshape((4, 3)),
                     index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                     columns=[['Ohio', 'Ohio', 'Colorado'],
                              ['Green', 'Red', 'Green']])
data
data.index.names = ['key1', 'key2']
data.columns.names = ['state', 'color']
data
print(data.index.get_level_values("key1"))
print("*" * 55)
print(data.index.get_level_values("key2"))<jupyter_output>Index(['a', 'a', 'b', 'b'], dtype='object', name='key1')
*******************************************************
Int64Index([1, 2, 1, 2], dtype='int64', name='key2')<jupyter_text>- What type of variable would the following return (DataFrame, Series, np.array,...)?```python    data['Ohio']````- How about the following?```python    frame['Ohio', 'Geeen']````<jupyter_code>print(type(data['Ohio']))
print("*" * 40)
data['Ohio']
print(type(data['Ohio', 'Green']))
data['Ohio', 'Green']<jupyter_output><class 'pandas.core.series.Series'><jupyter_text>- You can create a `MultiIndex` and assign it manually if needed<jupyter_code>data = pd.DataFrame(np.arange(12).reshape((4, 3)),
                     index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]])
data.index.names = ["letter", "number"]
data
pd.MultiIndex.from_arrays([['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']],
                       names=['state', 'color'])
data.columns = pd.MultiIndex.from_arrays([['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']],
                       names=['state', 'color'])
data<jupyter_output><empty_output><jupyter_text>Reordering and Sorting Levels- Swap levels takes two levels `numbers` or `names` and returns a new object with the levels interchanged<jupyter_code>data.index.names = ['key1', 'key2']
data
data.swaplevel('key1', 'key2')<jupyter_output><empty_output><jupyter_text>- You can also pass the level's index, instead of name<jupyter_code>data.sort_index(level=0, ascending=False)

data.swaplevel(0, 1).sort_index(level=0, ascending=False)
data
data.swaplevel('state', 'color', axis=1)<jupyter_output><empty_output><jupyter_text>Summary Statistics by Level- The summary statistics that we've seen before can also be applied by level- When you apply a function on the level, the remaining levels get `"squashed"`<jupyter_code>frame<jupyter_output><empty_output><jupyter_text>Setting and Resetting and Index* It' common to read in a data frame file and use one of its columns as an index  * Setting a column as an index is done using `set_index()`    * By default, the columns are removed from the `DataFrame`, but you can choose to leave them in   * `reset_index()` does the opposite; moves the index as a new column of the `DataFrame`    * Creates a new index from an `IndexRange()`<jupyter_code>data = pd.DataFrame({'a': range(7), 'b': range(7, 0, -1),
                      'c': ['one', 'one', 'one', 'two', 'two',
                            'two', 'two'],
                      'd': [0, 1, 2, 0, 1, 2, 3]})
data
data2 = data.set_index('b')
data2
data3 = data.set_index(['c', 'd'])
data3
data.set_index(['c', 'd'], drop=False)
data4 = data2.reset_index()
data4
frame4.index<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17165386170642943, 'fraction_numerical': 0.022442377678932472, 'mean_word_length': 4.348108108108108, 'pattern_counts': {'":': 2, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '561124', 'license': 'Apache-2.0', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1601, 'n_words': 486, 'path': 'morea/Data_Wrangling_and_Pandas_2/resources/.ipynb_checkpoints/9_hierarchical_indexes-checkpoint.ipynb', 'repo_name': 'mahdi-b/ICS434-Sp2022', 'size': '4946'}