<jupyter_start><jupyter_text>IntroductionA MultiIndex is an index with multiple, hierarchical levels. We use a multiindex for the column headers of a dataframe. This allows each column to have multiple keys associated with it--one key for each level. This makes it easy to associate multiple identifiers with a single column.  To show what this looks like we'll look at phosphoproteomics data in the endometrial cancer dataset.<jupyter_code># Throughout the tutorial we will be using the endometrial and colon dataset
import cptac
cptac.download(dataset="endometrial", version="latest")
cptac.download(dataset="colon", version="latest")
en = cptac.Endometrial()
phospho = en.get_phosphoproteomics()
phospho.head()<jupyter_output><jupyter_text>The first level of keys is the name of the gene associated with the column of data. The second level is the site of acetylation. Each column has its own Name and Site associated with it. Normally, as above, keys that repeat for adjacent columns are not displayed, to make it easier to read the dataframe. For example, in the table above the first 7 columns are all for the A2M gene, but the label is only shown for the first column.   Join functions with multiindices The join functions have been written to handle multiindices. More information on the join functions can be found in the joining_dataframes tutorial. An example of joining a multiindexed dataframe (in this case phosphoproteomics) with a non multiindexed dataframe (in this case CNV) is below.<jupyter_code>phospho_and_CNV = en.join_omics_to_omics(df1_name="CNV", df2_name="phosphoproteomics")
phospho_and_CNV.head()<jupyter_output>Warning: CNV data was not found for the following samples, so CNV data columns were filled with NaN for these samples: S105, S106, S107, S108, S109, S110, S111, S112, S113, S114, S115, S116, S117, S118, S119, S120, S121, S122, S123, S124, S125, S126, S127, S128, S129, S130, S131, S132, S133, S134, S135, S136, S137, S138, S139, S140, S141, S142, S143, S144, S145, S146, S147, S148, S149, S150, S151, S152, S153 (/Users/hannahboekweg/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py, line 1)<jupyter_text>Since the CNV dataframe doesn't have the level "Site", it is filled in with NANs, so that it can be joined to the phosphoproteomics dataframe.   How to select from multiindex  Selecting based on all levelsWe can select single columns by passing the proper keys for all levels of the multiindex. For example, to get the acetylproteomics for A2M at site K1176, we'd do the following:<jupyter_code>acetyl = en.get_acetylproteomics()
all_levels_selection = acetyl["A2M"]["K1176"]
all_levels_selection.head(10)<jupyter_output><empty_output><jupyter_text>Selecting based on one levelWe can easily select multiple columns from our multiindex dataframe, based on just the "Name" level of the multiindex:<jupyter_code>gene1_filter = acetyl.columns.get_level_values("Name").str.startswith("AA") # Select all columns where the gene starts with "AA". This will grab every column where the key "Name" starts with AA
gene1_data = acetyl.loc[:, gene1_filter]
gene1_data.head()<jupyter_output><empty_output><jupyter_text>Selecting based on a different level of the multiindexWe can also select based on one of the inner levels of the multiindex. For example, to get all sites from all genes where tyrosine is phosphorylated:<jupyter_code>y_site_filter = phospho.columns.get_level_values("Site").str.contains("Y") # Create a boolean filter selecting all columns where the Site level contains a "Y"

y_sites = phospho.loc[:, y_site_filter] # Select the columns
y_sites.head()<jupyter_output><empty_output><jupyter_text>How to use DataSet.reduce_multiindex()To make it easier to work with multi-level indices, we have provide the DataSet.reduce_multiindex function. It can both drop levels from a multiindex, and "flatten" a multi-level index into a single-level index by concatenating the keys from multiple levels into a single key for each column.  Dropping LevelsWe can drop levels based on index or name. We can also drop single or multiple levels at once. Note that it will warn you if duplicate column key combinations arise due to dropping levels.   Dropping by index or name<jupyter_code>en.reduce_multiindex(df=phospho, levels_to_drop="Site").head()<jupyter_output>Warning: Due to dropping the specified levels, dataframe now has 71331 duplicated column headers. (/Users/hannahboekweg/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py, line 1)<jupyter_text>Dropping single or multiple at onceBy passing a list (or array-like) to levels_to drop, we can drop multiple levels of the multiindex at the same time. Note that we must leave at least one existing level. We will show this with the colon data.<jupyter_code>colon = cptac.Colon()
phospho = colon.get_phosphoproteomics()
phospho.head()
#Drop evey level except Database_ID
drop = ["Name", "Site"]
colon.reduce_multiindex(df=phospho, levels_to_drop=drop)<jupyter_output>Warning: Due to dropping the specified levels, dataframe now has 29259 duplicated column headers. (/Users/hannahboekweg/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py, line 3)<jupyter_text>Combining levels (Flattening) We can combine levels of a multiindexed dataframe. When combined the levels will be sepereated by an underscore, by default. Note that we could specify a different seperator.<jupyter_code>colon.reduce_multiindex(df=phospho, flatten=True)
colon.get_transcriptomics().head()<jupyter_output><empty_output><jupyter_text>When flatteing levels , NaNs will automitically be dropped<jupyter_code>phospho_and_CNV = en.join_omics_to_omics(df1_name="CNV", df2_name="phosphoproteomics")
phospho_and_CNV.head()
#Notice that the NaNs have been dropped
en.reduce_multiindex(df=phospho_and_CNV, flatten=True).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07274905854159534, 'fraction_numerical': 0.03269428277986991, 'mean_word_length': 6.116930572472595, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '188928', 'license': 'Apache-2.0', 'n_tokens_mistral': 1932, 'n_tokens_neox': 1718, 'n_words': 692, 'path': 'tutorial4_multiindex.ipynb', 'repo_name': 'PayneLab/cptac_demos', 'size': '5842'}