<jupyter_start><jupyter_text>Tutorial 2: Accessing dataframes and using them with `pandas` A large portion of this package utilizes the functionality of pandas dataframes. Here we give a few specific instances of using dataframes with our package. For a more general overview of dataframes, see pandas [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) and [tutorials](https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python).  How do I install the package?Install the package from the [Python Package Index (PyPI)](https://pypi.org/project/cptac/) in the command line using the [pip package installer](https://pypi.org/project/pip/), with the name of the package:`pip install cptac`  How do I import the package once it has been installed?The cptac package has several cancer data sets. First import the entire package by entering the following:`import cptac`(Once the package has loaded, a message will display the package version, as well as instructions for viewing specific data sets.)Then, to access a specific data set (i.e. Endometrial data) and its functionalities, download the data for it:`cptac.download(dataset="endometrial", version="latest")`Then, load the dataset object and assign it to a variable, like this:`en = cptac.Endometrial()`As the dataset loads, it will update you on the progress of the data loading until it's finished.<jupyter_code>import cptac
cptac.download(dataset="endometrial", version="latest")
en = cptac.Endometrial()<jupyter_output><jupyter_text>Can I use multiple datasets at the same time?You can have multiple datasets loaded at the same time, once you have downloaded their data files. As you load each dataset, it will display its individual loading information.<jupyter_code>cptac.download(dataset="ovarian", version="latest")
cptac.download(dataset="colon", version="latest")

ov = cptac.Ovarian()
co = cptac.Colon()<jupyter_output><jupyter_text>**NOTE: When using multiple data sets, be sure to check that each function used matches the expected data set, as each data set uses the same API.** For example, the command for retrieving clinical data is `get_clinical()` for all data sets, so make sure not to retrieve ovarian clinical data `ov.get_clinical()` when you meant to get the endometrial clinical data `en.get_clinical()`.  How do I access a particular dataframe?You can access a specific dataframe by calling the dataset's "get" method for that dataframe. (To see all available dataframes, call the dataset's `list_data` function, e.g. `en.list_data()`.)<jupyter_code>proteomics = en.get_proteomics()
proteomics.head()<jupyter_output><empty_output><jupyter_text>How do I access specific columns in a dataframe?You'll probably want to get a feel for what data is in the dataframes you load. For example, say we want to know what kind of data is included in our proteomics dataframe. Each column in that dataframe contains the proteomics data for a different protein.You can view a list of column names (which is a list of protein names, in the case of the proteomics dataframe) by appending `.columns` to the end of a dataframe variable. (If you wish to see all of the column names, even if there are a lot, append `.columns.values` to the dataframe variable.)<jupyter_code>proteomics.columns<jupyter_output><empty_output><jupyter_text>To access specific column (which is a specific protein's data, in the case of the proteomics dataframe), slice the column out of the dataframe using either of the following methods:`proteomics["A1BG"]`or`proteomics.A1BG`Both return the column as a [pandas series](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html). The first method is useful when the name of the column you want is stored as a string variable.<jupyter_code>protein = "A1BG"
A1BG_col = proteomics[protein]
A1BG_col.head()<jupyter_output><empty_output><jupyter_text>This `dataframe["col_name"]` syntax also allows for selection of multiple columns by entering a list of column names.<jupyter_code>proteins = ["A1BG","PTEN","TP53"]
selected_prot = proteomics[proteins]
selected_prot.head()<jupyter_output><empty_output><jupyter_text>How do I access specific rows in a dataframe?You can access specific rows in a dataframe (which are specific samples, in the case of the CPTAC data) using the dataframe's `.iloc` (by row number) or `.loc` (by row name) method, which both return a pandas Series if you select one row, and a pandas DataFrame if you select multiple rows.<jupyter_code>proteomics.iloc[0:5]
S001_row = proteomics.loc["C3L-00006"]
S001_row.head()<jupyter_output><empty_output><jupyter_text>How do I access specific rows and columns?In addition to selecting specific rows, you can also use `.loc` to select a subset of rows and columns, using lists.<jupyter_code>samples = ["C3L-00006","C3L-00032","C3L-00413"]
proteins = ["A1BG","PTEN","TP53"]
proteomics.loc[samples, proteins]<jupyter_output><empty_output><jupyter_text>How can I search using conditional statements?There are a variety of ways to use boolean statements to traverse a dataframe. A common way is to pass a boolean statement that selects the data you want to the `.loc` function. For example, if we want to see all the data for samples that have a positive protein expression level for the A1BG protein, we would pass the `.loc` function the boolean statement asking for rows containing values above zero for the A1BG column.  `.loc` has many functionalities. For a full list, see pandas documentation for [.loc](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html) and [indexing and slicing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html).<jupyter_code>a1bg_positive = proteomics.loc[proteomics["A1BG"] > 0]
a1bg_positive.head()<jupyter_output><empty_output><jupyter_text>For another example, suppose we wanted to separate clinical information for samples that are serous vs. endometrioid, as recorded in the "Histologic_type" column.<jupyter_code>clinical = en.get_clinical()
endometrioid_clinical = clinical.loc[clinical["Histologic_type"] == "Endometrioid"]
serous_clinical = clinical.loc[clinical["Histologic_type"] == "Serous"]
endometrioid_clinical.head()
serous_clinical.head()<jupyter_output><empty_output><jupyter_text>How do I export a dataframe to a file?If you wish to export a dataframe to a file, call the dataframe's built-in `to_csv` method, specifying the path you wish to save to, and the separator you wish to use:<jupyter_code>clinical = en.get_clinical()
clinical.to_csv(path_or_buf="clinical_df.tsv", sep='\t')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08736842105263158, 'fraction_numerical': 0.007518796992481203, 'mean_word_length': 6.060509554140127, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '188373', 'license': 'Apache-2.0', 'n_tokens_mistral': 2031, 'n_tokens_neox': 1907, 'n_words': 768, 'path': 'notebooks/tutorial02_pandas.ipynb', 'repo_name': 'old-rob/cptac', 'size': '6650'}