<jupyter_start><jupyter_text>Pandas OverviewTo be able to use pandas you need to import it into the current python scope. It is common to import pandas using the alias pd becaue it makes calling it more succinct using the `import  as `. ```Pythonimport pandas as pd```Pandas has a custom data structure called a `DataFrame` which is a very powerful multidmensional, hetrogeneous data type storer. The data are stored using an index and a column name. For example, in the example below the indexes are r0-r4 and columns are c0-c3. |    |        c0 |       c1 |       c2 |       c3 ||:---|----------:|---------:|---------:|---------:|| r0 | 0.169489  | 0.226653 | 0.35706  | 0.199845 || r1 | 0.181289  | 0.289734 | 0.392262 | 0.228899 || r2 | 0.0492157 | 0.429355 | 0.299516 | 0.566478 || r3 | 0.304988  | 0.149374 | 0.801296 | 0.699043 || r4 | 0.236938  | 0.652044 | 0.132635 | 0.519792 | Creating a DataFrame```Pythonimport pandas as pdimport numpy as npdf = pd.DataFrame(np.random.random((5,4)),columns=['c0','c1','c2','c3'],index=['r0','r1','r2','r3','r4'])``` Importing from filesPandas can import data from many different regularly formatted datasets* When importing from a csv file you can specify:    * the delimiter - e.g. comma, semicolon, tab, space    * the header names for the column names    * Column index for the row index names, default is None and these are just assigned to be indexed 0-Nrows```Pythonimport pandas as pddf = pd.read_csv(filename,sep=',',header,index_col=None)```* Importing from excel requires    * sheet_name    * header row index    * index_col```Pythonimport pandas as pddf = pd.read_excel(filename,sheet_name,header,index_col=None)```* Python datasets and objects can also be saved in a binary format using Pickel. It is possible to save the files using this format as well. ```Pythonimport pandas as pddf = pd.read_pickle(filepath)```**Exercise**1. Create a pandas DataFrame 2. Load a csv file into pandas 3. Load an excel spreadsheet  Selecting data by column and rowUsing the same DataFrame as before:|    |        c0 |       c1 |       c2 |       c3 ||:---|----------:|---------:|---------:|---------:|| r0 | 0.169489  | 0.226653 | 0.35706  | 0.199845 || r1 | 0.181289  | 0.289734 | 0.392262 | 0.228899 || r2 | 0.0492157 | 0.429355 | 0.299516 | 0.566478 || r3 | 0.304988  | 0.149374 | 0.801296 | 0.699043 || r4 | 0.236938  | 0.652044 | 0.132635 | 0.519792 |We can access the elements inside the DataFrame by either:* using the column name and the index: `df.loc[index,column_name]````Pythondf.loc['r1','c3']0.2288986004071356```* Or we can access the same value using the column index and row index `df.iloc[rowindex,colindex]````Pythondf.iloc[1,3]0.2288986004071356```We can also use this syntax to update the values of the DataFrame and combine it with slicing```Pythondf.iloc[1,:] = 0df.loc['r3',:] = 10```This would update the previous dataframe to be|    |        c0 |       c1 |       c2 |       c3 ||:---|----------:|---------:|---------:|---------:|| r0 | 0.169489  | 0.226653 | 0.35706  | 0.199845 || r1 | 0.000000  | 0.000000 | 0.000000 | 0.000000 || r2 | 0.0492157 | 0.429355 | 0.299516 | 0.566478 || r3 | 10.000000  | 10.000000 | 10.000000 | 10.000000 || r4 | 0.236938  | 0.652044 | 0.132635 | 0.519792 |**Exercise**Using the following DataFrame```Pythonimport pandas as pddf = pd.read_csv("example_data_frame.csv")```Create a dataframe with a row index that has a keyword and a number. Use a boolean mask to update only the values of different experiments to be `**2` and others to be normalised and some can be standardised```Pythondef normalise(array):    '''    Make a dataset range from 0 to 1    '''    minval = np.min(array)    maxval = np.max(array)    array-=minval    array/=(maxval-minval)```1. Update every second row of the DataFrame to be squared2.<jupyter_code># df.loc['r1','c3']
df.iloc[1,:] = 0
df.loc['r3',:] = 10
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15535714285714286, 'fraction_numerical': 0.13392857142857142, 'mean_word_length': 3.2619565217391306, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '491082', 'license': 'MIT', 'n_tokens_mistral': 1672, 'n_tokens_neox': 1302, 'n_words': 447, 'path': '4b. Data input and output with pandas.ipynb', 'repo_name': 'lachlangrose/LearningPython', 'size': '3920'}