<jupyter_start><jupyter_text>Unit 2---1. [Introducing Pandas](section1)2. [Reading files](section2)3. [Selecting data](section3)4. [Conditional selection](section4)   1. Introducing Pandas---[Panda's documentation](https://pandas.pydata.org/pandas-docs/stable/) To begin we need to import pandasWhen you see pd, know it is referring to pandas<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Pandas is a popular Python library used for working in tabular data (similar to the data stored in a spreadsheet).There are two main data structure used by pandas- Series: equivalent to a vector or a list- Dataframe: equivalent to a table. Each column in a pandas Dataframe is a pandas Series data structure. We will mainly be looking at the Dataframe.We can easily create a Pandas Dataframe by reading a .csv file   2. Reading files---We will read the whole file at once using Pandas.Sometimes you might want to read the file line by line, and process each line. Thats possible of course. See for example [here.](https://www.geeksforgeeks.org/read-a-file-line-by-line-in-python/)We will read data on [COVID-19 vaccinations](https://github.com/owid/covid-19-data/tree/master/public/data/vaccinations)In order to do that, I retrieved the raw data's urlPress on raw either here:or here:and retrieve the link:<jupyter_code>url = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv'
vacc_df = pd.read_csv(url)<jupyter_output><empty_output><jupyter_text>read_csv has about 30 different options. See the [documentation](https://pandas.pydata.org/pandasdocs/stable/reference/api/pandas.read_csv.htmlpandas.read_csv)For example, sep='\t' is used for tab delimited files and 'usecol' reads only specific columns.<jupyter_code>type(vacc_df)<jupyter_output><empty_output><jupyter_text>view the shape of the dataframe:<jupyter_code>vacc_df.shape<jupyter_output><empty_output><jupyter_text>view basic information:<jupyter_code>vacc_df.info()
vacc_df.columns<jupyter_output><empty_output><jupyter_text>View the first few rows:<jupyter_code>vacc_df.head()<jupyter_output><empty_output><jupyter_text>What do you think that the 'tail' command does? Try it out!What happens if we just type vacc_df, without a head or a tail?<jupyter_code>vacc_df.tail()
vacc_df.describe()<jupyter_output><empty_output><jupyter_text>A summary of the functions so far:* `pd.read_csv` - Read data from a CSV file into a Pandas `DataFrame` object* `.info()` - View basic infomation about rows, columns & data types* `.describe()` - View statistical information about numeric columns* `.columns` - Get the list of column names* `.shape` - Get the number of rows & columns as a tuple* `.head()`, `.tail()` - View the beginning/end of the file  --- 3. Selecting data![](https://i.imgur.com/zfxLzEv.png)  Pandas format is similar to a dictionary, not to a lista list:<jupyter_code>covid_data_list = [
    {'date': '2020-08-30', 'new_cases': 1444, 'new_deaths': 1, 'new_tests': 53541},
    {'date': '2020-08-31', 'new_cases': 1365, 'new_deaths': 4, 'new_tests': 42583},
    {'date': '2020-09-01', 'new_cases': 996, 'new_deaths': 6, 'new_tests': 54395},
    {'date': '2020-09-02', 'new_cases': 975, 'new_deaths': 8 },
    {'date': '2020-09-03', 'new_cases': 1326, 'new_deaths': 6},
]
covid_data_list<jupyter_output><empty_output><jupyter_text>a dictionary:<jupyter_code>covid_data_dict = {
    'date':       ['2020-08-30', '2020-08-31', '2020-09-01', '2020-09-02', '2020-09-03'],
    'new_cases':  [1444, 1365, 996, 975, 1326],
    'new_deaths': [1, 4, 6, 8, 6],
    'new_tests': [53541, 42583, 54395, None, None]
}<jupyter_output><empty_output><jupyter_text>The index of a dataframe doesn't have to be numeric<jupyter_code>df = pd.DataFrame({'age':[30, 2, 12, 4, 32, 33, 69],
                   'color':['blue', 'green', 'red', 'white', 'gray', 'black', 'red'],
                   'food':['Steak', 'Lamb', 'Mango', 'Apple', 'Cheese', 'Melon', 'Beans'],
                   'height':[165, 70, 120, 80, 180, 172, 150],
                   'score':[4.6, 8.3, 9.0, 3.3, 1.8, 9.5, 2.2],
                   'state':['NY', 'TX', 'FL', 'AL', 'AK', 'TX', 'TX']
                   },
                  index=['Jane', 'Nick', 'Aaron', 'Penelope', 'Dean', 'Christina', 'Cornelia'])
df<jupyter_output><empty_output><jupyter_text>In our our file the index is numeric:<jupyter_code>vacc_df.head()<jupyter_output><empty_output><jupyter_text>return a single column as a series: note: using the `.` notation is possible only for columns whose names do not contain spaces or special characters.<jupyter_code>vacc_df.location
vacc_df['location']<jupyter_output><empty_output><jupyter_text>return a single column as a dataframe:<jupyter_code>type(vacc_df.location)
vacc_df[['location']]<jupyter_output><empty_output><jupyter_text>retrieve a specific cell<jupyter_code>vacc_df.location[600]<jupyter_output><empty_output><jupyter_text>retrieve two columns<jupyter_code>vacc_df[['location','date']]<jupyter_output><empty_output><jupyter_text>Seletcting subsets of rows and columnsOne way to do that is iloc. `.iloc` - selects subsets of rows and columns by integer location only<jupyter_code>#vacc_df.iloc[0]  #first row
vacc_df.iloc[-1] #last row as a series
vacc_df.iloc[3:5] #last row<jupyter_output><empty_output><jupyter_text>The : operator  - when used alone it means "everything"- also used to indicate a ***slice*** of values<jupyter_code>vacc_df.iloc[1:2] # second and third row
vacc_df.iloc[[-1,2,22]] #a few specific rows

# Columns:
vacc_df.iloc[:,0] # first column of data frame  
vacc_df.iloc[:,1] # second column of data frame  
vacc_df.iloc[:,-1] # last column of data frame

#Rows and columns
vacc_df.iloc[0:5] # first five rows of dataframe
vacc_df.iloc[:, 0:2] # first two columns of data frame with all rows
vacc_df.iloc[[0,3,6,24], [0,5,6]] # 1st, 4th, 7th, 25th row + 1st 6th 7th columns.
vacc_df.iloc[0:19]<jupyter_output><empty_output><jupyter_text>What if I want to select the 'daily_vaccinations' column, but I don't remember which column it is? Use `.loc``.loc` - selects subsets of rows and columns by label only. Allowed inputs are:- A single label, e.g. 5 or 'a', (note that 5 is interpreted as a label of the index, and never as an integer position along the index).- A list or array of labels, e.g. ['a', 'b', 'c'].- A slice object with labels, e.g. 'a':'f'.<jupyter_code>vacc_df.loc[2:3,['daily_vaccinations','date']]<jupyter_output><empty_output><jupyter_text>I'm missing the location. Let's add it.<jupyter_code>vacc_df.loc[0:3,['location','daily_vaccinations']]<jupyter_output><empty_output><jupyter_text>Semantics are similar to iloc. But note:- `iloc` excludes the last element.  `df.iloc[0:1000]` will return entries 0...999- `loc`, includes the last element.  `df.loc[0:1000]` will return entries 0...1000you try it! What is the difference between:> vacc_df.iloc[0:5]> vacc_df.loc[0:5]<jupyter_code>vacc_df.iloc[0:5]
vacc_df.loc[0:5]<jupyter_output><empty_output><jupyter_text>------Now you: what do you do to select:a. first five rows?b. first two columns, all rows?c. 1st and 3rd row and 2nd and 4th column?------ a. the first five rows are:<jupyter_code>vacc_df.head()


vacc_df.iloc [:,0:2]

vacc_df.iloc[[1,3],[2,4]]<jupyter_output><empty_output><jupyter_text>---A summary of the functions in this unit:* `.iloc` - selects rows and columns by integer location* `.loc` - selects rows and columns by label location Note: indexing operators as the ones working on dictionaries, will also work in pandas. But for more advanced operations, better get used to loc and iloc.---   4. Conditional selection<jupyter_code>vacc_df.loc[:,'location'] == 'Israel'<jupyter_output><empty_output><jupyter_text>This creates a series of true/false We can insert this into the dataframe to select only that task:<jupyter_code>vacc_df[vacc_df.loc[:,'location'] == 'Israel']<jupyter_output><empty_output><jupyter_text>Another way:<jupyter_code>vacc_df.loc[vacc_df.location == 'Israel']<jupyter_output><empty_output><jupyter_text>Select two countries:<jupyter_code>two_countries = vacc_df.loc[(vacc_df.location == 'Israel') | (vacc_df.location == 'Denmark')]
two_countries.head()<jupyter_output><empty_output><jupyter_text>only the indexs of the tasks:<jupyter_code>two_countries.index.values<jupyter_output><empty_output><jupyter_text>the index in the first place:<jupyter_code>two_countries.index.values[0]<jupyter_output><empty_output><jupyter_text>how many rows for the two countries?<jupyter_code>len(two_countries)
two_countries.count()<jupyter_output><empty_output><jupyter_text>At the end of the file we have some world data.Use str.contains if you're not sure how this location is called<jupyter_code>vacc_df[vacc_df['location'].str.contains('Worl')]<jupyter_output><empty_output><jupyter_text>Remove the world data:<jupyter_code>vacc_df_noWorld = vacc_df.loc[vacc_df.location != 'World']
vacc_df_noWorld.tail()<jupyter_output><empty_output><jupyter_text>Find the country with the maximum vaccinations:<jupyter_code>max_vacc = vacc_df_noWorld['total_vaccinations'].max()
max_vacc
vacc_df_noWorld.loc[vacc_df_noWorld.total_vaccinations == max_vacc]<jupyter_output><empty_output><jupyter_text>What do you think this function does?<jupyter_code>vacc_df_noWorld.total_vaccinations.mean()<jupyter_output><empty_output><jupyter_text>---- Your turn: Select the number of daily vaccinations in Israel on date 2021-02-06 (hint: use &)<jupyter_code>vacc_df.loc[(vacc_df.location == 'Israel') & (vacc_df.date == '2021-07-20')]<jupyter_output><empty_output><jupyter_text>Find all the countries with more than 3000000 vaccinations<jupyter_code>vacc_df_noWorld.loc[(vacc_df.daily_vaccinations > 300000)]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13564701064701065, 'fraction_numerical': 0.03419328419328419, 'mean_word_length': 5.879577464788732, 'pattern_counts': {'":': 0, '<': 137, '<?xml version=': 0, '>': 140, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '140215', 'license': 'CC0-1.0', 'n_tokens_mistral': 3765, 'n_tokens_neox': 3416, 'n_words': 891, 'path': 'unit2.ipynb', 'repo_name': 'YuLe33/data-analytics', 'size': '9768'}