<jupyter_start><jupyter_text><jupyter_code><jupyter_output><empty_output><jupyter_text>![SPARK Banner](https://i.imgur.com/3vCmTns.png) SPARK | Day 1 | July 12th, 2021The agenda for today:1. About the Tools2. Introduction to Pandas (What is it?, Reading Excel and CSV Files, Viewing DataFrames)3. Indexing DataFrames4. Cleaning DataFrames5. Version Control - Introduction to Git6. How to Give Feedback  About the tools  Google Colab**Google Colab** is an online version of Jupyter Notebooks. It has cells where you can write notes and execute lines of code. We will be running the programming language called **Python** within these notebooks.To use Google Colab, you will need to have [Google Chrome](https://www.google.com/chrome/) intalled on your computer. You will also need a Google Account to be able to access [Google Drive](https://drive.google.com), where the Colab notebooks can be accessed.   Pandas  What is Pandas?Pandas are adorable little animals. But, they are also an open-source **data analysis and manilpulation tool**. It's built on top Python. Programmers like animals — can you tell?**You can think of Pandas as cuter, better and more powerful version EXCEL**  Importing the Pandas Library**Pandas** is a library that is built on top of the **Numpy** library.**Libraries:** are a collection of functions on methods that allow us to form specific actions ---Before we can use Pandas and Numpy, we need to **import** their libraries in Python.  This is the standard way of importanting the Pandas and Numpy LibraryWhen we import pandas as **pd**, every time we want to access the pandas library we can just write "pd" instead of "pandas". Likewise, importing **numpy** as np, allows us to access the numpy library with the shorter **np**<jupyter_code>import pandas as pd    
import numpy as np<jupyter_output><empty_output><jupyter_text>Reading a Excel (.xlsx) and CSV (.csv) FilesWe can use Pandas to read and import data in different file formats.Two common data file formats are **(1) Excel** and **(2) CSV**.   ExcelExcel (.xlsx) files are workbook files that are specifically created for Microsoft Excel, but they work in Google Sheets as well.  --- Reading Excel Files To import excel files in Pandas we use the [pd.read_excel](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) function> df = pd.read_excel ("path-of-file.xlsx")In this case:*   **df** is the variable that we are saving the data frame to*   **=** is how we assign data to a variable*   **pd** is the name of the *library**   **read_excel** is the name of the *function**   **path-of-file.xlsx** is the *parameter* (i.e. the name of the file)  Exercise 1: Reading your first excel fileTry importing your first excel file. The name of the file is **https://github.com/AutismResearchCentre/Spark_2020_Day_01/blob/master/US_School_Census_2019.xlsx?raw=true** (*note* The name of the file is in quotations because it string. We put all strings in single quotes ('string') or double quotes ("string"). Save the string to a variable called **census_2019**.<jupyter_code># TO-DO: Read in the US School Census Data file from 2019

path = "https://github.com/AutismResearchCentre/Spark_Datasets/blob/master/US_School_Census_2019.xlsx?raw=true"
census_2019 = pd.read_excel(path)
# [VARIABLE_NAME] = pd.read_excel([PATH_NAM<jupyter_output><empty_output><jupyter_text>To view the data frame just call the name of the variable that you saved the data frame to.<jupyter_code># Type the name of the variable of the data frame and run the cell

census_2019<jupyter_output><empty_output><jupyter_text>CSVCSV stands for Comma Separated Values. It is a plain text file where the values are separated by commas. CSV files can be opened by any spreadsheet program (e.g., Excel, Open Office, Google Sheets). You can also open and edit CSV files in simple text editors as well (e.g., NotePad) Reading Excel Files To import excel files in Pandas we use the [pd.read_csv](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) function`df = pd.read_csv ("path-of-file.csv")`In this case:*   **df** is the variable that we are saving the data frame to*   **=** is how we assign data to a variable*   **pd** is the name of the *library**   **read_csv** is the name of the *function**   **path-of-file.csv** is the *parameter* (i.e. the name of the file)  Exercise 2: Reading your first csv fileTry importing your first csv file. It's very similar to reading an excel file. The path of the file is **https://raw.githubusercontent.com/AutismResearchCentre/Spark_Datasets/master/US_School_Census_2020.csv** Save the string to a variable called **census_2020**.BONUS: What do you think would happen if we tried to use pd.read_csv to read an excel file?<jupyter_code># TO-DO: Read your first CSV file below
census_2019_2 = pd.read_csv(path)<jupyter_output><empty_output><jupyter_text>Viewing Data FramesThe DataFrame is a powerful tool in Pandas. To be able to use it properly, we need to know how to view the data in our DataFrames.As seen above, simply calling the name of our DataFrame we can see our rows and columns. If our DataFrame is super large, Jupyter NoteBook will just display the first and last rows and columns.<jupyter_code>census_2020 = pd.read_csv("https://raw.githubusercontent.com/AutismResearchCentre/Spark_Datasets/master/US_School_Census_2020.csv")

# Simply call the name of the DataFrame to display table

census_2020<jupyter_output><empty_output><jupyter_text>Shape of the Data FrameHow do we know how many data points (rows) we have? How do we know how many features (columns) we have?To determine how much data we have, we can use [df.shape](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html), which will give us information of how ```df.shape```*   **df** is the name of the Pandas DataFrame*   **.shape** is the attribute of the data framedf.shape will return a tuple in the form of (x, y)*   **x** represents the number of rows*   **y** represents the number of columns<jupyter_code>census_2020.shape<jupyter_output><empty_output><jupyter_text>Therefore, our data frame has **250 rows** and **60 columns**  HeadTo view the first rows of a DataFrame, we can use [pd.head()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html)---```df.head()```By deafult` pd.head()` will display thie** first 5 rows** of the DataFrame```df.head(n)```Where **n is an integer** value that represents the number of rows. For example, `pd.head(15)` will display the first 15 rows.<jupyter_code>census_2020.head()<jupyter_output><empty_output><jupyter_text>TailTo view the last rows of a DataFrame, we can use [pd.tail()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html)```df.tail()```By deafult` df.tail()` will display by default **last 5 rows** of the DataFrame```df.tail(n)```Where **n is an integer** value that represents the number of rows. For example, `df.tail(15)` will display the last 15 rows.   Exercise 3: Males vs FemalesDisplay the last 6 rows of the DataFrame? How many of the students are female? How many are male?<jupyter_code># TO-DO: Display the last 6 rows of the Census_2020 DataFrame

census_2020.tail(6)


# TO-DO: How many students are female

print ('3 students are female, and 3 students are male') # Print statments let us see string in the console<jupyter_output>3 students are female, and 3 students are male<jupyter_text>Indexing DataFrames  DataFrame IndexThe **DataFrame index** is the row labels of the table. *   DataFrame Indexes could be **numbers, words, or phrases***   DataFrame Indexes have to be **unique** (you can think of it as an ID number). We can't have two people with the same ID or name because that would be confusing.  Integer DataFrame IndexFor the Census 2020. Data we can see that that the DataFrame index is the bolded values **(0, 1, 2, 3 ...)**. This helps us identify each row.<jupyter_code>census_2020.head()<jupyter_output><empty_output><jupyter_text>String DataFrame Index**String** is just another word another way to say a series of characters. Examples of strings include:*   "Hello"*   "Password123"*   "12345"*   'Bob the Builder!'*   "$%&$^$^!*  *&%*salkdfjlksf"Notice how strings are wrapped in double ("") or single quotations (''). Strings can also be DataFrame indexes. Take the example below of the cities DataFrame. The index of is the name of Finish cities (e.g., Helsink, Espoo, Tampere)<jupyter_code># Getting the cities DataFrame
cities = pd.DataFrame(['Helsinki', 'Espoo', 'Tampere', 'Vantaa', 'Oulu'])
cities['Population'] = [643272, 279044, 231853, 223027, 201810]
cities['Total area'] = [715.48, 528.03, 689.59, 240.35, 3817.52]
cities.set_index([0], inplace=True)

# Viewing the cities DataFrame
cities<jupyter_output><empty_output><jupyter_text>Setting a DataFrame IndexWe can actually change the index of DataFrames. [df.set_index](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_index.html) allows us to set the DataFrame index (row labels) with existing columns```df.set_index('[COLUMN_NAME]', inplace=True)```*   **df** is the name of the DataFrame*   **.set_index** is the function that we are apply to the dataframe*   **'[COLUMN_NAME]'** is the name of the pre-existing column that we would *   **inplace** gives us the option to save over our pre-exisiting DataBase. If we set *inplace=True*, our edited DataFrame overwrites the original. If we set *inplace=False*, then we keep our original DataFrame. By deafult, inplace is set to False.<jupyter_code># Changing the index from city name to population

cities.set_index('Population')
# The original cities still exists

cities
# Now lets use set_index with the inplace attribute
cities.set_index('Population', inplace=True)


# When we call cities we can see that we have saved our changes

cities<jupyter_output><empty_output><jupyter_text>Exercise 3: Mountains! Below is a table about some mountains. What column be an appropriate index for the data? Why? Create a new index of the mtns DataFrame. (Do not forget to use the `inplace=True` attribute).<jupyter_code># TO-DO: Run this cell to get assign the mtns DataFrame
mtns = pd.DataFrame([
    {'name': 'Mount Everest',
        'height (m)': 8848,
        'summited': 1953,
        'mountain range': 'Mahalangur Himalaya'},
    {'name': 'K2',
        'height (m)': 8611,
        'summited': 1954,
        'mountain range': 'Baltoro Karakoram'},
    {'name': 'Kangchenjunga',
        'height (m)': 8586,
        'summited': 1955,
        'mountain range': 'Kangchenjunga Himalaya'},
    {'name': 'Lhotse',
        'height (m)': 8516,
        'summited': 1956,
        'mountain range': 'Mahalangur Himalaya'},
])

mtns
# TO-DO: Set an new index for the table

mtns = pd.DataFrame([
    {'name': 'Mount Everest',
        'height (m)': 8848,
        'summited': 1953,
        'mountain range': 'Mahalangur Himalaya'},
    {'name': 'K2',
        'height (m)': 8611,
        'summited': 1954,
        'mountain range': 'Baltoro Karakoram'},
    {'name': 'Kangchenjunga',
        'height (m)': 8586,
        'summited': 1955,
        'mountain range': 'Kangchenjunga Himalaya'},
    {'name': 'Lhotse',
        'height (m)': 8516,
        'summited': 1956,
        'mountain range': 'Mahalangur Himalaya'},
])

mtns.set_index('name', inplace=True)
mtns<jupyter_output><empty_output><jupyter_text>Loc and Label-based Indexing[df.loc[]](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html) is used to slice and index a DataFrame based on the labels of the columns and the rows.The inputs of .loc can be:---  Using Loc for a Single Row```df.loc['ROW_LABEL']```Returns a single row of the DataFrame<jupyter_code># Read the csv file
hospitals = pd.read_csv("https://raw.githubusercontent.com/AutismResearchCentre/Spark_Datasets/master/Toronto_Hospitals.csv")

# Set the index to Hospital "Name"
hospitals.set_index('Name', inplace=True)

# View hospital DataFrame
hospitals
# Get the row for Holland Bloorview

hospitals.loc['Holland Bloorview'] # Case Sensitve
# Get the row for Centric Health Surgical Centre

hospitals.loc['Centric Health Surgical Centre']<jupyter_output><empty_output><jupyter_text>Using Loc for a Single Cell```df.loc[ROW_NAME, COLUMN_NAME]```Returns a specific cell of the DataFrame<jupyter_code># Get the year Etobicoke General was founded

hospitals.loc["Etobicoke General", "Founded"]
# Get the district of Casey House 

hospitals.loc["Casey House", "District"]
# Get the former name of Bellwood 

hospitals.loc["Bellwood", "Former name(s)"] # Return nan, which indicates missing data<jupyter_output><empty_output><jupyter_text>Using Loc to Subset the DataFrameSometimes we only want a subset of our DataFrame.   The slice operator (:)```[START_INDEX : STOP_INDEX]```The slice operator cuts a sequence that starts at the `START_INDEX` and ends with `END_INDEX`<jupyter_code># Slicing Rows: Get rows Bellwood to Etobicoke General

hospitals.loc['Bellwood': 'Etobicoke General']
# Slicing Rows and Columns: 

# Get Rows from Casey House to Holland Bloorview, and columns from District to Network

hospitals.loc['Casey House': 'Holland Bloorview', 'District' : 'Network']<jupyter_output><empty_output><jupyter_text>```[:]```A single colon (with nothing before or after it), means get everything<jupyter_code># Get the whole hospital DataFrame

hospitals.loc[:] # This is the same as just calling hospitals
# Get all the rows, but only the 'Network' and 'Former name(s)' Columns

hospitals.loc[:, 'Network': 'Former name(s)']<jupyter_output><empty_output><jupyter_text>```[: END_INDEX]```Assumes the Start Index is the first entry of the DataFrame```[START_INDEX : ]```Assumes the End Index is  the last entry of the DataFrame<jupyter_code># Get all rows from "Sunnybrook" to the end of the list

hospitals.loc['Sunnybrook':]
# Get all rows up to "Bridgepoint" (Inclusive)

hospitals.loc[ :"Bridgepoint"]<jupyter_output><empty_output><jupyter_text>Arrays for IndexingArrays can help us select the specifc items (columns or rows) that we need. ```[ INDEX1, INDEX2, INDEX3 ... INDEX8 ]```This helps us select the specific indexes that we want. ```[ INDEX13, INDEX2, INDEX34 ... INDEX2 ]```Indexes do not need to be in order<jupyter_code># Get the Years that Sunnybrook, Humber River, Mount Sinai were founded 

hospitals.loc[['Sunnybrook', 'Humber River', 'Mount Sinai'], 'Founded'] # Notice how he names of the hospitals are in an array
# We can also save the array of of indexes to variables

row_indexes = ['Birchmount', 'Baycrest', 'Toronto General']
column_indexes = ['District', 'Founded']

hospitals.loc[row_indexes, column_indexes]<jupyter_output><empty_output><jupyter_text>Exercise 4: Pokedex (Indexing with Pokemon)Below is a table of Generation 1 Pokemon. Answer the question below.<jupyter_code>pokemon =  pd.read_csv("https://raw.githubusercontent.com/AutismResearchCentre/Spark_2020_Day_01/master/Gen1_Pokemon.csv")
pokemon<jupyter_output><empty_output><jupyter_text>**Set the index of Pokemon DataFrame to `"Name"`**<jupyter_code># TO-DO: Set the `"Name"` as the DataFrame Index (Row Labels)


pokemon.set_index('Name',inplace=True)
pokemon<jupyter_output><empty_output><jupyter_text>**What is the type of Pokemon is the `"Abra"`?**<jupyter_code># TO-DO: Find the type of Abra?

pokemon.loc['Abra','Type']<jupyter_output><empty_output><jupyter_text>**What are the evolutions and numbers of `"Charmander"`, `"Pikachu"` and `"Squirtle"`?**<jupyter_code># TO-DO: Get the evolutions of these three pokemon
pokemon.loc[['Charmander',"Pikachu","Squirtle"],"Evolves_Into"]<jupyter_output><empty_output><jupyter_text>**Cut the Table from `"Eevee"` to `"Mewtwo"`. What is the range of numbers in the Pokedex**to<jupyter_code># TO-DO: Cut the table from Evee to Dragonite

pokemon.loc['Eevee':'Mewtwo']<jupyter_output><empty_output><jupyter_text>Cleaning DataBefore we can start analyzing data, we need to clean it. IBM Data Analytics estimates that data scientists can spend up to 80% of time cleaning data. Missing ValuesOne of the biggest data cleaning tasks is the problem of **missing values**. Sources of Missing ValuesData can be missing for a variety of different reasons. For example,* The participant forget to fill in a field in a survey* Data was lost while transferring files* There was a programming error* Data was not collected for a specific participant We are going to load a csv file with some propery data. As you can see there is some issue with this Data.![Imgur](https://i.imgur.com/cpDUkBk.png)<jupyter_code># Import libraries
import pandas as pd
import numpy as np

# Read csv file into a pandas DataFrame
df_property = pd.read_csv("https://raw.githubusercontent.com/nguyenjenny/spark_shared_repo/main/datasets/Property_Data.csv")

# Print out the Data
df_property<jupyter_output><empty_output><jupyter_text>Features and Data TypesBefore we begin cleaning our data we need to understand what features we have (i.e. what do the columns of the data represent) and what are the expected data types. What are the features?- `PID`: Property ID number- `ST_NUM`: Street number- `ST_NAME`: Street name- `OWN_OCCUPIED`: Is the residence owner occupied- `NUM_BEDROOMS`: Number of bedrooms- `NUM_BATH`: Number of bathrooms- `SQ_FT`: Square footage of the property- `Extra`: Extra column with no data in it What are the expected types?- `PID`: float (decimal number) or int (whole number), something numeric- `ST_NUM`: float (decimal number) or int (whole number), something numeric- `ST_NAME`: string (a mix of letters or numbers)- `OWN_OCCUPIED`: string (where Y = "yes" and N "no")- `NUM_BEDROOMS`: float (decimal number) or int (whole number), something numeric- `NUM_BATH`: float (decimal number) or int (whole number), something numeric- `SQ_FT`: float (decimal number) or int (whole number), something numeric- `Extra`: NaN (missing data)  How Pandas Deals with Missing Values (`NaN`)Missing data in pandas is often called `NaN` which stands for **Not a Number**.Here we can see that row `4`, column `"PID"` is  nan.<jupyter_code># Index row 4, column "PID"
df_property.loc[4, "PID"]
## How to check if data is considered missing<jupyter_output><empty_output><jupyter_text>How to check if Data is Considered Missing (`isnull`)There is a method called `isnull` that we can use to determine if data is missing.- Missing or null data will return `True`- Non-missing or non-null data will return `False`<jupyter_code># Check is the data is null or not
df_property.isnull()<jupyter_output><empty_output><jupyter_text>We can also check if nulls on a single column<jupyter_code># We can check a single column
print(df_property['ST_NUM'])
df_property['ST_NUM'].isnull()
# We can also count how many missing/null values we have using the `sum()` method
df_property['ST_NUM'].isnull().sum()
# We can use the sum method on the entire DataFrame too
df_property.isnull().sum()<jupyter_output><empty_output><jupyter_text>Standard Missing ValuesPandas will recognize the following automatically as standard missing values- empty cells- "n/a"- "NA"- "na"As we can see all the items in blue, have been importated as `NaN`![Imgur](https://i.imgur.com/cpDUkBk.png)<jupyter_code># Return the table
df_property
# Non standard missing val<jupyter_output><empty_output><jupyter_text>Non-standard Missing ValuesThere are also some missing values that are not automatically recognized as missing. For example `"--"` is being recognized as a string instead of a missing value. In this case, we tell pandas what our missing values are by using the parameter `na_values`, when we use `pd.read_csv`.<jupyter_code># Create a list of missing values and save as a variable
missing_values = ['n/a', 'na', 'NaN', 'NA', '--']

# Read csv as as pandas data frame, but use the paramter na_values
df_property = pd.read_csv("https://raw.githubusercontent.com/nguyenjenny/spark_shared_repo/main/datasets/Property_Data.csv", na_values=missing_values)

# Return df_property, now "--" has been replaced with NaN

df_property<jupyter_output><empty_output><jupyter_text>Dealing with Errors in the DataThere are also a few errors in the data (yellow cells). ![Imgur](https://i.imgur.com/cpDUkBk.png)We can use `replace` to mass replace incorrect cells. Or, we can use fix errors on a case-by-case basis by overwriting them using indexing (`loc`).  Replacing Errors (`replace`)This method works well, when there's a value in a data set that makes no sense. In the data set we have, "HURLEY" is incorrect, so we can replace with nan (`np.nan`)We use the `replace` method.```df.replace("[OLD_VALUE]", "[NEW_VALUE]")```<jupyter_code># Replace "Hurley" with np.nan
df_property = df_property.replace("HURLEY", np.nan)

# Show data frame
df_property<jupyter_output><empty_output><jupyter_text>Overwriting Errors with Indexing (`loc`)To overwrite a data point or error. We simply index it using `loc` and the make it equal to our new value.```df.loc[ "ROW_NAME" , "COLUMN_NAME"] = [NEW_VALUE]```We want to overwrite errors using indexing when we have to deal with data on a case by case basis.  For example, it doesn't make sense to have a value of `12` for the column `"OWN_OCCUPIED"`. But we don't want to just replace all values of `12` with `np.nan`. For example, we could have a house with a street number of 12 and that's not an error. This is where indexing comes in handy.  The row name is `3` and the row_columns is `"OWN_OCCUPIED"` and we want to replace this with `np.nan`<jupyter_code># Overwrite row 3, column "OWN_OCCUPIED" with np.nan
df_property.loc[3, "OWN_OCCUPIED"] = np.nan

# Show the updated dataframe
df_property<jupyter_output><empty_output><jupyter_text>Example 5: Replacing missing data using overwriting with indexingOverwriting with indexing can also be used to replacing missing values with actual values. Lets replace this missing value in column "PID" with 100005000 Lets replace this missing value in column `"PID"` with `100005000`<jupyter_code>### TO-DO: Repalce this missing ID in the "PID" column with 100005000<jupyter_output><empty_output><jupyter_text>Replacing Missing Data (`fillna`)The `fillna` method is used to replace missing data. ```df = df.fillna(VALUE_TO_FILL)df["COLUMN_NAME"] = df["COLUMN_NAME"].fillna(VALUE_TO_FILL)``` Let's say we want to assume if we don't have information about the number of bathrooms (`"NUM_BATH"`), we can assume there is only 1 bathroom. In this case, we want to fill value to be 1. We can apply `fillna`<jupyter_code># Fill all nan values
df_property["NUM_BATH"] = df_property["NUM_BATH"].fillna(1)

# Show the new dataframe
df_property<jupyter_output><empty_output><jupyter_text>For number of bedrooms (`"NUM_BEDROOMS"`), we can fill missing data with the mean number of bedrooms<jupyter_code># Calculate the mean number of bedrooms first and save as variable 

mean_beds = df_property["NUM_BEDROOMS"].mean() # You will learn more about this later

print(mean_beds)
# Fill all nan values for NUM_BEDRROMS with the mean

# Fill all nan values
df_property["NUM_BEDROOMS"] = df_property["NUM_BEDROOMS"].fillna(mean_beds)

# Show the new dataframe
df_property<jupyter_output><empty_output><jupyter_text>Example 6: Filling missing data for the `"OWN_OCCUPIED"` columnIf data on whether a place is `"OWN_OCCUPIED"` is missing, fill it with `"N"`, meaning that it is not occupied.<jupyter_code>### TO-DO: Fill missing "OWN_OCCUPIED" data with "N"<jupyter_output><empty_output><jupyter_text>Dropping rows/indexes and columns with missing data (`dropna`)Sometimes we may not want to use rows/indexes or columns if data is missing. This is where the  `dropna` function comes in handy.Documentation for this function is found here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html?highlight=dropnaParameters: -  `axis`: the axis that you want to drop data on   - can be "index" or "columns"  - `axis="index"` drops rows with missing data  - `axis="columns"` drops columns with missing data- `how`: how you want to drop the data  - can be "any" or "all"  - `how="any"` If any NA values are present, drop that row or column  - `how="all"` If all values are NA, drop that row or column. Examples Drop columns where all the values are missing```df.dropna(axis="columns", how="all") ```Drop columns where there are any missing values```df.dropna(axis="columns", how="any") ```Drop rows/index where all the values are missing```df.dropna(axis="index", how="all") ```Drop rows/index where there are any missing values```df.dropna(axis="index", how="any") ```<jupyter_code># Drop columns where all values are missing
df_property.dropna(axis="columns", how="all") 
# Drop columns where any values are missing
df_property.dropna(axis="columns", how="any")<jupyter_output><empty_output><jupyter_text>Exercise 7: Drop rows/indexes where all and any of the data is missing.<jupyter_code>### TO-DO: drop rows/indexes where all values are missing




### TO-DO: drop rows/indexes where any values are missing<jupyter_output><empty_output><jupyter_text>GitHub**Git** is a version control programming language.   Installing Git and GitHub     * **Installing Git on Windows**      Download and install git from [Git for Windows](https://gitforwindows.org/) *  **Installing Git on Mac**      Install [Git OSX Installer](https://sourceforge.net/projects/git-osx-installer/files/ ) 2. Create a [GitHub Account](https://github.com/join)        3. Configure Git to your GitHub account  * Open git bash  * Set a Git username     * `$ git config --global user.name "<NAME>"`    * Confirm that you have set the Git username correctly      * `$ git config --global user.name`       * `> <NAME>`   * Setting your commit email address in Git    * `$ git config --global user.email "<EMAIL>"`    * Confirm that you have set the Git email correctly      * `$ git config --global user.email`       * `> <EMAIL>` 4. Install **Git for Desktop*** Install [git for desktop here](https://desktop.github.com/)4. Share your git username with us in the Zoom chat and we will add you to repository  Git WorkflowBelow is an image of the git work flow and the common commands we use.![](https://i.imgur.com/0T7u7bK.png) Remote RepositoryA repo/repository is a place where all your files for a project are stored.The remote repository is the repository shared with everyone and hosted on github.com. Because it's the shared version, we don't want to make changes to it directly. Local RepopositoryThe local repository is our version of the repository that we have saved on our personal comptuers. We can try things out and change things without affecting the remote version of the code that everyone else uses.  Once Working DirectoryWhen we make changes to any of the the file in the repository. It goes into the the working directory. We can choose to keep (commit) or discard these changes  Staging AreaThe staging area is where you want you add files that you have changed to get ready to commit `git clone` To get a local repository we first need to clone it using the command git clone `git add`To add files to the staging area we use git add. This moves files from the working directory into the staging area.  `git commit`Once we have collected all the files that we would like to move you our local repository. We can use the command `git commit`. When we commit something, we must included a commit message that explains what the changes were. This helps us keep track of the changes incase we need to revert to an old version or solve a bug. `git push`Once we are comfortable with all our chages and commit and we are ready to share it with our team, we need `git push` our commits from the local repository to the remote repository `git pull`Other people will make changes to remote repository. To get those changes and to make sure our local repository is up-to-date, we need to `git pull` all the changes `git checkout`Git checkout is used in more complicated situations when we have to worry about branching `git checkout`Git checkout is used in more complicated situations when we have to worry about branching. It is used to change branches `git merge`Git merge is used when we want to combine separate branches into a single branch  Cloning your first repository1. Open **GitHub for Desktop**2. Find the repository called **Shared Spark Repo** and select `Clone [your_username]/spark_shared_repo`* ![image](https://i.imgur.com/7J9vRCl.png)3. Clone the repository* ![](https://i.imgur.com/2S3uPRy.png)4.  Select `View the files of repository in Explorer` to see the files.* You should be able to see all the cloned files saved locally* ![](https://i.imgur.com/UlY2f8l.png)We have just created our own local repository on our computer!  Making your first commit1. Open the README.md file located in the local repository in notebook or any text editor2. Add your name under contributors and save the file  * README files are in markdown. So `*` and a space represents a bullet point  * ![](https://i.imgur.com/KtxIKam.png)3. When you return to GitHub Desktop, you'll notice that the change will have been detected  * Make sure the files you want to commit are ticked off. This move files from our working directory into the staging area.  * Add a **commit message** at the bottom    * Commit messages usually start with a **captialized word**    * And start with a **verb in simple tense** (e.g., Add, Change, Remove, Fix)  *![](https://i.imgur.com/zqx25UD.png)We have just commit our changes to our local directory!  Pushing the changes to remote repositoryThe changes we made are in our local repo are ready to be "pushed" to the main branch of your remote repo.To push our changes to remote repository we can click any of the two circled push buttons.![](https://i.imgur.com/RdX6Swn.png)<jupyter_code><jupyter_output><empty_output><jupyter_text>Fetching and pulling up-to-date changes from the main remote repositoryBefore we make changes to our local repository, we want to get into the habit of fetching and pulling changes from the remote repository to make sure we are working with the most up-to-date version.1. Fetch the from origin (i.e. the remote repository)  * Fetch gets and list the commits that have been made on the remote branch  * ![](https://i.imgur.com/BMckPsE.png)2. To get those changes on our local machine, we need to pull them.  * ![](https://i.imgur.com/4I9TQNk.png)  Exercise 8: Adding your name to the contributor listGo through all the steps listed above to add your name to contributor list. Since we are all editing the same README.md file, we want to make these changes one at a time.**NB: Remember to `git pull` before you add your name, to get the most up-to-date version that your peers have added to**  Exercise 9: Add your notebook from day 1 to GitHub RepositoryThere is a folder in the repository called `day01_notebooks`: https://github.com/nguyenjenny/spark_shared_repo/tree/main/day_01_notebooks1. Download the notebook from Google Collab onto your computer as an .ipynb   * ![](https://i.imgur.com/CHJDwJc.png)   * Rename it `SPARK-Day1-[FirstName].ipynb`2. Copy it to the correct `day01_notebooks` folder in your local repository3. Stage and commit to your local repository  * Be sure to add an informative message4. Push the change to remote repository  Accessing data files through GitHubGitHub can be used to host and access datafiles through Google Collab.Hotel booking demand data is hosted here: https://www.kaggle.com/jessemostipak/hotel-booking-demand?select=hotel_bookings.csvWe can download it to our local machine and copy it to our local repository, then commit and push the datasets to GitHub.In the `datasets` folder of our shared repository (https://github.com/nguyenjenny/spark_shared_repo/tree/main/datasets), we have saved the hotel booking information in the file called Hotel_Booking.csv* ![](https://i.imgur.com/5BmOZF7.png)If we click `view raw`, it will lead us to the raw CSV file* ![](https://i.imgur.com/vVEPQHK.png)This leads us to the raw CSV file. We can use the URL as the file path when importing our data from pandas* ![](https://i.imgur.com/VZfxDgv.png)We can use `pd.read_csv("https://raw.githubusercontent.com/nguyenjenny/spark_shared_repo/main/datasets/Hotel_Bookings.csv")` to load our data  Peer Programming: Loading and cleaning the hotel data   Part 1: Load the hotel booking data using `pd.read_csv` and save it under the varialbe `df_hotels`The na_values are "non-standard". Look at the data to figure out what it is: https://raw.githubusercontent.com/nguyenjenny/spark_shared_repo/main/datasets/Hotel_Bookings.csv*Hint there are multiple "non-standard" notations for missing data. It might help to look at the data in excel. *<jupyter_code>### TO-DO: Load the hotel booking data
import pandas as pd
import  numpy as np


### TO_DO: What are the missing values called in this data set
missing_values = [""] # replace with what the missing values are called (hint there are multiple nam)



### ### TO_DO: Import the data; use pd.read_csv to import the missing data, be sure to set `na_values = missing_values`
df_hotels = "" # replace with read statement




missing_values = ["NULL", "missing_data"]
df_hotels = pd.read_csv("https://raw.githubusercontent.com/nguyenjenny/spark_shared_repo/main/datasets/Hotel_Bookings.csv", na_values = missing_values)<jupyter_output><empty_output><jupyter_text>Part 2: How many features (columns) and datapoints (rows/indexes) are three?<jupyter_code># TO-DO: Get the number of features and datapoints




df_hotels.shape<jupyter_output><empty_output><jupyter_text>Part 3: How much missing data do you have for each column?  Which column has the most missing data? Which column has the least missing data?<jupyter_code>### TO-DO: Figure out how much missing data there is in each column




df_hotels.isnull().sum()<jupyter_output><empty_output><jupyter_text>Part 4: Use `loc` to get the `"hotel"`, `"adults"`, `"children"`, `"babies"` for the rows of `40600`, `40667`, `40679`  and `41160`. What do you notice about this data?<jupyter_code>### TO-DO: Use loc to get a subset of the dataset




df_hotels.loc[[40600, 40667, 40679,  41160], ['hotel', 'adults', 'children', 'babies']]<jupyter_output><empty_output><jupyter_text>Part 5: Fill missing data for `"children"` with the value `0`. Confirm that there is no missing data using `isnull`<jupyter_code>### TO-DO: Fill missing data for the "children" column with the value 0





### TO-DO: Confirm that there is now no missing data in that column<jupyter_output><empty_output><jupyter_text>Part 6: Drop all indexes/rows with any missing data save it as `df_hotels_processed`? How many data points do you have now?---<jupyter_code>### TO-DO: Drop all indexes/rows with missing data

df_hotels_processed = "" # replace with drop statement<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09855886313497407, 'fraction_numerical': 0.011832793742658224, 'mean_word_length': 5.0210453682939455, 'pattern_counts': {'":': 0, '<': 184, '<?xml version=': 0, '>': 187, 'https://': 45, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '272712', 'license': 'MIT', 'n_tokens_mistral': 11098, 'n_tokens_neox': 10384, 'n_words': 4356, 'path': 'day_01_notebooks/SPARK_Day1_Santiago.ipynb', 'repo_name': 'nguyenjenny/spark_shared_repo', 'size': '34929'}