<jupyter_start><jupyter_text>![alt text](https://github.com/callysto/callysto-sample-notebooks/blob/master/notebooks/images/Callysto_Notebook-Banner_Top_06.06.18.jpg?raw=true)  Santa Visiting Homes in Strathcona County What you need to doThere are a lot of homes in Strathcona County and Santa's internal GPS is malfunctioning. We [think](https://www.sciencealert.com/turns-out-we-have-no-idea-why-the-northern-lights-wreak-havoc-on-our-satellite-technology) the GPS interference is due to strong aurora borealis (aka Northern Lights) activity which are a result of intense solar storms. Luckily [Strathcona County's Open Data Portal](https://data.strathcona.ca/) includes all of the location data of homes in the county.That’s where you as a data scientist come in. They’ve given you data and you need to reprogram Santa's GPS to figure out how to visit homes in the county on Christmas Eve in the most efficient manner possible.  Importing CodeFirst we'll import some Python libraries that we'll use. These libraries are code that other people have written to help make our programming easier.<jupyter_code># Start by importing modules

# We will store the data into a 'dataframe' using pandas
import pandas as pd
# We want to be as precise as possible in keeping tree coordinates
#from decimal import *
# We will visualize the coordinates in a map using the folium 
#!pip install folium
import folium
# We want to cluster them using the FastMarkerCluster submodule from folium plugins
from folium.plugins import FastMarkerCluster
# The geopy library will help us calculate distances on a map
from geopy import distance
#load "cufflinks" library under short name "cf" for graphics plotting
import cufflinks as cf
#command to display graphics correctly in Jupyter notebook
cf.go_offline()

#enable interactive plotting in Colab
def enable_plotly_in_cell():
    import IPython
    from plotly.offline import init_notebook_mode
    display(IPython.core.display.HTML('''<script src="/static/components/requirejs/require.js"></script>'''))
    init_notebook_mode(connected=False)
get_ipython().events.register('pre_run_cell', enable_plotly_in_cell)

print('Successfully imported Python libraries')<jupyter_output><empty_output><jupyter_text>Getting Data About HomesNext we will: 1. Retrieve the data from the Strathcona County Open Data Portal.2. Put the data in a dataframe named `home_data`. Think of a dataframe as a powerful spreadsheet.3. Have a look at the first five rows using `.head()`.<jupyter_code>data_url = 'https://data.strathcona.ca/api/views/c9fr-ivqf/rows.csv?accessType=DOWNLOAD'
home_data = pd.read_csv(data_url)
home_data.head()<jupyter_output><empty_output><jupyter_text>Visualizing Home LocationsLet's use folium to plot the home locations in our dataframe on an interactive map.<jupyter_code>m = folium.Map(location=[53.5701, -113.0741], zoom_start=10)
m.add_child(FastMarkerCluster(home_data[['LATITUDE', 'LONGITUDE']].values.tolist()))

display(m)<jupyter_output><empty_output><jupyter_text>Counting HomesThat's a lot of homes for Santa to visit, and this is just in Strathcona County. To find out how many homes are in the data set we can use `.shape`.<jupyter_code>home_data.shape<jupyter_output><empty_output><jupyter_text>Calculating Travel TimeWe can approximate Santa's travel time using the equation $t = \frac{d}{v}$ where $t$ is time, $d$ is distance, and $v$ is speed or velocity.Start by assuming that Santa can travel close to the speed of sound, or about 300 meters per second, and that he spends about 30 seconds in each home.<jupyter_code># Function with Santa's speed and time per home to calculate time required
# Feel free to change the values for flight_speed and time_per_home and re-run this cell
def calculate_required_time(travel_distance, number_of_homes):
    flight_speed = 300
    time_per_home = 30
    time_required = travel_distance / flight_speed + time_per_home
    return time_required
print('We have defined the function calculate_required_time.')
number_of_homes = home_data.shape[0]
total_distance = 0
previous_location = (53.5701, -113.0741) # starting from the middle of Strathcona County
for record in home_data.iterrows():
    current_location = (record[1]['LATITUDE'], record[1]['LONGITUDE'])
    travel_distance = distance.distance(previous_location, current_location).meters
    total_distance = total_distance + travel_distance
    previous_location = current_location

print(total_distance, 'meters')
required_time = calculate_required_time(total_distance, number_of_homes)
print(required_time, 'seconds required.')
print(required_time/3600, 'hours required')<jupyter_output><empty_output><jupyter_text>That seems like a long time, a little over a week. You can of course change the values in the calculate_required_time function so Santa travels faster or spends less time in each home.  Visualizing PathA better way to decrease the travel time, though, would be to visit homes in an optimal order. We will visualize this using the Cufflinks library for Plotly. Right now we just have Santa visiting homes in the order they are listed in the data:<jupyter_code>home_data.iplot(x="LONGITUDE",y="LATITUDE",xTitle="Longitude",yTitle="Latitude")<jupyter_output><empty_output><jupyter_text>Looking at just the first 50 homes, we can see that this is not an efficient path:<jupyter_code>home_data[0:50].iplot(x="LONGITUDE",y="LATITUDE",xTitle="Longitude",yTitle="Latitude")<jupyter_output><empty_output><jupyter_text>Travelling Salesman ProblemOptimizing Santa's travel path is a version of the classic [travelling salesman problem](https://simple.wikipedia.org/wiki/Travelling_salesman_problem), which is actually a very hard mathematical problem to compute.There hasn't yet been a good solution, and there is a [$1,000,000 prize](http://www.claymath.org/millennium-problems/p-vs-np-problem) available to anyone who solves it.  Filtering and Sorting DataAssuming that you haven't solved the travelling salesman problem already, we'll try to optimize Santa's route by eliminating some homes. Let's see what data categories are available to us in our `home_data` dataframe:<jupyter_code>home_data.columns<jupyter_output><empty_output><jupyter_text>There are a couple of column names that are interesting for our purposes: `'ASSESSCLAS'` and perhaps `'FIREPLACE'`. Let's look at the types of "assessment classes":<jupyter_code># We are using a for loop to identify the unique values of "assessment classes"
for assessment_class in home_data['ASSESSCLAS'].unique():
    print(assessment_class)<jupyter_output><empty_output><jupyter_text>To get just the homes that are `'Residential'` try to run the following. Notice, we create a new dataframe called `home_data_filtered`.<jupyter_code>condition = home_data['ASSESSCLAS']=='Residential'
home_data_filtered = home_data[condition]
home_data_filtered.shape<jupyter_output><empty_output><jupyter_text>Or how about just the homes with fireplaces, try:`condition = home_data['FIREPLACE']=='Y'`<jupyter_code># enter your code below<jupyter_output><empty_output><jupyter_text>You can also specify two conditions like this, try it yourself:---`condition1 = home_data['ASSESSCLAS'] == 'Residential'``condition2 = home_data['YEAR_BUILT'] < 2000``home_data_filtered = home_data[(condition1) & (condition2)]`---`&` means **and**`|` means **or**<jupyter_code># enter your code below<jupyter_output><empty_output><jupyter_text>Ordering the data by latitude might also help. Notice that we create a new dataframe called `home_data_sorted`. `home_data_sorted = home_data_filtered.sort_values(by=['LONGITUDE'])`<jupyter_code>home_data_sorted = home_data_filtered.sort_values(by=['LONGITUDE'])
# Below we are plotting the first 50 rows of this filtered and sorted data
home_data_sorted[0:50].iplot(x='LONGITUDE',y='LATITUDE',xTitle="Longitude",yTitle="Latitude")<jupyter_output><empty_output><jupyter_text>Graphing DataHere are some graphs to help you visualize the data and make decisions about which homes to include in Santa's route.<jupyter_code>#home_data['FIREPLACE'].value_counts().plot.pie()
home_data["FIREPLACE"].value_counts().iplot(kind="bar")
#home_data['ASSESSCLAS'].value_counts().plot.pie()
home_data['ASSESSCLAS'].value_counts().iplot(kind="bar")<jupyter_output><empty_output><jupyter_text>ChallengeAfter running the two code cells above, edit this cell and explain what you see in the plotted graphs.1. Enter your explanation for the "FIREPLACE" graph here. Does this help you to decide which homes to include in Santa's route?2. Enter your explanation for the "ASSESSCLAS" graph here. Does this help you to decide which homes to include in Santa's route?3. In the code cell below, try to plot another graph based on a different category. Remember, you can list all the categories or column names using `home_data.columns`.<jupyter_code># enter your code below<jupyter_output><empty_output><jupyter_text>What did you try graphing? What do you see in your graph? Does this help you to decide which homes to include in Santa's route?* Enter your explanation for the graph you created.  Calculating Total Time From a DataframeTo make things easier, let's define a function that calculates Santa's total time using data in a dataframe.After running this cell, we'll be able to see Santa's total time by calling this function with one of your dataframes like this:`calculate_time_from_dataframe(home_data_sorted)`<jupyter_code># Function with Santa's speed and "time per home" to calculate total time required
def calculate_time_from_dataframe(df):
    flight_speed = 300  # in m/s, the speed of sound = 331 + 0.6*T where T is temperature is Celsius
    time_per_home = 30  # in seconds
    number_of_homes = df.shape[0]
    total_distance = 0
    previous_location = (53.5701, -113.0741) # the middle of Strathcona County
    for row in df.iterrows():
        current_location = (row[1]['LATITUDE'], row[1]['LONGITUDE'])
        travel_distance = distance.distance(previous_location, current_location).meters
        total_distance = total_distance + travel_distance
        previous_location = current_location
    time_required = total_distance / flight_speed + time_per_home
    print(total_distance, 'meters')
    print(time_required, 'seconds')
    print(time_required/3600, 'hours')
    return time_required
print('We have defined the function calculate_time_from_dataframe.')<jupyter_output><empty_output><jupyter_text>Analysis ChallengeTry out different filtering and sorting ideas to see how best to minimize the time that Santa takes to visit homes in Strathcona County.<jupyter_code># enter your code below

# enter your code below<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08420756837766954, 'fraction_numerical': 0.01077182465342825, 'mean_word_length': 6.160965794768612, 'pattern_counts': {'":': 1, '<': 72, '<?xml version=': 0, '>': 71, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '584124', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3286, 'n_tokens_neox': 3131, 'n_words': 1106, 'path': 'StrathconaCountyOpenData/santa-visiting-homes-challenge.ipynb', 'repo_name': 'BryceHaley/hackathon', 'size': '10676'}