<jupyter_start><jupyter_text>WeatherPy---- Note* Instructions have been included for each segment. You do not have to follow them exactly, but they are included to help you think through the steps.<jupyter_code># Dependencies and Setup
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import requests
import time
from scipy.stats import linregress

# Import API key
from api_keys import weather_api_key

# Incorporated citipy to determine city based on latitude and longitude
from citipy import citipy

# Output File (CSV)
output_data_file = "../output_data/cities.csv"
clean_output_data_file = "../output_data/clean_cities.csv"

# Range of latitudes and longitudes
lat_range = (-90, 90)
lng_range = (-180, 180)<jupyter_output><empty_output><jupyter_text>Generate Cities List<jupyter_code># List for holding lat_lngs and cities
lat_lngs = []
cities = []

# Create a set of random lat and lng combinations
lats = np.random.uniform(lat_range[0], lat_range[1], size=1500)
lngs = np.random.uniform(lng_range[0], lng_range[1], size=1500)
lat_lngs = zip(lats, lngs)

# Identify nearest city for each lat, lng combination
for lat_lng in lat_lngs:
    city = citipy.nearest_city(lat_lng[0], lat_lng[1]).city_name
    
    # If the city is unique, then add it to a our cities list
    if city not in cities:
        cities.append(city)

# Print the city count to confirm sufficient count
len(cities)
cities<jupyter_output><empty_output><jupyter_text>Perform API Calls* Perform a weather check on each city using a series of successive API calls.* Include a print log of each city as it'sbeing processed (with the city number and city name).<jupyter_code>url = "http://api.openweathermap.org/data/2.5/weather?"
units = "imperial"
city = "busselton"

# # Build query URL
query_url = url + "appid=" + weather_api_key + "&units=" + units + "&q=" + city
print(query_url)

output = requests.get(query_url).json()

output

#cities


city_name = []
country = []
lat = []
long = []
temp = []
temp_min = []
temp_max = []
humidity = []
cloud = []
wind = []
date = []

count = 0

print("Beginning Data Retrieval")
print("-----------------------------")

# Loop through the list of cities and perform a request for data on each
for city in cities:
#for city in ["new york"]:
    
    count = count + 1
    
    # Build query URL
    query_url = url + "appid=" + weather_api_key + "&units=" + units + "&q=" + city
    #print(query_url)

    try:
        output = requests.get(query_url).json()
        
        # Check the temperature first. If the retrieval was bad, it will keep all the others the same length
        temp.append(output["main"]["temp"])
        temp_min.append(output["main"]["temp_min"])
        temp_max.append(output["main"]["temp_max"])
        humidity.append(output["main"]["humidity"])
        cloud.append(output["clouds"]["all"])
        wind.append(output["wind"]["speed"])
        date.append(output["dt"])
        
        lat.append(output["coord"]["lat"])
        long.append(output["coord"]["lon"])
        country.append(output["sys"]["country"])
        city_name.append(city)

        print(f"Processing Record {count} of Set 1 | {city}")
        
    except:
        print("City not found. Skipping...")
              
print("-----------------------------")
print("Data Retrieval Complete")
print("-----------------------------")<jupyter_output>Beginning Data Retrieval
-----------------------------
Processing Record 1 of Set 1 | touros
Processing Record 2 of Set 1 | lorengau
Processing Record 3 of Set 1 | jamestown
Processing Record 4 of Set 1 | ahipara
Processing Record 5 of Set 1 | inhambane
Processing Record 6 of Set 1 | rikitea
Processing Record 7 of Set 1 | bauchi
Processing Record 8 of Set 1 | voyvozh
Processing Record 9 of Set 1 | avarua
Processing Record 10 of Set 1 | mar del plata
Processing Record 11 of Set 1 | kruisfontein
Processing Record 12 of Set 1 | ancud
Processing Record 13 of Set 1 | altamont
Processing Record 14 of Set 1 | punta arenas
Processing Record 15 of Set 1 | atuona
Processing Record 16 of Set 1 | jacareacanga
Processing Record 17 of Set 1 | new norfolk
Processing Record 18 of Set 1 | tasiilaq
Processing Record 19 of Set 1 | albany
Processing Record 20 of Set 1 | port alfred
Processing Record 21 of Set 1 | bethel
Processing Record 22 of Set 1 | marakkanam
Processing Record 23 of Set 1 | ushuaia
Pro[...]<jupyter_text>Convert Raw Data to DataFrame* Export the city data into a .csv.* Display the DataFrame<jupyter_code># create a dictionary to contain all the list variables

list_of_lat = [float(item) for item in lat]
list_of_long = [float(item) for item in long]
list_of_temp = [float(item) for item in temp]
list_of_temp_min = [float(item) for item in temp_min]
list_of_temp_max = [float(item) for item in temp_max]
list_of_humid = [float(item) for item in humidity]
list_of_cloud = [float(item) for item in cloud]
list_of_wind = [float(item) for item in wind]

weather_dict = {
    "City": city_name,
    "Country": country,    
    "Latitude": list_of_lat,
    "Longitude": list_of_long,
    "Temperature (F)": list_of_temp,
    "Min Temperature (F)": list_of_temp_min,
    "Max Temperature (F)": list_of_temp_max,
    "Humidity (%)": list_of_humid,
    "Cloudiness (%)": list_of_cloud,
    "Wind (MPH)": list_of_wind
}

# convert the dictionary into a dataframe
weather_data_df = pd.DataFrame(weather_dict)

weather_data_df.to_csv(output_data_file)
weather_data_df.head()

weather_data_df.shape<jupyter_output><empty_output><jupyter_text>Inspect the data and remove the cities where the humidity > 100%.----Skip this step if there are no cities that have humidity > 100%.<jupyter_code># Get the indices of cities that have humidity over 100%.
humidity_outliers_df = weather_data_df[weather_data_df["Humidity (%)"] > 100]
humidity_outliers_df.index.values
# Make a new DataFrame equal to the city data to drop all humidity outliers by index.
# Passing "inplace=False" will make a copy of the city_data DataFrame, which we call "clean_city_data".
clean_city_data_df = weather_data_df.drop(humidity_outliers_df.index.values, inplace=False)
clean_city_data_df

weather_data_df.shape
clean_city_data_df.shape

# Export the City_Data into a csv
clean_city_data_df.to_csv(clean_output_data_file)
# Extract relevant fields from the data frame<jupyter_output><empty_output><jupyter_text>Plotting the Data* Use proper labeling of the plots using plot titles (including date of analysis) and axes labels.* Save the plotted figures as .pngs.<jupyter_code># compute for the current date and convert into mm/dd/yy format
from datetime import date

today = date.today()
analysis_date = today.strftime("%B %d, %Y")
print("Analysis Date is ", analysis_date)<jupyter_output>Analysis Date is  August 02, 2020<jupyter_text>Latitude vs. Temperature Plot<jupyter_code>title = "Temperature (F) vs Latitude"
plt.scatter(clean_city_data_df["Latitude"], clean_city_data_df["Temperature (F)"])
plt.xlabel("Latitude")
plt.ylabel("Temperature (F)")
plt.title(f"{title}  {analysis_date}")

plt.savefig(f"../output_data/{title}.png")<jupyter_output><empty_output><jupyter_text>Latitude vs. Humidity Plot<jupyter_code>title = "Humidity (%) vs Latitude"

plt.scatter(clean_city_data_df["Latitude"], clean_city_data_df["Humidity (%)"])
plt.xlabel("Latitude")
plt.ylabel("Humidity (%)")
plt.title(f"{title}  {analysis_date}")

plt.savefig(f"../output_data/{title}.png")<jupyter_output><empty_output><jupyter_text>Latitude vs. Cloudiness Plot<jupyter_code>title = "Cloudiness (%) vs Latitude"

plt.scatter(clean_city_data_df["Latitude"], clean_city_data_df["Cloudiness (%)"])
plt.xlabel("Latitude")
plt.ylabel("Cloudiness (%)")
plt.title(f"{title}  {analysis_date}")

plt.savefig(f"../output_data/{title}.png")<jupyter_output><empty_output><jupyter_text>Latitude vs. Wind Speed Plot<jupyter_code>title = "Wind (MPH) vs Latitude"

plt.scatter(clean_city_data_df["Latitude"], clean_city_data_df["Wind (MPH)"])
plt.xlabel("Latitude")
plt.ylabel("Wind (MPH)")
plt.title(f"{title}  {analysis_date}")

plt.savefig(f"../output_data/{title}.png")<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code># OPTIONAL: Create a function to create Linear Regression plots
# Create Northern and Southern Hemisphere DataFrames
north_weather_data_df = clean_city_data_df[clean_city_data_df["Latitude"] >= 0]
south_weather_data_df = clean_city_data_df[clean_city_data_df["Latitude"] < 0]
north_weather_data_df.head()
# Declare the function and create the UDF
# LRS stands for Linear_Regression_Scatter
def plot_LRS_func(x_values: pd.Series, y_values: pd.Series, title, x_title, y_title, date) :
    
    (slope, intercept, rvalue, pvalue, stderr) = linregress(x_values, y_values)
    regress_values = x_values * slope + intercept
    line_eq = "y = " + str(round(slope,2)) + "x + " + str(round(intercept,2))
    plt.scatter(x_values,y_values)
    plt.plot(x_values,regress_values,"r-")
    plt.annotate(line_eq,(min(x_values),min(y_values)),fontsize=15,color="red")
    plt.title(f"{title}  {date}")
    plt.xlabel(x_title)
    plt.ylabel(y_title)
    
    plt.savefig(f"../output_data/{title}.png")
    
    return<jupyter_output><empty_output><jupyter_text>Northern Hemisphere - Max Temp vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(north_weather_data_df["Latitude"], 
             north_weather_data_df["Max Temperature (F)"],
             "Northern Hemisphere - Max Temperature (F) vs Latitude",
             "Latitude",
             "Max Temperature (F)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Southern Hemisphere - Max Temp vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(south_weather_data_df["Latitude"], 
             south_weather_data_df["Max Temperature (F)"],
             "Southern Hemisphere - Max Temperature (F) vs Latitude",
             "Latitude",
             "Max Temperature (F)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Northern Hemisphere - Humidity (%) vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(north_weather_data_df["Latitude"], 
             north_weather_data_df["Humidity (%)"],
             "Northern Hemisphere - Humidity (%) vs Latitude",
             "Latitude",
             "Humidity (%)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Southern Hemisphere - Humidity (%) vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(south_weather_data_df["Latitude"], 
             south_weather_data_df["Humidity (%)"],
             "Southern Hemisphere - Humidity (%) vs Latitude",
             "Latitude",
             "Humidity (%)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Northern Hemisphere - Cloudiness (%) vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(north_weather_data_df["Latitude"], 
             north_weather_data_df["Cloudiness (%)"],
             "Northern Hemisphere - Cloudiness (%) vs Latitude",
             "Latitude",
             "Cloudiness (%)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Southern Hemisphere - Cloudiness (%) vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(south_weather_data_df["Latitude"], 
             south_weather_data_df["Cloudiness (%)"],
             "Southern Hemisphere - Cloudiness (%) vs Latitude",
             "Latitude",
             "Cloudiness (%)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Northern Hemisphere - Wind Speed (mph) vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(north_weather_data_df["Latitude"], 
             north_weather_data_df["Wind (MPH)"],
             "Northern Hemisphere - Wind (MPH) vs Latitude",
             "Latitude",
             "Wind (MPH)",
              analysis_date)<jupyter_output><empty_output><jupyter_text>Southern Hemisphere - Wind Speed (mph) vs. Latitude Linear Regression<jupyter_code>plot_LRS_func(south_weather_data_df["Latitude"], 
             south_weather_data_df["Wind (MPH)"],
             "Southern Hemisphere - Wind (MPH) vs Latitude",
             "Latitude",
             "Wind (MPH)",
              analysis_date)
# Trends and observations:

# I've noticed that the further South from the equator one measures the temperature the colder it generally becomes.
# The same can also be said for measuring further and further North of the equator

# There appears to no correlation with the latitude of a locality and the humidity, wind speed, and cloudiness

# However, I did notice that there are grouping of cloudiness. Specifically around 100% and 0%. There are also 
# groupings around 90%, 75%, 40%, and 20%. This is probably due to human simplification.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11813617822403515, 'fraction_numerical': 0.009805459679949797, 'mean_word_length': 4.134514699959726, 'pattern_counts': {'":': 10, '<': 76, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '325853', 'license': 'ADSL', 'n_tokens_mistral': 4370, 'n_tokens_neox': 4058, 'n_words': 1095, 'path': 'WeatherPy/WeatherPy.ipynb', 'repo_name': 'csadlo/python-api-challenge', 'size': '12748'}