<jupyter_start><jupyter_text>Practice Loading DatasetsThis assignment is purposely semi-open-ended you will be asked to load datasets both from github and also from CSV files from the [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php). Remember that the UCI datasets may not have a file type of `.csv` so it's important that you learn as much as you can about the dataset before you try and load it. See if you can look at the raw text of the file either locally, on github, using the `!curl` shell command, or in some other way before you try and read it in as a dataframe, this will help you catch what would otherwise be unforseen problems.  1) Load a dataset from Github (via its *RAW* URL)Pick a dataset from the following repository and load it into Google Colab. Make sure that the headers are what you would expect and check to see if missing values have been encoded as NaN values:<jupyter_code># Import Pandas
import pandas as pd
# Fetch data from github url and read into a pandas dataframe
data_url = "https://raw.githubusercontent.com/ryanleeallred/datasets/master/churn.csv"
churn_data = pd.read_csv(data_url)
# More browsing to be sure everything worked correctly
churn_data.head(16)
# From browsing the data it seems NaN values are encoded as "?"
# As was explored in the lecture, I should double check the exact string
# Going to start by looking at row 11 column 4
churn_data.iloc[11][4]
# Looks like the NaN are encoded as "?" - no spaces
# Just to be sure, I'll check one more cell in a different column
churn_data.iloc[13][15]
# Looks good, time to replace them with NaN
# This confirms what I saw while initially browsing the raw data in the browser

# Re-read in the dataset with the NaN argument
data_url = "https://raw.githubusercontent.com/ryanleeallred/datasets/master/churn.csv"
churn_data = pd.read_csv(data_url, na_values="?")
# Make sure it worked correctly
churn_data.head(16)
# Verify NaN, pt. 2
churn_data.isnull().sum()
# Look at the summary stats
churn_data.describe()<jupyter_output><empty_output><jupyter_text>2) Load a dataset from your local machineDownload a dataset from the [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php) and then upload the file to Google Colab either using the files tab in the left-hand sidebar or by importing `files` from `google.colab` The following link will be a useful resource if you can't remember the syntax: While you are free to try and load any dataset from the UCI repository, I strongly suggest starting with one of the most popular datasets like those that are featured on the right-hand side of the home page. Some datasets on UCI will have challenges associated with importing them far beyond what we have exposed you to in class today, so if you run into a dataset that you don't know how to deal with, struggle with it for a little bit, but ultimately feel free to simply choose a different one. - Make sure that your file has correct headers, and the same number of rows and columns as is specified on the UCI page. If your dataset doesn't have headers use the parameters of the `read_csv` function to add them. Likewise make sure that missing values are encoded as `NaN`.<jupyter_code># Upload file to Colab server
from google.colab import files
uploaded = files.upload()

# Set up column headers
column_headers = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 
                 'marital-status', 'occupation', 'relationship', 'race', 'sex', 
                 'capital-gain', 'capital-loss', 'hours-per-week', 
                 'native-country', 'income']

# Read in data with the headers defined above
adult_data = pd.read_csv("adult.data", names=column_headers)
print(adult_data.shape)
adult_data.head()
# Loaded dataset has 32,561 rows compared with 48,842 according to the site
# Reading into the "adult.names" file gives some clues for the discrepancy

# Split into train-test using MLC++ GenCVFiles (2/3, 1/3 random).
# 48842 instances, mix of continuous and discrete    (train=32561, test=16281)

# Convert Unknown to "?"
# Although it says no space, maybe there is a space...

# Reload the dataset with the NaN value argument
adult_data = pd.read_csv("adult.data", names=column_headers, na_values="?")
print(adult_data.shape)
adult_data.head(16)
adult_data.isnull().sum()
# Without a space did not do anything, so time to investigate
adult_data.iloc[14][13]
# Reload the dataset with the new na value argument
adult_data = pd.read_csv("adult.data", names=column_headers, na_values=" ?")
print(adult_data.shape)
adult_data.head(16)
# Confirm the NaN values went through correctly
adult_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>3) Load a dataset from UCI using `!wget`"Shell Out" and try loading a file directly into your google colab's memory using the `!wget` command and then read it in with `read_csv`.With this file we'll do a bit more to it.- Read it in, fix any problems with the header as make sure missing values are encoded as `NaN`.- Use the `.fillna()` method to fill any missing values.  - [.fillna() documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html)- Create one of each of the following plots using the Pandas plotting functionality: - Scatterplot - Histogram - Density Plot<jupyter_code># Load the "adult" data using the `wget` shell command
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data
# Use same column headers and NaN value from earlier
adult_data = pd.read_csv("adult.data.1", names=column_headers, na_values=" ?")

print(adult_data.shape)
adult_data.head(16)
# Make sure NaN values are showing up correctly
adult_data.isnull().sum()
# Now time to do something about the NaN values
# First, it would be useful to know more about the columns
# So I know what methods might work best for filling the missing values
(adult_data.dtypes)
# Looks like all of the columns with NaN values are "object" datatype
# Looking at the data, none of the missing values are the first row
# That means that the forward fill method should work
adult_data["workclass"] = adult_data["workclass"].fillna(method="ffill")
adult_data["occupation"] = adult_data["occupation"].fillna(method="ffill")
adult_data["native-country"] = adult_data["native-country"].fillna(method="ffill")

# Confirm that the missing values were filled
adult_data.isnull().sum()
# Take a look
adult_data.head(16)
# Plotting the dataset with pandas
# Histogram of Hours per Week
adult_data["hours-per-week"].hist(bins=16);
# Density Plot of Hours per Week
adult_data["hours-per-week"].plot.density();
# Histogram of Years of Education
adult_data["education-num"].hist(bins=24);
# Density Plot of Years of Education
adult_data["education-num"].plot.density();
# Scatterplot of hours per week and years of education
adult_data.plot.scatter("hours-per-week", "education-num");
# Scatterplot of age and years of education
adult_data.plot.scatter("age", "education-num");<jupyter_output><empty_output><jupyter_text>Stretch Goals - Other types and sources of dataNot all data comes in a nice single file - for example, image classification involves handling lots of image files. You still will probably want labels for them, so you may have tabular data in addition to the image blobs - and the images may be reduced in resolution and even fit in a regular csv as a bunch of numbers.If you're interested in natural language processing and analyzing text, that is another example where, while it can be put in a csv, you may end up loading much larger raw data and generating features that can then be thought of in a more standard tabular fashion.Overall you will in the course of learning data science deal with loading data in a variety of ways. Another common way to get data is from a database - most modern applications are backed by one or more databases, which you can query to get data to analyze. We'll cover this more in our data engineering unit.How does data get in the database? Most applications generate logs - text files with lots and lots of records of each use of the application. Databases are often populated based on these files, but in some situations you may directly analyze log files. The usual way to do this is with command line (Unix) tools - command lines are intimidating, so don't expect to learn them all at once, but depending on your interests it can be useful to practice.One last major source of data is APIs: https://github.com/toddmotto/public-apisAPI stands for Application Programming Interface, and while originally meant e.g. the way an application interfaced with the GUI or other aspects of an operating system, now it largely refers to online services that let you query and retrieve data. You can essentially think of most of them as "somebody else's database" - you have (usually limited) access.*Stretch goal* - research one of the above extended forms of data/data loading. See if you can get a basic example working in a notebook. Image, text, or (public) APIs are probably more tractable - databases are interesting, but there aren't many publicly accessible and they require a great deal of setup. <NAME> found [this article](https://www.dataquest.io/blog/python-api-tutorial/) that goes through using the International Space Station API. I'm going to try it out.<jupyter_code># To make API requests over http I'll be using rthe requests library
import requests
# Make a GET request to OpenNotify for the astronauts in space
astros_url = "http://api.open-notify.org/astros.json"
response = requests.get(astros_url)
# To test it out, simply print the response status code
print(response.status_code)
# Now try to get a response
data = response.json()
print(data["number"])
print(data)
# Now to get the current location of the ISS
iss_now_url = "http://api.open-notify.org/iss-now.json"
response = requests.get(iss_now_url)
iss_now_data = response.json()
print(iss_now_data)<jupyter_output>{'message': 'success', 'timestamp': 1567552812, 'iss_position': {'latitude': '-9.9156', 'longitude': '-10.3329'}}<jupyter_text>------ I spent some more time looking around and found another API that I thought I could use and that might be interesting -  [OpenBrewery DB API.](https://www.openbrewerydb.org/)<jupyter_code># The base endpoint for the openbrewery DB
brew_url = "https://api.openbrewerydb.org/breweries"
r = requests.get(brew_url)
brew_data = r.json()
# Create a pandas dataframe from the converted json response
brew_df = pd.DataFrame(brew_data)
brew_df.head()
print(brew_df.dtypes)<jupyter_output>brewery_type    object
city            object
country         object
id               int64
latitude        object
longitude       object
name            object
phone           object
postal_code     object
state           object
street          object
tag_list        object
updated_at      object
website_url     object
dtype: object
{'fraction_non_alphanumeric': 0.07740164684354986, 'fraction_numerical': 0.007959743824336687, 'mean_word_length': 4.599897540983607, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '300825', 'license': 'MIT', 'n_tokens_mistral': 3145, 'n_tokens_neox': 2950, 'n_words': 1424, 'path': 'module2-loadingdata/LS_DS_112_Loading_Data_Assignment.ipynb', 'repo_name': 'tobias-fyi/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '10939'}