<jupyter_start><jupyter_text>Tutorial :: Pandas, numpy, scipy and matplotlib  Tutorial- **Jupyter Notebook:** The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more. https://jupyter.org/- **Pandas:** Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. https://pandas.pydata.org/- **Numpy:** Besides its scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. http://www.numpy.org/- **Scipy:** SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering. https://www.scipy.org/- **Matplotlib:** Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. Matplotlib can be used in Python scripts, the Python and IPython shells, the Jupyter notebook, web application servers, and four graphical user interface https://matplotlib.org/  Dataset https://www.kaggle.com/harlfoxem/housesalespredictionThis dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014 and May 2015.  Jupyter Notebook The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.- Run a first example:    - Control + ENTER    - Shift + ENTER- Review the toolbar options- Different cell types: Heading, Markdown and Code- Error messages- Stop / Restart a kernel- Download as ...  Pandas Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. https://pandas.pydata.org/<jupyter_code># import the library
%matplotlib inline

import pandas as pd
import numpy as np
import scipy
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Load filesThe I/O API from Pandas support loadinf data from multiple filetypes:https://pandas.pydata.org/pandas-docs/stable/io.html<jupyter_code># read file from csv
df = pd.read_csv('data/kc_house_data.csv')<jupyter_output><empty_output><jupyter_text>Viewing DataFrames<jupyter_code># view a dataframe
df
# set the options to view number of rows and columns
pd.set_option('display.max_columns', 100)
pd.set_option('display.max_rows', 100)
# viewing the first X rows
df.head()
# viewing the last X rows
df.tail()
# view the dataframe index
df.index
# view the dataframe columns
df.columns
# view the dataframe values
df.values
# view the dataframe shape
df.shape<jupyter_output><empty_output><jupyter_text>Sorting DataFrames<jupyter_code># sort the dataframe by price
df.sort_values(by='price')
# sort the dataframe by price descending
df.sort_values(by='price', ascending=False)
# sorting by multiple columns
df.sort_values(by=['bedrooms','bathrooms'], ascending=False)<jupyter_output><empty_output><jupyter_text>Slicing the DataFrames<jupyter_code># get a single column
df['zipcode']
# get multiple columns
df[['lat', 'long']]<jupyter_output><empty_output><jupyter_text>Selecting data<jupyter_code># select rows by a single column
df[df['bedrooms']==4]
# select rows by multiple column
df[(df['bedrooms']==4)&(df['sqft_living']>2000)].sort_values(by='sqft_living')
# select multiple rows by range
df[10:20]
# select multiple rows and columns by range
df.iloc[0:5,0:10]<jupyter_output><empty_output><jupyter_text>Grouping data<jupyter_code># grouping the data by a single column
df.groupby(by='bedrooms')['price'].mean()
# grouping the data by a single column
df.groupby(by='bedrooms')['sqft_living'].mean().round()
# grouping the data by multiple column
df.groupby(by=['bedrooms', 'bathrooms'])['price'].mean().round()
# grouping the data considering multiple aggregating functions
df.groupby(by='bedrooms')['sqft_living'].agg([np.sum, np.mean, np.std, np.median, np.var, np.min, np.max])<jupyter_output><empty_output><jupyter_text>Pivot Tables<jupyter_code># pivot table a dataframe
df.pivot_table(index='zipcode', columns='bedrooms', values='price', aggfunc=np.mean).round()<jupyter_output><empty_output><jupyter_text>Working with Dates / Time<jupyter_code>df.head()
# transform string to datetime
df['date'] = pd.to_datetime(df['date'])
df.head()
# group the prices by date
df.groupby(by='date')['price'].sum()
# create new columns based on the dates (month, year, day of the week)
df['month'] = df['date'].dt.month
df['year'] = df['date'].dt.year
df['dow'] = df['date'].dt.dayofweek
df.head()<jupyter_output><empty_output><jupyter_text>Operations<jupyter_code># create new columns based on other columns
df['sqm_living'] = df['sqft_living']/10.764
df['sqm_lot'] = df['sqft_lot']/10.764
# calculate descriptive statistics for numerical columns
df.describe()
# transpose the dataframe for visualization
df.describe().T
# descriptive statistics
np.mean(df['price'])<jupyter_output><empty_output><jupyter_text>Data Visualization<jupyter_code># Scatter plot
plt.plot(df['price'], df['sqft_living15'], 'o')
plt.show()
# Scatter plot multiple series
plt.figure(figsize=(12,6))
plt.plot(df['price'], df['sqft_living15'], 'o', color='lightgray')
plt.plot(df[df['zipcode']==98002]['price'], df[df['zipcode']==98002]['sqft_living15'], 'o', color='red')
plt.plot(df[df['zipcode']==98039]['price'], df[df['zipcode']==98039]['sqft_living15'], 'o', color='limegreen')
plt.show()
# Line chart
plt.figure(figsize=(12,6))
df_group = df.groupby(by='date')['price'].sum()
plt.plot(df_group.index, df_group.values, '-')
plt.show()
# Histogram
plt.figure(figsize=(12,6))
plt.hist(df['price'], bins=100)
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the price of a house<jupyter_code># prediction with linear regression
from sklearn import linear_model

# Create linear regression object
model = linear_model.LinearRegression()

columns = ['bedrooms', 'sqft_living15', 'grade', 'condition']

# Train the model using the training sets
model.fit(df[columns], df['price'])

# Print the Coefficients
print('Coefficients', np.round(model.coef_,1))
print('Interception', round(model.intercept_,1))
print('')
for i, col in enumerate(columns):
    print(col, round(model.coef_[i],1))

print('')
prediction = model.predict([[4, 3000, 10, 4]])
print('Prediction', prediction.round(1)[0])
def prediction(v):
    # Print the Coefficients
    print('Coefficients', np.round(model.coef_,1))
    print('Interception', round(model.intercept_,1))
    print('')
    for i, col in enumerate(columns):
        print(col, round(model.coef_[i],1))
    print('')
    prediction = model.predict([v])
    print('Prediction', prediction.round(1)[0])
    print('')

# prediction with linear regression
from sklearn import linear_model

# Create linear regression object
model = linear_model.LinearRegression()

columns = ['bedrooms', 'bathrooms', 'sqft_living15', 'grade', 'condition']

train_vector = [3, 2, 2000, 7, 4]

# Train with the lower price/sqft ratio zipcode
df_zipcode = df[df['zipcode']==98002]
model.fit(df_zipcode[columns], df_zipcode['price'])
prediction(train_vector)

# Train with the higher price/sqft ratio zipcode
df_zipcode = df[df['zipcode']==98039]
model.fit(df_zipcode[columns], df_zipcode['price'])
prediction(train_vector)<jupyter_output><empty_output><jupyter_text>Extra Questions<jupyter_code># calculate the total sales by month and year

# calculate the total sales by day of the week

# calculate the average price by zipcode

# identify the zipcodes with highest and lowest price per square foot<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12719841466435472, 'fraction_numerical': 0.014862521674510775, 'mean_word_length': 6.625118035882909, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '89388', 'license': 'MIT', 'n_tokens_mistral': 2650, 'n_tokens_neox': 2507, 'n_words': 740, 'path': 'week2/introduction-pandas-matplotlib-sklearn.ipynb', 'repo_name': 'Clercy/timeseries_class', 'size': '8074'}