<jupyter_start><jupyter_text>Exercise Solution: Exploring Data With Jupyter, Pandas, and Matplotlib Fact Finding:Find the answer to each of these questions:* What was the most expensive property sold in the dataset?* How many sales were for less than $10    * How could this possibly be right? (Hint: read the data documentation on Kaggle...)* How many of the properties sold were built prior to 1950?* What is the smallest gross square feet property sold?    * What was the largest?* Which zip code had the fewest number of sales?<jupyter_code># Before we can start, we should import the libraries we're going to use
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
# Then load the data
path_to_ny_sales = '../../datasets/nyc-property/nyc-rolling-sales.csv'
sales_df = pd.read_csv(path_to_ny_sales)
sales_df.head()
# To make things cleaner, I'm also going to drop rows that have a missing:
# price, gross square feet, land square feet, or year built.

# This code is in the example notebook
columns_to_convert = [
    'LAND SQUARE FEET',
    'GROSS SQUARE FEET',
    'SALE PRICE',
    'YEAR BUILT'
]

for column_name in columns_to_convert:
    sales_df[column_name] = pd.to_numeric(sales_df[column_name], errors='coerce')
    sales_df = sales_df[sales_df[column_name].notna()]
    
sales_df.describe()
# What was the most expensive property sold?
  # Actually, we can see this in the information above from the .describe() function!
  # Max sale price was 2.210000e+09 aka 2.21 BILLION DOLLARS!! 
# But you can also find it this way:
sales_df['SALE PRICE'].max()

# Relevant documentation: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.max.html
# How many properties sold were built prior to 1950?
# The easiest way to do this is filter the dataframe, then count the rows.
before_1950 = sales_df[sales_df['YEAR BUILT'] < 1950]
print(len(before_1950))

# Note, you could also use the "count" function, though it gives more information than we need.
  # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.count.html
before_1950.count()
# Smallest and largest gross square feet
  # Once again, this information was in the "describe()" output
    # Smallest: 0 square feet (weird right?)
    # Largest: 3.750565e+06 aka 3,750,565 square feet (HOLY S***)

# Again, you could also find these values using .min() and .max() on the proper columns:
print(sales_df['GROSS SQUARE FEET'].min())
print(sales_df['GROSS SQUARE FEET'].max())

# But, inquiring minds want to know Lets find out what the smallest non-zero property is
# I imagine any 0 values are more likely "missing" than being a property that actually 
# doesn't have a size...
non_zero_gross_sq_feet = sales_df[sales_df['GROSS SQUARE FEET'] != 0]
non_zero_gross_sq_feet['GROSS SQUARE FEET'].min() # 60. Wow, that's a small property. 
# Which zip code had the fewest number of sales?
# The easiest way to do this is to use the .value_counts() function on the ZIP CODE column.
# This will tell us how many times each zip code appears in the overall data.
  # Documentation: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.value_counts.html

print(sales_df['ZIP CODE'].value_counts())
# The data is returned in sorted order based on the count.
# So, zip code 11201 has the most sales: 1324
# and 10803 only has 1 sale. 
# We also learned that there are 180 zip codes represented in this dataset.<jupyter_output>11201    1324
11235    1312
11234    1165
11229     916
11215     899
         ... 
10167       1
11005       1
10006       1
10044       1
10803       1
Name: ZIP CODE, Length: 180, dtype: int64
1324<jupyter_text>Chart Making:Create the following charts:* A boxplot showing how many properties were sold in each borough.    * Use the data documentation to find the names of each borough rather than the 1-5 values.* A pie chart showing the share of sales by borough.    * Use the data documentation to find the names of each borough rather than the 1-5 values.* A boxplot showing the average (mean) sale price of property in each zip code.* A scatterplot showing the sales price by the gross square feet.    * **Bonus points**: show the least squares regression line as well!<jupyter_code># A boxplot showing how many properties were sold in each borough.
# We can use value_counts for this too:
sales_by_borough = sales_df['BOROUGH'].value_counts()

# Replace the numbered boroughs with their names:
  # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html
sales_by_borough.rename(index={
        1: 'Manhattan',
        2: 'Bronx',
        3: 'Brooklyn',
        4: 'Queens',
        5: 'Staten Island'
    }, 
    inplace=True
)

# Make the plot using pandas!
  # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.bar.html
sales_by_borough.plot.bar()
# We already have the data, so a pie chart is very easy to make (as long as you know what function to use):
  # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.pie.html
sales_by_borough.plot.pie()
# A barchart showing the average (mean) sale price of property in each zip code.
# This one is a bit trickier... we need to group the data based on the zip code:
  # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html
sales_and_zip = sales_df[['ZIP CODE', 'SALE PRICE']]

sales_grouped_by_zip = sales_and_zip.groupby(['ZIP CODE']).mean()
sales_grouped_by_zip.plot.bar(figsize=(100, 20))
# Hmmmm... that's not very interesting or usable. 
# 180 values is just too many, and the range of sale prices is also too broad.
sales_and_zip = sales_df[['ZIP CODE', 'SALE PRICE']]

sales_grouped_by_zip = sales_and_zip.groupby(['ZIP CODE']).mean()

# Now, lets sort it by average sale price and just display the top 15
sorted_sales_by_zip = sales_grouped_by_zip.sort_values(by=['SALE PRICE'], ascending=False)
top_15 = sorted_sales_by_zip[0:15]
top_15.plot.bar()

# Fun fact, the 10167 zip code is ONE CITY BLOCK on Park Ave.
  # (https://www.zip-codes.com/zip-code/10167/zip-code-10167.asp)
  # https://en.wikipedia.org/wiki/245_Park_Avenue  
# Okay, one more, lets leave out the 245 Park Street sale 
# and look at the next top 15 zips to get a better picture overall:
sorted_sales_by_zip[1:16].plot.bar()
# Finally, A scatterplot showing the sales price by the gross square feet. 
sales_df.plot.scatter(x='GROSS SQUARE FEET', y='SALE PRICE')
# To get the regression line we need to do a bit more work and use some lower level libraries directly
  # pyplot plot function: https://matplotlib.org/3.2.1/api/_as_gen/matplotlib.pyplot.plot.html
  # numpy polyfit: https://numpy.org/doc/1.18/reference/generated/numpy.polyfit.html
x = sales_df['GROSS SQUARE FEET']
y = sales_df['SALE PRICE']

slope, y_intercept = np.polyfit(x, y, 1) # one is for "first degree polynomial" aka, a line.

# Plot the scatter, then the line, then show the plot:
plt.plot(x, y, 'o') # 'o' is for "dots"
plt.plot(x, y_intercept + slope * x, '-') # '-' is for "line"<jupyter_output>393539.4820124962 207.02811915510165
{'fraction_non_alphanumeric': 0.0978687605159843, 'fraction_numerical': 0.035193494111048794, 'mean_word_length': 4.153901734104046, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 7, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '298299', 'license': 'Unlicense', 'n_tokens_mistral': 2536, 'n_tokens_neox': 2301, 'n_words': 855, 'path': 'lessons/04-working-with-libraries/03e-exercise-soultion.ipynb', 'repo_name': 'ShreyasSub/intro-python-for-machine-learning', 'size': '7132'}