<jupyter_start><jupyter_text>Leasson 3: ActivityIn this acitvity, you will work with **Boston Housing Price dataset**. The Boston house-price data has been used in many machine learning papers that address regression problems. You will read the data from a CSV file into a Pandas DataFrame and do some data basic wrangling with it.Following are the details of the attributes of this dataset for your reference. You may have to refer them while answering question on this activity.* **CRIM**: per capita crime rate by town* **ZN**: proportion of residential land zoned for lots over 25,000 sq.ft.* **INDUS**: proportion of non-retail business acres per town* **CHAS**: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)* **NOX**: nitric oxides concentration (parts per 10 million)* **RM**: average number of rooms per dwelling* **AGE**: proportion of owner-occupied units built prior to 1940* **DIS**: weighted distances to five Boston employment centres* **RAD**: index of accessibility to radial highways* **TAX**: full-value property-tax rate per 10,000 dollars* **PTRATIO**: pupil-teacher ratio by town* **B**: 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town* **LSTAT**: % of lower status of the population* **PRICE**: Median value of owner-occupied homes in $1000's  Load necessary libraries<jupyter_code># Write your code here
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Read in the Boston housing data set (given as a .csv file) from the local directory<jupyter_code># Hint: The Pandas function for reading a CSV file is 'read_csv'.
# Don't forget that all functions in Pandas can be accessed by syntax like pd.{function_name}
# write your code here
df=pd.read_csv("Boston_housing.csv")<jupyter_output><empty_output><jupyter_text>Check first 10 records<jupyter_code># Write your code here
df.head(10)<jupyter_output><empty_output><jupyter_text>In total, how many records are there?<jupyter_code># Write your code here to answer the question above
df.shape<jupyter_output><empty_output><jupyter_text>Create a smaller DataFrame with columns which do not include 'CHAS', 'NOX', 'B', and 'LSTAT'<jupyter_code># Write your code here
df1=df[['CRIM','ZN','INDUS','RM','AGE','DIS','RAD','TAX','PTRATIO','PRICE']]<jupyter_output><empty_output><jupyter_text>Check the last 7 records of the new DataFrame you just created<jupyter_code># Write your code here
df1.tail(7)<jupyter_output><empty_output><jupyter_text>Can you plot histograms of all the variables (columns) in the new DataFrame?You can of course plot them one by one. But try to write a short code to plot all of them once.***Hint***: 'For loop'!***Bonus problem***: Can you also show each plot with its unique title i.e. of the variable that it is a plot of?<jupyter_code># Write your code here
for c in df1.columns:
    plt.title("Plot of "+c,fontsize=15)
    plt.hist(df1[c],bins=20)
    plt.show()<jupyter_output><empty_output><jupyter_text>Crime rate could be an indicator of house price (people don't want to live in high-crime areas). Create a scatter plot of crime rate vs. Price.<jupyter_code># Write your code here
plt.scatter(df1['CRIM'],df1['PRICE'])
plt.show()<jupyter_output><empty_output><jupyter_text>We can understand the relationship better if we plot _log10(crime)_ vs. Price. Create that plot and make it nice. Give proper title, x-axis, y-axis label, make data points a color of your choice, etc...***Hint***: Try `np.log10` function<jupyter_code># Write your code here
plt.scatter(np.log10(df1['CRIM']),df1['PRICE'],c='red')
plt.title("Crime rate (Log) vs. Price plot", fontsize=18)
plt.xlabel("Log of Crime rate",fontsize=15)
plt.ylabel("Price",fontsize=15)
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Can you calculate the mean rooms per dwelling?<jupyter_code># Write your code here
df1['RM'].mean()<jupyter_output><empty_output><jupyter_text>Can you calculate median Age?<jupyter_code># Write your code here
df1['AGE'].median()<jupyter_output><empty_output><jupyter_text>Can you calculate average (mean) distances to five Boston employment centres?<jupyter_code># Write your code here
df1['DIS'].mean()<jupyter_output><empty_output><jupyter_text>Tricky question: Can you calculate the percentage of houses with low price (< $20,000)?<jupyter_code># Create a Pandas series and directly compare it with 20
# You can do this because Pandas series is basically Numpy array and you have seen how to filter Numpy array
low_price=df1['PRICE']<20
# This creates a Boolean array of True, False
print(low_price)
# True = 1, False = 0, so now if you take an average of this Numpy array, you will know how many 1's are there.
# That many houses are priced below 20,000. So that is the answer. 
# You can convert that into percentage by multiplying with 100
pcnt=low_price.mean()*100
print("\nPercentage of house with <20,000 price is: ",pcnt)<jupyter_output>0      False
1      False
2      False
3      False
4      False
5      False
6      False
7      False
8       True
9       True
10      True
11      True
12     False
13     False
14      True
15      True
16     False
17      True
18     False
19      True
20      True
21      True
22      True
23      True
24      True
25      True
26      True
27      True
28      True
29     False
       ...  
476     True
477     True
478     True
479    False
480    False
481    False
482    False
483    False
484    False
485    False
486     True
487    False
488     True
489     True
490     True
491     True
492    False
493    False
494    False
495    False
496     True
497     True
498    False
499     True
500     True
501    False
502    False
503    False
504    False
505     True
Name: PRICE, Length: 506, dtype: bool

Percentage of house with <20,000 price is:  41.50197628458498
{'fraction_non_alphanumeric': 0.09145299145299145, 'fraction_numerical': 0.043931623931623934, 'mean_word_length': 4.319090909090909, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '341585', 'license': 'MIT', 'n_tokens_mistral': 1981, 'n_tokens_neox': 1775, 'n_words': 743, 'path': 'Lesson03/Activity05/Activity05_Generating_Statistics_from_CSV_File_Solution.ipynb', 'repo_name': 'RileyAfencl/DSC-540', 'size': '5850'}