<jupyter_start><jupyter_text>I began tackling this project by focusing on the first requirement as outlined by Dr <NAME>; Describe (20%) Describe (20%) : *Good summary of the dataset, repository well laid-out and organised. **Reasonable commits to the repository.**Create a git repository and make it available online for the lecturer to clone.* *The repository should contain all your work for this assessment. **Within the repository, create a jupyter notebook that uses descriptive statistics and plots to describe the Boston House Prices  dataset. **This part is worth 20% of your overall mark.*My first goal was to research online for a good summary of the dataset, i.e. figure out how I can upload the dataset to my Jupyter Notebook, and then summarise it. I reviewed the Boston Standard Metropolitan Statistical Area - Boston house prices dataset on https://www.kaggle.com/c/boston-housing.According to the description on www.kaggle.com; *this dataset details the Housing Values in Suburbs of Boston. Data description; The Boston data frame has 506 rows and 14 columns.*We are provided with a ‘key’ which describes what the information in each column represents;**crim**per capita crime rate by town.**zn**proportion of residential land zoned for lots over 25,000 sq.ft.**indus**proportion of non-retail business acres per town.**chas**Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).**nox**nitrogen oxides concentration (parts per 10 million).**rm**average number of rooms per dwelling.**age**proportion of owner-occupied units built prior to 1940.**dis**weighted mean of distances to five Boston employment centres.**rad**index of accessibility to radial highways.**tax**full-value property-tax rate per $10,000.**ptratio**pupil-teacher ratio by town.**black**1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town.**lstat**lower status of the population (percent).**medv**median value of owner-occupied homes in $1000s.Source;*	<NAME>. and <NAME>. (1978) Hedonic prices and the demand for clean air. J. Environ. Economics and Management 5, 81–102.*	<NAME>., <NAME>. and <NAME>. (1980) Regression Diagnostics. Identifying Influential Data and Sources of Collinearity. New York: Wiley.I was immediately struck by one piece of information mentioned;**black**1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town.This seemed very strange to me, and slightly offensive. I wondered if this would be explained a bit more when I became more familiar with the dataset. Over the course of this module a fellow student raised this same concern on our Outlook group which is used for news, announcements and discussion. Dr. Ian addressed this on our Outlook group, so I felt less confused after this.When I clicked on ‘Data’ on www.kaggle.com I expected to be presented with the dataset, but nothing appeared. I revisited Moodle to see if Ian had mentioned that we need to create an account on on www.kaggle.com to access the dataset. I noticed one comment from Ian in our Outlook group which is used for news, announcements and discussion. Ian’s comment was made in reply to a question posted by one of my fellow students;*You'll find the common datasets in many different configurations - part of the assignment is to source the dataset in a form conducive to what you need to do with it.* This is important knowledge for me while completing this assignment. It also explains why I couldn’t immediately view the dataset in https://www.kaggle.com/c/boston-housing.I discovered a website 'packtpub' which seemed to explain how to load the dataset in Jupyter Notebook;https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset.This website describes the dataset as follows;*The dataset we'll look at in this section is the so-called Boston housing dataset. It contains US census data concerning houses in various areas around the city of Boston. Each sample corresponds to a unique area and has about a dozen measures. We should think of samples as rows and measures as columns. The data was first published in 1978 and is quite small, containing only about 500 samples.*I read through the information supplied on this website which provides examples of code to load the dataset into a Jupyter Notebook, along with code to Explore the Boston housing dataset and an introduction to Predictive Analytics with Jupyter Notebooks. I thought that I could slightly adapt the code provided by this website for my purposes, in order to meet the requirements laid out by Dr. Ian for the second portion of the assessment;*	(20%) Infer: To the jupyter notebook, add a section where you use inferential statistics to analyse whether there is a signiﬁcant diﬀerence in median house prices between houses that are along the Charles river and those that aren’t. You should explain and discuss your ﬁndings within the notebook. This part is also worth 20%.So, I decided to follow the steps outlined on the https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset website. The website describes how to complete the following; “Load the data into Jupyter using a Pandas DataFrame”.*Oftentimes, data is stored in tables, which means it can be saved as a comma-separated variable (CSV) file. This format, and many others, can be read into Python as a DataFrame object, using the Pandas library. Other common formats include tab-separated variable (TSV), SQL tables, and JSON data structures. Indeed, Pandas has support for all of these. In this example, however, we are not going to load the data this way because the dataset is available directly through scikit-learn.* The https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset website also mentions; *An important part after loading data for analysis is ensuring that it's clean. For example, we would generally need to deal with missing data and ensure that all columns have the correct datatypes. The dataset we use in this section has already been cleaned, so we will not need to worry about this.*(Source: https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset).I followed the website’s recommended steps and the code provided as follows;<jupyter_code>from sklearn import datasets
boston = datasets.load_boston()
type (boston)
from sklearn.utils import Bunch
Bunch?
['DESCR', 'target', 'data', 'feature_names']
boston['DESCR']<jupyter_output><empty_output><jupyter_text>As we can see from the error above 'NameError: name 'boston' is not defined' I ran into some issues while running the code supplied on https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset. I played around with the code and managed to get the desired result;<jupyter_code>boston = load_boston()
boston['DESCR']
['DESCR', 'target', 'data', 'feature_names']
boston = load_boston()
boston['DESCR']<jupyter_output><empty_output><jupyter_text>I managed to load the dataset description into my Jupyter notebook as we can see in the above cell. This resembles the outcome of Step 5 as described on https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789958171/1/ch01lvl1sec04/our-first-analysis-the-boston-housing-dataset; *Run the cell to print the dataset description contained in boston['DESCR'] .*I continued on to Step 6 as outlined on the website;<jupyter_code>import pandas as pd
pd.DataFrame?<jupyter_output><empty_output><jupyter_text>The website informs me that the *The docstring reveals the DataFrame input parameters. We want to feed in boston['data'] for the data and use boston['feature_names'] for the headers,* (https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789958171/1/ch01lvl1sec04/our-first-analysis-the-boston-housing-dataset).Progressing on to Step 7;<jupyter_code>#What does the data look like?
boston[ 'data' ]<jupyter_output><empty_output><jupyter_text>I tried to copy the code supplied on the website, and initially made a mistake as I encountered the above error. I realised I had inserted 2 spaces before & after the ‘data’ which I should not have done – I amended this and tried again;<jupyter_code>#What does the data look like?
boston ['data']<jupyter_output><empty_output><jupyter_text>This did not resolve the issue, as I still encounter an error message. I tried playing around with the code some more as shown below to see if I could get it working;<jupyter_code>#What does the data look like?
boston ['data']
#What does the data look like?
boston = load_boston()
boston['data']
boston = load_boston()
boston['data']<jupyter_output><empty_output><jupyter_text>I kept encountering the same error again ‘NameError: name 'boston' is not defined’.    I established that there must be something wrong with a previous step in my Jupyter Notebook, and I suspect it is that I have done something wrong at step 4. I searched online for guidance and found some information pertaining to this error message;**NameError: global name '---' is not defined**    *Python knows the purposes of certain names (such as names of built-in functions like print). Other names are defined within the program (such as variables). If Python encounters a name that it doesn't recognize, you'll probably get this error.Some common causes of this error include:** Forgetting to give a variable a value before using it in another statement* Misspelling the name of a built-in function (e.g., typing "inpit" instead of "input")(Source: https://www2.cs.arizona.edu/people/mccann/errors-pythonFour)I played around with my Jupyter Notebook, changing the code in the cells and eventually realised that if I ran all steps again from 1 to 7 it worked correctly for me;<jupyter_code>from sklearn import datasets
boston = datasets.load_boston()
type (boston)
from sklearn.utils import Bunch
Bunch?
['DESCR', 'target', 'data', 'feature_names']
boston['data']
boston['data'].shape<jupyter_output><empty_output><jupyter_text>As per the https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset website;*Looking at the output, we see that our data is in a 2D NumPy array. Running the command boston['data'].shape returns the length (number of samples) and the number of features as the first and second outputs, respectively.*At this point I compared the results in Out[16] above with the Boston Data Set according to https://www.kaggle.com/c/boston-housing and noticed 2 discrepancies; 1.	medv – This is included in the Boston Standard Metropolitan Statistical Area - Boston house prices dataset as per https://www.kaggle.com/c/boston-housingAccording to this description, this dataset details the Housing Values in Suburbs of Boston. *Data description; The Boston data frame has 506 rows and 14 columns.We are provided with a ‘key’ which describes what the information in each column represents, and **medv** (median value of owner-occupied homes in $1000s)is one of the 14 keys.* However, **medv** is not included in the output in my Jupyter Notebook or on the https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789958171/1/ch01lvl1sec04/our-first-analysis-the-boston-housing-dataset website. According to the results in Jupyter Notebook after running the code from the website, there are only 13 keys.<jupyter_code>boston['feature_names']<jupyter_output><empty_output><jupyter_text>2.	**black**This is included in the Boston Standard Metropolitan Statistical Area - Boston house prices dataset as per https://www.kaggle.com/c/boston-housing.According to this description, this dataset details the Housing Values in Suburbs of Boston. *Data description; The Boston data frame has 506 rows and 14 columns.We are provided with a ‘key’ which describes what the information in each column represents, and **black** (1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town) is one of the 14 keys.*However, this is represented by ‘B’ in my Jupyter Notebook as shown in the above output cell, and on the https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789958171/1/ch01lvl1sec04/our-first-analysis-the-boston-housing-dataset website. I continued on to Step 8 as per the https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789958171/1/ch01lvl1sec04/our-first-analysis-the-boston-housing-dataset website;8. Load the data into a Pandas DataFrame df by running the following:df = pd.DataFrame(data=boston['data'], columns=boston['feature_names'])<jupyter_code>df = pd.DataFrame(data=boston['data'], 
columns=boston['feature_names'])<jupyter_output><empty_output><jupyter_text>As we can see from the above I encountered a similar situation to the one I encountered previously; NameError: name 'pd' is not defined.I decided to address this in the same manner as I had previously and run steps 1 to 8 again;<jupyter_code>from sklearn import datasets
boston = datasets.load_boston()
type (boston)
from sklearn.utils import Bunch
Bunch?
['DESCR', 'target', 'data', 'feature_names']
boston['data']
boston['data'].shape
boston['feature_names']
df = pd.DataFrame(data=boston['data'], 
columns=boston['feature_names'])
df = pd.DataFrame(data=boston['data'], 
columns=boston['feature_names'])<jupyter_output><empty_output><jupyter_text>Still the error message appeared.According to the website 'In machine learning, the variable that is being modeled is called the target variable; it's what you are trying to predict given the features. For this dataset, the suggested target is MEDV, the median house value in 1,000s of dollars.'I decided to try step 9 of the website to see if this helps resolve the issue.The instructions for step 9 were as follows;Run the next cell to see the shape of the target:<jupyter_code># Still need to add the target variable
boston['target'].shape<jupyter_output><empty_output><jupyter_text>The result is the same for me as it is described on the website. As per the website 'We see that it has the same length as the features, which is what we expect. It can therefore be added as a new column to the DataFrame.'Step 10; Add the target variable to df by running the cell with the following:df['MEDV'] = boston['target']<jupyter_code>df['MEDV'] = boston['target']<jupyter_output><empty_output><jupyter_text>Step 11 according to the website; Move the target variable to the front of df by running the cell with the following code:<jupyter_code>y = df['MEDV'].copy() 
del df['MEDV']
df = pd.concat((y, df), axis=1)<jupyter_output><empty_output><jupyter_text>This is done to distinguish the target from our features by storing it to the front of our DataFrame.Here, we introduce a dummy variable y to hold a copy of the target column before removing it from the DataFrame. We then use the Pandas concatenation function to combine it with the remaining DataFrame along the 1st axis (as opposed to the 0th axis, which combines rows).Step 12;Implement df.head() or df.tail() to glimpse the data and len(df) to verify that number of samples is what we expect. Run the next few cells to see the head, tail, and length of df:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>This is not the correct result according to the website. I think this may be because I've forgotten to include & run one step;boston = load_boston()boston['DESCR'].I will start from Step 1 again;<jupyter_code>from sklearn import datasets
boston = datasets.load_boston()
type (boston)
from sklearn.utils import Bunch
Bunch?
['DESCR', 'target', 'data', 'feature_names']
boston = load_boston()
boston['DESCR']
boston = load_boston()
boston['DESCR']
boston = load_boston()
boston['DESCR']<jupyter_output><empty_output><jupyter_text>Stuck here. I tried the below and it worked;<jupyter_code>boston['DESCR']
import pandas as pd
pd.DataFrame?
#What does the data look like?
boston[ 'data' ]
boston['data'].shape
boston['feature_names']
df = pd.DataFrame(data=boston['data'], 
columns=boston['feature_names'])
# Still need to add the target variable
boston['target'].shape
df['MEDV'] = boston['target']
y = df['MEDV'].copy() 
del df['MEDV']
df = pd.concat((y, df), axis=1)
df.head()
df.tail()<jupyter_output><empty_output><jupyter_text>Finally, I got the desired result for steps 12 & 13 as per to the website (as seen above). I implemented df.head() and df.tail() *to glimpse the data and len(df) to verify that number of samples is what we expect. I ran these cells to see the head, tail, and length of df.* As per the website; *each row is labeled with an index value, as seen on the left side of the table. By default, these are a set of integers starting at 0 and incrementing by one for each row.* I notice that for all 10 records highlighted above the **CHAS** category shows 0. As per https://www.kaggle.com/c/boston-housing; **chas** - Charles River dummy variable (= 1 if tract bounds river; 0 otherwise). This means for the 10 records above the tract does not bound the river for these cases. I assessed the **medv** (median value of owner-occupied homes in $1000s) for these 10 examples where the river is not bound, and noticed that the lowest figure shown is 11.9 and highest is 36.2. The general range is between 11.9 and 36.2. Based on this I calculated that the general median value of owner-occupied homes range for these 10 records which do not bound the river is between 11,900 dollars & 36,200 dollars. Beginning again due to an error : NameError: name 'df' is not defined. I think this occurrs each time I step away from my notebook and then attempt to run code again when I return to it.  Step 1 (Taken from https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789958171/1/ch01lvl1sec04/our-first-analysis-the-boston-housing-dataset);<jupyter_code>from sklearn import datasets
boston = datasets.load_boston()
type (boston)
from sklearn.utils import Bunch
Bunch?
['DESCR', 'target', 'data', 'feature_names']
boston['DESCR']
import pandas as pd
pd.DataFrame?
#What does the data look like?
boston[ 'data' ]
boston['data'].shape
boston['feature_names']
df = pd.DataFrame(data=boston['data'], 
columns=boston['feature_names'])
# Still need to add the target variable
boston['target'].shape
df['MEDV'] = boston['target']
y = df['MEDV'].copy() 
del df['MEDV']
df = pd.concat((y, df), axis=1)
df.head()
df.tail()
len(df)
 df.dtypes<jupyter_output><empty_output><jupyter_text>According to https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset *printing df.dtypes will show the datatype contained within each column.For this dataset, we see that every field is a float and therefore most likely a continuous variable, including the target. This means that predicting the target variable is a regression problem.* As per the website *The next thing we need to do is clean the data by dealing with any missing data, which Pandas automatically sets as NaN values. These can be identified by running df.isnull() , which returns a Boolean DataFrame of the same shape as df. To get the number of NaN's per column, we can do df.isnull().sum() . Run the next cell to calculate the number of NaNvalues in each column:*<jupyter_code>df.isnull()
# Identify and NaNs
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>The website tells us that *For this dataset, we see there are no NaN's, which means we have no immediate work to do in cleaning the data and can move on.*<jupyter_code>df.describe().T<jupyter_output><empty_output><jupyter_text>The website instructs me to run the above code as *This computes various properties including the mean, standard deviation, minimum, and maximum for each column. This table gives a high-level idea of how everything is distributed. Note that we have taken the transform of the result by adding a .T to the output; this swaps the rows and columns. Going forward with the analysis, we will specify a set of columns to focus on.* Source: https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-datasetNext step: *Run the cell where these "focus columns" are defined:*<jupyter_code>cols = ['RM', 'AGE', 'TAX', 'LSTAT', 'MEDV']
df[cols].head()<jupyter_output><empty_output><jupyter_text>I altered the code provided on the website slightly for my own purposes. The code as per the website is;cols = ['RM', 'AGE', 'TAX', 'LSTAT', 'MEDV'] I altered this to cols = ['CHAS', 'AGE', 'TAX', 'LSTAT', 'MEDV']The reason behind this is that I'd like to focus on the Charles River and Pricing aspects to prepare to address the second requirement for this assignment as set by Dr. <NAME>.  Infer (20%) Infer (20%):*Add a section where you use inferential statistics to analyse whether there is a signiﬁcant diﬀerence in median house prices between houses that are along the Charles river and those that aren’t. You should explain and discuss your ﬁndings within the notebook. This part is also worth 20%.* So, I replaced **'RM'** with **'CHAS'** (Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).), and kept all financial related topics in the code, i.e. **'MEDEV'** (median value of owner-occupied homes in USD 1,000s), and **'TAX'**(full-value property-tax rate per USD 10,000.).<jupyter_code>cols = ['CHAS', 'AGE', 'TAX', 'LSTAT', 'MEDV']<jupyter_output><empty_output><jupyter_text>*This subset of columns can be selected from df using square brackets. Display this subset of the DataFrame by running df[cols].head() :*<jupyter_code>df[cols].head()<jupyter_output><empty_output><jupyter_text>As per the website; *To look for patterns in this data, we can start by calculating the pairwise correlations using pd.DataFrame.corr. Calculate the pairwise correlations for our selected columns by running the cell containing the following code:*<jupyter_code>df[cols].corr()<jupyter_output><empty_output><jupyter_text>As per the website;*This resulting table shows the correlation score between each set of values. Large positive scores indicate a strong positive (that is, in the same direction) correlation. As expected, we see maximum values of 1 on the diagonal.* I follow the guidance and code provided on the website to create a 'heatmap' which will have the **CHAS** and **medev** factors included so I can begin to easily compare the correlation between median house prices between houses that are along the Charles river and those that aren’t; *Instead of straining our eyes to look at the preceding table, it's nicer to visualize it with a heatmap. This can be done easily with Seaborn.Run the next cell to initialize the plotting environment, as discussed earlier in the chapter. Then, to create the heatmap, run the cell containing the following code:*<jupyter_code>import matplotlib.pyplot as plt 
import seaborn as sns
%matplotlib inline
ax = sns.heatmap(df[cols].corr(),
cmap=sns.cubehelix_palette(20, light=0.95,
dark=0.15))
ax.xaxis.tick_top() # move labels to the top
plt.savefig('../figures/lesson-1-boston-housing-corr.png', bbox_inches='tight', dpi=300)<jupyter_output><empty_output><jupyter_text>*We call sns.heatmap and pass the pairwise correlation matrix as input. We use a custom color palette here to override the Seaborn default. The function returns a matplotlib.axes object which is referenced by the variable ax. The final figure is then saved as a high resolution PNG to the figures folder.**For the final step in our dataset exploration exercise, we'll visualize our data using Seaborn's pairplot function.*Source: https://subscription.packtpub.com/book/programming/9781789804744/1/ch01lvl1sec11/our-first-analysis-the-boston-housing-dataset<jupyter_code>sns.pairplot(df[cols],
plot_kws={'alpha': 0.6},
diag_kws={'bins': 30})<jupyter_output><empty_output><jupyter_text>*Having previously used a heatmap to visualize a simple overview of the correlations, this plot allows us to see the relationships in far more detail.* I notice that for all results which pertain to **CHAS** they are very focused on 0 or 1, there are none at 0.5. I presume the reason for this is because as per https://www.kaggle.com/c/boston-housing **chas** - *Charles River dummy variable (= **1** if tract bounds river; **0** otherwise).*I notice that for the result on the bottom left corner of the above, **MEDEV**, **CHAS**;* **CHAS** 0.0 (houses which do not run alongside the river) ranges from **MEDEV** (median value of owner-occupied homes in USD 1000s) 0 - 50, and * **CHAS** 1.0 (houses which are on the river) start at **MEVED** 15, and ranges form around 15 to 50(median value of owner-occupied homes in USD 1000s). I.e. Houses which do not border the river range from less than USD 15,000 to USD 50,000, and houses which do border the river are valued between a minimum of USD 15,000 and USD 50,000.Houses which border the river are valued higher than those which do not border the river.To me, this indicates that houses on the Charles River (1) have a higher value as the starting price is 1.5 to 50, whereas starting prices for houses not on the river start as low as 0 on the chart. For the result in the top right corner, we see houses with 1 (located on Charles River) start at a **MEDEV** of approx 15, whereas houses not located on the river (0) start around 5.  Predict (60%) Predict (60%) :*Use keras to create a neural network that can predict the median house price based on the other variables in the dataset.* **Etain's notes on Keras based on review of Dr. I<NAME>'s lecture videos on this topic 2019;**Etain’s notes from video ‘Individual neurons in keras’ (2);Keras is a package for Python which can be used to make predictions. *Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.* *It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.* (https://keras.io/)Based on Ian’s video ‘Individual neurons in keras‘, it sounds like I will be able to use keras in this assignment to complete the below requirements;Within the repository, create a jupyter notebook that uses descriptive statistics and plots to describe the Boston House Prices dataset. Again using the same notebook, use keras to create a neural network that can predict the median house price based on the other variables in the dataset.Neurons are mentioned.Ian demonstrates how to insert content to keras in order to have the exact same content expelled/output by keras. (1.45min)Ian mentioned that the following 3 commands are the main commands I will use to create my neuron network;<jupyter_code># Create a new neural network.
m = kr.models.Sequential()

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, input_dim=1, activation="linear", kernel_initializer='ones', bias_initializer='zeros'))

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")<jupyter_output><empty_output><jupyter_text>(Source: https://github.com/ianmcloughlin/jupyter-teaching-notebooks/blob/master/keras-neurons.ipynb)Input – dataset. Output – you sometimes have an idea what this will look like already. E.g. the dataset could have several values, but you want the output to have only 2 potential outcomes such as ‘yes’ and ‘no’.* Sequential – explained at 39mins* Dense – explained at 41mins*TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.*(https://www.tensorflow.org/) I reviewed Ian's instructions on how to install the Keras package. I tried following his instructions but for some reason I continuously ran into issues on my system. I also attempted to follow the instruction on the Keras website (https://keras.io/) but I found it too confusing.I researched online and came accros a YouTube video which described how to complete the install of Keras & Tensorflow via the Anaconda Navigator https://www.youtube.com/watch?v=V9cDjjRXS08.However, this did not work for me as neither Tensorflow or Keras were available under ' Not Installed' when I searched for them. I decided to review the comments in our Outlook group which is used for news, announcements and discussion to see if any other students encountered issues.One student advised that they followed the instructions from YouTube video https://www.youtube.com/watch?v=59duINoc8GM. I decided to attempt the same. I noticed while watching this video that I had been running Ian's instruction in the incorrect command prompt area - I should have been opening the Anaconda Prompt (ForAnaconda) and then attempting "-conda install -c conda-forge keras tensorflow". I tried this. Something seemed to be happening so I let the system run for a few minutes.I was hopeful that Keras and Tensorflow had been successfully installed. I can now see 'Keras' and 'Tensorflow' in my Anaconda Navigator under 'Installed'.<jupyter_code>import keras as kr<jupyter_output>Using TensorFlow backend.<jupyter_text>It seems from the above as tough I have installed everything correctly, so I moved on to Ian's next video 'Individual neurons in keras'.<jupyter_code>import keras as kr
import numpy as np
import mathplotlib.pyplot as plt
import keras as kr
import numpy as np
import mathplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>I copy Ian's code;<jupyter_code># Create a new neural network.
m = kr.models.Sequential()

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, input_dim=1, activation="linear", kernel_initializer='ones', bias_initializer='zeros'))

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.0, 10.0, 1)

# Run each x value through the neural network.
y = m.predict(x)
x
y
# Plot the values.
plt.plot(x, y, 'k.')
# Plot the values.
plt.plot(x, y, 'k.')<jupyter_output><empty_output><jupyter_text>I encounter the above error so I decide to run Ian's code again from the beginning to see if this helps, although I am concerned that this error is appearing due to the ModuleNotFoundError: No module named 'mathplotlib' error.<jupyter_code>import keras as kr
import numpy as np
import mathplotlib.pyplot as plt
# Create a new neural network.
m = kr.models.Sequential()

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, input_dim=1, activation="linear", kernel_initializer='ones', bias_initializer='zeros'))

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.0, 10.0, 1)

# Run each x value through the neural network.
y = m.predict(x)
x
y
# Plot the values.
plt.plot(x, y, 'k.')
# Plot the values.
plt.plot(x, y, 'k.')<jupyter_output><empty_output><jupyter_text>Unfortunately, it appears as though I cm having issues generating the plot as I encounter the above error. I decide to Google 'NameError: name 'plt' is not defined'.I found a page which seems relevant; https://github.com/konstantint/matplotlib-venn/issues/33The notes on this page suggest I try 'from matplotlib import pyplot as plt'.<jupyter_code>from matplotlib import pyplot as plt
import keras as kr
import numpy as np
import mathplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Still I encounter the same error.I try not to get bogged down at this point and remind myself again what outcome I'm looking for using Keras;*Use keras to create a neural network that can predict the median house price based on the other variables in the dataset.*I consider using the 3 main commands with some figures from the Boston House Prices dataset as input instead of the figures in Dr. Ian's example. I decide to give this a go substituting some of Ian's figures in his example with the **MEDEV** figures from the dataset - x = np.arange(0.175260, 0.376955, 1.000000);<jupyter_code># Create a new neural network.
m = kr.models.Sequential()

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, input_dim=1, activation="linear", kernel_initializer='ones', bias_initializer='zeros'))

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.175260, 0.376955, 1.000000)

# Run each x value through the neural network.
y = m.predict(x)
x
y<jupyter_output><empty_output><jupyter_text>I'd like to create a network with more than one Neuron so I copy Ian's code again but instead of 1 I insert 10 figures;<jupyter_code># Create a new neural network.
m = kr.models.Sequential()

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, 2, 3, 4, 5, 6, 7, 8, 9, 10 input_dim=1, activation="linear", kernel_initializer='ones', bias_initializer='zeros'))

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create a new neural network.
m = kr.models.Sequential()

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(10 input_dim=1, activation="linear", kernel_initializer='ones', bias_initializer='zeros'))

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")<jupyter_output><empty_output><jupyter_text>I try to substitute 1 for 0 figures to create a network with more than one Neuron, but I encounter SyntaxError: invalid syntax.I review a resporce posted by Ian on Moodle - 'Playing with neurons' Jupyter Notebook (https://nbviewer.jupyter.org/github/ianmcloughlin/jupyter-teaching-notebooks/blob/master/playing-with-neurons.ipynb). I notice here Ian has included the import matplotlib.pyplot as plt as the first line of code, so I decide to attempt this too;<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import keras as kr
plt.rcParams['figure.figsize'] = (10, 10)
x = np.linspace(-10.0, 10.0, 2000)

y_l = (3.0 * x) + 5.0
y_p = (x * x) - 41.0
y_s = 10.0 * np.sin(2.0 * x)
plt.plot(x, y_l, label='$3x + 5$')
plt.plot(x, y_p, label='$x^2 - 41$')
plt.plot(x, y_s, label='$10 \sin(2x)$')

plt.legend()<jupyter_output><empty_output><jupyter_text>I followed the above code as supplied by Ian in https://nbviewer.jupyter.org/github/ianmcloughlin/jupyter-teaching-notebooks/blob/master/playing-with-neurons.ipynb and get the same result on the graph as in Ian's Notebook. I decide to substitute some of the figures with the **CHAS** and **MEDEV** figures from the Boston House Prices dataset.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import keras as kr
plt.rcParams['figure.figsize'] = (10, 10)
x = np.linspace(0, 1, 2000)

y_l = (24.0 * x) + 5.0
y_p = (x * x) - 0.376955
y_s = 10.0 * np.sin(2.0 * x)
plt.plot(x, y_l, label='$24x + 5$')
plt.plot(x, y_p, label='$x^2 - 0.376955$')
plt.plot(x, y_s, label='$10 \sin(2x)$')

plt.legend()<jupyter_output><empty_output><jupyter_text>Etain's explination of where I came up with the figures;x = np.linspace(0, 1, 2000)y_l = (3.0 * x) + 5.0y_p = (x * x) - 41.0y_s = 10.0 * np.sin(2.0 * x)__________________________________0 = not on river1 = house on river2000 = left as is in Ian's code_________________________y_l = (24.0 * x) + 5.024 from MEDEV, everything else left as is_________________y_p = (x * x) - 0.376955- 0.376955 taken from MEDEV___________y_s = 10.0 * np.sin(2.0 * x)Left as it I think I should attempt a plot with keyword strings as shown on the Mathplotlib Pyplot Tutorial page to display the results in their best form https://matplotlib.org/3.1.1/tutorials/introductory/pyplot.html. I know that my code thusfar has not 'trained' my neural network, so this is something I need to work on in order to predict the median house price based on the other variables in the dataset. I wonder if I can use the 'Two neurons' approach demonstrated by Ian. I would consider using the two factors **chas** (Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)), and **tax** (full-value property-tax rate per USD 10,000) to determine the **medv** (median value of owner-occupied homes in USD 1000s). I know for sure I have some figures above for **chas** and **medev**, and **tax** so it would be easy for me to focus on these three factors.I would still be fulfilling the requirements laid out for this assessment, as Dr. Ian has stated "You are free to interpret this as you wish — for example, you may use all the other variables, or select a subset", so I could use these 3 factors as my subset. I copy Ian's code below, and insert some figures in place of the sample figures he used. The figures I have chosen to use are;1. **CHAS** 0.0 , 0.02. **TAX** 296.0 , 242.03. **MEDEV** 24.0 , 21.6I have taken these from the figures generated earlier in this Jupyter Notebook when I ran 'df.head()' and received 'Out[28]'.<jupyter_code># Create a new neural network.
m = kr.models.Sequential()

# Add a two neurons in a single layer.
m.add(kr.layers.Dense(2, input_dim=1, activation="linear"))

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, activation="linear", kernel_initializer=kr.initializers.Constant(value=1), bias_initializer=kr.initializers.Constant(value=0)))

# Set the weight/bias of the two neurons.
m.layers[0].set_weights([np.matrix([0.0, 296.0]), np.array([0.0, 242.0])])

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create a new neural network.
m = kr.models.Sequential()

# Add a two neurons in a single layer.
m.add(kr.layers.Dense(2, input_dim=1, activation="linear"))

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, activation="linear", kernel_initializer=kr.initializers.Constant(value=1), bias_initializer=kr.initializers.Constant(value=0)))

# Set the weight/bias of the two neurons.
m.layers[0].set_weights([np.matrix([0.0, 296.0]), np.array([0.0, 242.0])])

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.0, 10.0, 1)

# Run each x value through the neural network.
y = m.predict(x)
# Plot the values.
plt.plot(x, y, 'k.')<jupyter_output><empty_output><jupyter_text>We can see above I encounter error messages.<jupyter_code># Create a new neural network.
m = keras.models.Sequential()

# Add a two neurons in a single layer.
m.add(kr.layers.Dense(2, input_dim=1, activation="linear"))

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, activation="linear", kernel_initializer=kr.initializers.Constant(value=1), bias_initializer=kr.initializers.Constant(value=0)))

# Set the weight/bias of the two neurons.
m.layers[0].set_weights([np.matrix([0.0, 296.0]), np.array([0.0, 242.0])])

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")<jupyter_output><empty_output><jupyter_text>I tried changing 'kr' to 'keras' to see if this helps but I still encounter the error message.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import keras as kr
# Create a new neural network.
m = kr.models.Sequential()

# Add a two neurons in a single layer.
m.add(kr.layers.Dense(2, input_dim=1, activation="linear"))

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, activation="linear", kernel_initializer=kr.initializers.Constant(value=1), bias_initializer=kr.initializers.Constant(value=0)))

# Set the weight/bias of the two neurons.
m.layers[0].set_weights([np.matrix([0.0, 296.0]), np.array([0.0, 242.0])])

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.0, 10.0, 1)

# Run each x value through the neural network.
y = m.predict(x)
# Plot the values.
plt.plot(x, y, 'k.')<jupyter_output><empty_output><jupyter_text>I decide to re-run the below code and then try the code again to see if this elimimates the errors, and as we can see it worked;import matplotlib.pyplot as pltimport numpy as npimport keras as kr I'm uncertain what the resulting graph indicates in terms of predicting the median house price based on the other variables in the dataset. I decide to attempt the graph again with a different factor besides **CHAS**, so I instead run the same code using **lstat** (lower status of the population (percent)). 1. ~~CHAS 0.0 , 0.0~~2. TAX 296.0 , 242.03. MEDEV 24.0 , 21.64. LSTAT 4.98, 9.14<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import keras as kr
# Create a new neural network.
m = kr.models.Sequential()

# Add a two neurons in a single layer.
m.add(kr.layers.Dense(2, input_dim=1, activation="linear"))

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, activation="linear", kernel_initializer=kr.initializers.Constant(value=1), bias_initializer=kr.initializers.Constant(value=0)))

# Set the weight/bias of the two neurons.
m.layers[0].set_weights([np.matrix([4.98, 296.0]), np.array([0.0, 242.0])])

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.0, 1)

# Run each x value through the neural network.
y = m.predict(x)
# Plot the values.
plt.plot(x, y, 'k.')
import matplotlib.pyplot as plt
import numpy as np
import keras as kr
# Create a new neural network.
m = kr.models.Sequential()

# Add a two neurons in a single layer.
m.add(kr.layers.Dense(2, input_dim=1, activation="linear"))

# Add a single neuron in a single layer, initialised with weight 1 and bias 0.
m.add(kr.layers.Dense(1, activation="linear", kernel_initializer=kr.initializers.Constant(value=1), bias_initializer=kr.initializers.Constant(value=0)))

# Set the weight/bias of the two neurons.
m.layers[0].set_weights([np.matrix([4.98, 296.0]), np.array([9.14, 242.0])])

# Compile the model.
m.compile(loss="mean_squared_error", optimizer="sgd")
# Create some input values.
x = np.arange(0.0, 10.0, 1)

# Run each x value through the neural network.
y = m.predict(x)
# Plot the values.
plt.plot(x, y, 'k.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09021424707791444, 'fraction_numerical': 0.02384704028597634, 'mean_word_length': 5.154580981328702, 'pattern_counts': {'":': 0, '<': 164, '<?xml version=': 0, '>': 164, 'https://': 33, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '508055', 'license': 'Apache-2.0', 'n_tokens_mistral': 13213, 'n_tokens_neox': 12305, 'n_words': 5491, 'path': "Etain's Jupyter Notebook containing the main body of my work for this assignment - final subission.ipynb", 'repo_name': 'EtainUpton/Assessment2019MachineLearning-Statistics', 'size': '42567'}