<jupyter_start><jupyter_text>Load Dataset from Registry<jupyter_code>from azureml.core import Workspace
ws = Workspace.from_config()<jupyter_output><empty_output><jupyter_text>We will start by loading version 2 of our dataset, where we did not convert everything into numerical values yet.<jupyter_code>from azureml.core import Dataset
dataset_name = 'Melbourne Housing Dataset'

# Get a dataset by name
melb_ds = Dataset.get_by_name(workspace=ws, name=dataset_name, version=2)

# Load a TabularDataset into pandas DataFrame
df = melb_ds.to_pandas_dataframe()
df<jupyter_output><empty_output><jupyter_text>Results so far First, let us remember our preliminary results from Chapter 5 (Feature Importance for target Price and Price_log):![alt text](feature_chap5.png "Feature Importance for Price and log(Price)") Let us talk about these results a bit further than we did in Chapter 5. For that, let us group these feature into what they actually convey.**Housing Properties**- Type: seems to have a very high indication for price. It is also helpful, that there are only 3 possible "settings" for this feature.- Parking: Having 5 or 7 parking spaces is probably not a big difference anymore. We could have a look at making this more discrete by dividing it into maybe three groups 0, 1 and 2+ parking spaces for example.- YearBuilt: From our understanding, the age of a house should have an impact on the price, yet it seems very small. We could transform the data into a discrete age 0-10, 10-20, 20-30 etc.- BuildingArea: One would think, this should have a much higher influence. Therefore, let us divide this as well into groups.- Landsize: Same argument as BuildingArea.- Bathrooms, Bedrooms, Rooms: We would have to drill down deeper into these. As seen before, there seem to be some questionable combinations. Still this has some impact on the price (as we would expect).**Housing Location**- Suburb: As predicted, suburb has too many possible values (around 500) to be of much use. Therefore, we will ignore this as well for now.- SuburbPropCount: Once again, this value is too detailed and therefore does not seems to have nearly no predictive value. Once again, we could think of building a discrete feature breaking it in 3-5 groups.- CouncilArea: has some impact, but we see it dwindleds when looking at the logarithmic price.- Distance (from city center): seems to have a high impact and could be improved by discretization.- Region: draws a clearer picture for the price compared to CouncilArea or suburb.- Longitude&Lattitude: Seems to have a high impact. They seem to convey something better than the CouncilArea.**Others**- Method: The method how the house or apartment is bought seems to have not much importance, as we would presume in most cases. Therefore, we can remove this certainly.- Date: (not included in the Feature Importance graph) The sell date over a couple of years. We might find a small increase due to constantly increasing housing prices, if we only look at the year.   Visualisations Let us look at histograms of all the features we might want to transform.  Parking<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="Parking")
fig.show()<jupyter_output><empty_output><jupyter_text>Remember, that we replaced the missing values with the median = 2, therefore be advised, that the most properties probably have 1 parking spot. As discussed, we could bin these into 0, 1, 2++ groups.  YearBuilt<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="YearBuilt")
fig.show()<jupyter_output><empty_output><jupyter_text>We can see our missing values create a spike in the middle of the dataset. Equal binning of 10 year spans might be a good starting point. Let us have a look at that:<jupyter_code>import plotly.express as px
import numpy as np

counts, bins = np.histogram(df["YearBuilt"], bins=range(1850, 2030, 10))
bins = 0.5 * (bins[:-1] + bins[1:])

fig = px.bar(x=bins, y=counts, labels={'x':'Distance', 'y':'count'})
fig.show()<jupyter_output><empty_output><jupyter_text>BuildingArea<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="BuildingArea")
fig.show()<jupyter_output><empty_output><jupyter_text>Also here, we can see our missing value replacement for the building area. Using equidistant bins might be of interest here as well. Let us look at that:<jupyter_code>import plotly.express as px
import numpy as np

counts, bins = np.histogram(df["BuildingArea"], bins=range(0, 350, 25))
bins = 0.5 * (bins[:-1] + bins[1:])

fig = px.bar(x=bins, y=counts, labels={'x':'Distance', 'y':'count'})
fig.show()<jupyter_output><empty_output><jupyter_text>Distance<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="Distance")
fig.show()<jupyter_output><empty_output><jupyter_text>An equal distance binning with steps of 5 miles could be an option here. Let's have a look at that.<jupyter_code>import plotly.express as px
import numpy as np

counts, bins = np.histogram(df["Distance"], bins=range(0, 60, 5))
bins = 0.5 * (bins[:-1] + bins[1:])

fig = px.bar(x=bins, y=counts, labels={'x':'Distance', 'y':'count'})
fig.show()<jupyter_output><empty_output><jupyter_text>CouncilArea<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="CouncilArea")
fig.show()<jupyter_output><empty_output><jupyter_text>An option here could be to fill out the missing CouncilAreas. We have street addresses for all of them, so either pulling in external data or checking the suburbs to CouncilArea matching should give us the missing data.  Region<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="Region",)
fig.show()<jupyter_output><empty_output><jupyter_text>Looking at this, we could combine the houses outside of the metropolitan area into one group (Victoria) or we could even create only two groups (Metropolitan, Victoria).  Date<jupyter_code>import plotly.express as px
df["Year Offered/Sold"] = df['Date'].dt.year.astype(int)

fig = px.histogram(df, x="Year Offered/Sold")
fig.show()<jupyter_output><empty_output><jupyter_text>As we presumed, the data is taken during two years (2016 and 2017).  SuburbPropCount<jupyter_code>import plotly.express as px
fig = px.histogram(df, x="SuburbPropCount")
fig.show()<jupyter_output><empty_output><jupyter_text>We can see a lot of different amounts here. Once again, binning might be helpful. For example doing bins with a size of 5000:<jupyter_code>import plotly.express as px
import numpy as np

counts, bins = np.histogram(df["SuburbPropCount"], bins=range(0, 25000, 5000))
bins = 0.5 * (bins[:-1] + bins[1:])

fig = px.bar(x=bins, y=counts, labels={'x':'SuburbPropCount', 'y':'count'})
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09361447584866159, 'fraction_numerical': 0.013907581875280395, 'mean_word_length': 5.6679960119641075, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '222852', 'license': 'MIT', 'n_tokens_mistral': 2116, 'n_tokens_neox': 2004, 'n_words': 829, 'path': 'chapter06/02_fe_melbhousing.ipynb', 'repo_name': 'PacktPublishing/Mastering-Azure-Machine-Learning-Second-Edition', 'size': '6687'}