<jupyter_start><jupyter_text>Preliminary CleaningThis data is very messy, and needs some work right off the bat.  For starters, many rows have 0 or null values for key columns we want to draw conclusions on.  Some of these make sense, for instance empty lots will have a property area and assessed improvement value of 0.  Others, like a lot area of 0 seem to indicate some error in reporting.Next, there are a number of properties where PropertyArea > LotArea * NumberofStories, which should be impossible.Finally, a couple *huge* lots in the millions of square feet are excluded from analysis since they dwarf all others by orders of magnitude<jupyter_code>ass_data = ass_data.query('PropertyArea > 0 and LotArea > 0 and NumberofStories > 0')

# about 4 thousand properties are listed as having more square feet than is possible, not sure why
ass_data = ass_data.query('PropertyArea <= LotArea * NumberofStories')

ass_data = ass_data.query('LotArea < 300000')<jupyter_output><empty_output><jupyter_text>Let's also fix the import issue with the type:<jupyter_code>print(ass_data.keys()[26])
ass_data["ExemptionCode"].value_counts()<jupyter_output><empty_output><jupyter_text>It looks like it's trying to be an integer, but there are some strange inputs, primarily `no` and `st`<jupyter_code>ass_data.loc[ass_data["ExemptionCode"].isin(["no", "st"])]<jupyter_output><empty_output><jupyter_text>Hm, it looks just like a writing error. As we don't know what they should be, and taking into account that they make up a grand total of `8` data points, we can safely remove them.<jupyter_code>ass_data = ass_data.query('ExemptionCode != "no" and ExemptionCode != "st"')
ass_data["ExemptionCode"].value_counts()
ass_data["ExemptionCode"] = ass_data["ExemptionCode"]\
    .apply(lambda x: (str(x) != "nan") and int(str(x)[:str(x).find(".")]) or x)
ass_data["ExemptionCode"].describe()
ass_data.dtypes<jupyter_output><empty_output><jupyter_text>The following pulled from [the dataset's source](https://sfassessor.org/news-information/property-data-0)|Key                                     |Desc||:---------------------------------------|:---||Property Location                       |Situs/Location, as well as room/unit number||Neighborhood Code                       |ASR neighborhood code based on MLS districts||Block and Lot Number                    |Block and Lot Number (Full APN)||Volume Number                           |Volume number||Property Class Code                     |Property type||Year Property Built                     |Year improvement was built (can be blend of original and newer constructon)||Number of Bathrooms                     |Number of bathrooms (BA with no shower or tub is 1/2 bathroom)||Number of Bedrooms                      |Number of bedrooms (bedrooms have a closet)||Number of Rooms                         |Number of rooms, excluding bathrooms, halls, closets, etc.||Number of Stories                       |Number of stories||Number of Units                         |Number of units||Zoning Code                             |Zone code||Construction Type                       |Generally type of construction||Lot Depth                               |Depth of lot in linear feet||Lot Frontage                            |Linear footage of front facing side of lot (front foot)||Property Area in Square Feet            |Same as lot area||Basement Area                           |Square footage of basement||Lot Area                                |Square footage of lot||Lot Code                                |lot shapes, could be Rectangle, Square or Other||Prior Sales Date (YYMMDD)               |prior sale date||Tax Rate Area Code                      |Tax rate dependent on location within the City||Percent of Ownership                    |Percent of ownership||Closed Roll Exemption Type Code         |Exemption Code (see below for descriptions)||Closed Roll Status Code                 |i.e. Taxable, Non-Taxable, SBE, etc.||Closed Roll Misc. Exemption Value       |ExemptiSameons such as welfare||Closed Roll Homeowner Exemption Value   |Homeowner's exemption||Closed Roll Year                        |Roll YeSameSamear||Current Sales Date (YYMMDD)             |current sale date||Closed Roll Assessed Fixtures Value     |Assessed value of fixtures||Closed Roll Assessed Improvement Value  |Assessed value of improvements||Closed Roll Assessed Land Value         |Assessed value of land||Closed Roll Assessed Personal Prop Value|Assessed value of personal property|<jupyter_code>ass_data["AssessedTotalValue"] = ass_data["AssessedFixturesValue"] + ass_data["AssessedImprovementValue"]\
    + ass_data["AssessedLandValue"] + ass_data["AssessedPersonalPropertyValue"]
ass_data[["AssessedFixturesValue", "AssessedImprovementValue", "AssessedLandValue", "AssessedPersonalPropertyValue",
          "AssessedTotalValue"]].head()<jupyter_output><empty_output><jupyter_text>Data ExplorationAs we would expect, a definite positive correlation exists between property area and property value.  Some anomalies can already be picked out by eye from this graph<jupyter_code>ass_data.plot.scatter(x='PropertyArea', y='AssessedImprovementValue')
ass_data.plot.scatter(x='LotArea', y='PropertyArea')
ass_data.plot.scatter(x='AssessedLandValue', y='AssessedImprovementValue')
ass_data.plot.scatter(x='LotArea', y='AssessedLandValue')<jupyter_output><empty_output><jupyter_text>Initial attempts at anomaly detection seem to identify "mansion" and "non-mansion" categories, which isn't too helpfull...Feature engineering with value / sqare foot might help us out here<jupyter_code>model = EllipticEnvelope()
labels = model.fit_predict(ass_data[['PropertyArea', 'AssessedTotalValue']])
plt.figure(figsize=(10, 5))
scatter = plt.scatter(ass_data['PropertyArea'], ass_data['AssessedTotalValue'], c=labels, s=10)
plt.title('Anomaly Detection')
plt.xlabel('Property Area (sq ft)')
plt.ylabel('Assessed Total Value (USD)')

plt.xlim([0, 10000])
plt.ylim([0, 10000000])

plt.colorbar(scatter)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09755694948827996, 'fraction_numerical': 0.006437768240343348, 'mean_word_length': 3.555639097744361, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '138584', 'license': 'MIT', 'n_tokens_mistral': 1700, 'n_tokens_neox': 1585, 'n_words': 556, 'path': 'notebooks/Anomaly.ipynb', 'repo_name': 'EmiliaPaz/EECS731_FinalProject', 'size': '6058'}