<jupyter_start><jupyter_text>It looks like most of the missing values are Location.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns


from scripts import project_functions
df = project_functions.load_and_process(r"C:\Users\Nolan\Desktop\DATA301\course-project-solo_311\data\raw\Meteorite_Landings.xlsx")

df.shape
df.head()
df.nunique()
df.describe()
df[df["year"]>2020]<jupyter_output><empty_output><jupyter_text>Apparently I have one asteroid that fell in **2101**. This is obviously wrong. I will add a filter to my initial load script to scrap anything after 2020.Something like: `df["year"] = df[df["year"]<2021]`<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns


from scripts import project_functions
df = project_functions.load_and_process(r"C:\Users\Nolan\Desktop\DATA301\course-project-solo_311\data\raw\Meteorite_Landings.xlsx")
df.describe()<jupyter_output><empty_output><jupyter_text>There we go, now the max year is 2013.  I think the Geolocation column is redundant since it is useless in the brackets. Reclat and Reclong are easier to use.I will also add this into my project_functions. `df.drop(columns=['GeoLocation'])` None of these columns are directly related except for the latitude and longtitude so I shouldn't expect any correlation<jupyter_code>corr = df.corr()# plot the heatmap
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns, annot=True, cmap=sns.diverging_palette(220, 20, as_cmap=True), mask=mask)<jupyter_output><empty_output><jupyter_text>Interestingly, Longitude is somewhat negatively correlated with latitude. Other things to note, year is correlated to id. This makes sense since the later the meteorite is discovered the higher the id. Mass is not correlated with anything, as expected. Before I get any farther, I will rename reclong -> long and reclat -> lat for ease of use. I will add this to my load script I am curious if we can see evidence of the westward colonization of humans if more meteorites are discovered in the west in later years.<jupyter_code>df_year_long = df[['year','long']].copy().groupby(['year']).mean().reset_index()
sns.set_theme(font_scale=1.5,style='ticks')
sns.regplot(data=df_year_long, x="year", y="long")
plt.xlabel("Year")
plt.ylabel("Longtitude")
plt.title("Average Longtitude of Meteorites found by Year")<jupyter_output><empty_output><jupyter_text>Interesting, not exactly what I expected but I think I can interperet this.It looks like the time period from $1700 \Rightarrow 1960$ shows a clear downward trend and then after $1960$ a lot more asteroids are discovered in Asia.This could mean that as time went on, there was a growing number of scientists discovering meteorites in the west, but after 1960 there was a boom in scientists discovering meteorites in Asia. Next I want to make a histogram of the distribution of asteroid types.<jupyter_code>df["class"].nunique()<jupyter_output><empty_output><jupyter_text>Looks like there are 422 unique classes, I will make a script to only include the top 10 asteroid classes.<jupyter_code>df_classcount = df[["id",'class']].groupby(['class']).count().sort_values(by="id",ascending=False)
df_classcount
def top_classes(df, n=10):
    df_classcount = df[["id",'class']].groupby(['class']).count().sort_values(by="id",ascending=False)
    df2 = df[(df["class"].isin(df_classcount[:n].index.to_list()))]
    df2 = (
        df2.reset_index()
        .drop(columns=['index'])
    )
    return df2
df = top_classes(df,n=10)
sns.histplot(data=df,x="class",color="skyblue")
plt.xlabel("Class")
plt.ylabel("Count")
plt.title("Number of Meteorites found by Class")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10686015831134564, 'fraction_numerical': 0.017941952506596307, 'mean_word_length': 6.12593984962406, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '413235', 'license': 'MIT', 'n_tokens_mistral': 1285, 'n_tokens_neox': 1176, 'n_words': 388, 'path': 'analysis/EDA_milestone2.ipynb', 'repo_name': 'data301-2020-winter1/course-project-solo_311', 'size': '3790'}