<jupyter_start><jupyter_text>Data for the year 2014 has many details which is not present in the previous data files<jupyter_code>df['state_ut'] = df.state_ut.str.replace('\s+&\s+', '&')
df['district_ut'] = df.district.apply(lambda x: x.replace('\s+&\s+', '&'))
list(df14.columns)<jupyter_output><empty_output><jupyter_text>As said earlier, some columns may seem similar that they can be merged by preprocessing. But the data file for 2014 has columns which is a aggregate of two or more other columns. For example, 'counterfeit' column is an aggregate value of other columns related to counterfeiting added. We will now group them all by year and look at the trend<jupyter_code>no_total = df.loc[df.district != 'Total', :]
yr_totals = no_total.groupby('year')
yr_agg = yr_totals.sum().reset_index()<jupyter_output><empty_output><jupyter_text>Since most columns are only present in the 2014 data file, we will encounter NAs in all but one row for every column now. So, we drop those columns. Also, we will perform imputation.<jupyter_code>for cl in yr_agg.columns:
    if cl == 'year':
        continue
    if yr_agg[cl].isnull().sum() > 10:
        yr_agg = yr_agg.drop(cl, axis=1)
    else:
        mean_cl = np.mean(yr_agg[cl])
        yr_agg[cl] = yr_agg[cl].fillna(mean_cl)
        yr_agg[cl] = yr_agg[cl].apply(lambda x: x/10000)
yr_agg.describe()
cols = list(yr_agg.columns)
cols.remove('year')
cols.remove('total_ipc_crimes')
cols.remove('other_ipc_crimes')
fig = mplt.figure()
fig.set_size_inches(15, 10)
ax = mplt.subplot(111)
# ax.set_xlim([2001, 2014])
ax.set_title("Crime committed in India (in ten thousands)")
ax.set_xlabel("Year")
ax.set_ylabel("Number of cases filed")
for col in cols:
    ax.plot(yr_agg.year, yr_agg[col], label=col.replace('_', ' '))
ax.legend(loc=5, bbox_to_anchor=(1.5, .5))
corr = yr_agg[cols].corr()
mask = np.array(corr)
mask[np.tril_indices_from(mask)] = False
fig, ax = mplt.subplots()
fig.set_size_inches(15, 15)
sn.heatmap(corr, mask=mask, ax=ax, square=True)
group_by = ["state_ut"]
columns_of_interest = ["total_ipc_crimes"]
state_grp = no_total[columns_of_interest + group_by].groupby(["state_ut"])
state_agg = state_grp.sum().reset_index().sort_values(by='total_ipc_crimes')
state_agg['total_ipc_crimes'] = state_agg.total_ipc_crimes
fig = mplt.figure()
fig.set_size_inches(20, 5)
ax = fig.add_subplot(111)
ax.set_xticklabels(state_agg.state_ut, rotation=90)
b = sn.barplot(x='state_ut', y='total_ipc_crimes', data=state_agg, ax=ax)
columns_of_interest = [
    'state_ut',
    'district',
    'year',
    'murder',
    'rape',
    'kidnapping_abduction_total',
    'dacoity',
    'dacoity_with_murder',
    'robbery',
    'theft',
    'unlawful_assembly',
    'riots',
    'riots_communal',
    'riots_industrial',
    'riots_political',
    'riots_caste_conflict',
    'riots_students',
    'counterfeiting',
    'dowry_deaths',
    'sexual_harassment',
    'importation_of_girls_from_foriegn_country',
    'humantrafficking',
    'unnatural_offence'
]
df.to_csv(
    "../data/section_01.csv",
    columns=columns_of_interest,
    index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14024975984630164, 'fraction_numerical': 0.01761127121357669, 'mean_word_length': 5.0660194174757285, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '576719', 'license': 'MIT', 'n_tokens_mistral': 1244, 'n_tokens_neox': 1168, 'n_words': 247, 'path': 'notebooks/district_wise_crimes_committed.ipynb', 'repo_name': 'Ajju7669/crime_data_analysis', 'size': '3123'}