<jupyter_start><jupyter_text>To make sense of the data, I'll take a case study.<jupyter_code>satara = monthly_data[monthly_data.APMC == 'Satara']
satara.head()
satara.shape
satara_potato = satara[satara.Commodity == 'Potato']
satara_potato.head()
satara_potato = satara_potato.drop(['APMC', 'Commodity'], 1)
satara_potato.head()
plt.figure(figsize = (20, 5))
plt.plot(satara_potato['min_price'], label = 'min_price')
plt.plot(satara_potato['max_price'], label = 'max_price')
plt.plot(satara_potato['modal_price'], label = 'modal_price')
plt.legend(loc = 'best')
satara_potato_2014 = satara_potato[satara_potato['Year'] == 2014]
satara_potato_2015 = satara_potato[satara_potato['Year'] == 2015]
satara_potato_2016 = satara_potato[satara_potato['Year'] == 2016]<jupyter_output><empty_output><jupyter_text>Now let's try to plot the quantity arrived versus price plots to see how the parameters are correlated.<jupyter_code>plt.scatter(x = satara_potato_2014['arrivals_in_qtl'], y = satara_potato_2014['modal_price'])
plt.show()
plt.scatter(x = satara_potato_2015['arrivals_in_qtl'], y = satara_potato_2015['modal_price'])
plt.show()
plt.scatter(x = satara_potato_2016['arrivals_in_qtl'], y = satara_potato_2016['modal_price'])
plt.show()<jupyter_output><empty_output><jupyter_text>Unfortunately, there are not enough data values to find any conclusion.<jupyter_code>monthly_data['Commodity'] = monthly_data['Commodity'].replace(['AMBAT CHUKA', 'Ambat Chuka'], 'Ambat chuka')
monthly_data.Commodity.value_counts()<jupyter_output><empty_output><jupyter_text>Let me carry out the outlier detection for monthly_data. A univariate outlier might not be a regression outlier, so simply plotting boxplots and removing the outliers might be dangerous. Since using many advanced algorithms might chop out significant information from the data, I will follow a rudimentary approach to find out the outliers for monthly data. Earlier when I plotted the min, max and modal price values for potatoes in Satara, it was clear that for the particular case, the three values must be directly proportional since they have similar patterns, and justifiably so.Lets check that intuition for the combined dataset.<jupyter_code>ax1 = monthly_data.sort_values(by=['Commodity']).plot.scatter(x = 'min_price', y = 'max_price')
ax2 = monthly_data.sort_values(by=['Commodity']).plot.scatter(x = 'min_price', y = 'modal_price')
ax3 = monthly_data.sort_values(by=['Commodity']).plot.scatter(x = 'modal_price', y = 'max_price')<jupyter_output><empty_output><jupyter_text>There are three points that are clear outliers here, two with minimum price > 500000 and one with maximum price > 1400000. We can manually get rid of these values.<jupyter_code>monthly_data_cleaned = monthly_data[(monthly_data.min_price <= 500000) & (monthly_data.max_price <= 1400000)]
monthly_data_cleaned.shape
monthly_data.shape
ax1 = monthly_data_cleaned.sort_values(by=['Commodity']).plot.scatter(x = 'min_price', y = 'max_price')
ax2 = monthly_data_cleaned.sort_values(by=['Commodity']).plot.scatter(x = 'min_price', y = 'modal_price')
ax3 = monthly_data_cleaned.sort_values(by=['Commodity']).plot.scatter(x = 'modal_price', y = 'max_price')<jupyter_output><empty_output><jupyter_text>From the shapes of the cleaned data and the original data, we can tell that the 3 outlying values have been removed. After plotting the three scatterplots, I can still see a point that lies further away from the general pattern, but looking at its scale, it is negligible compared to our previous outliers. Moreover, there could be another justifiable reason for that behaviour which is beyond the scope of my understanding at this point. Further, blindly deleting data points can lead to loss of valuable information, so I'll keep this data point here. As far as msp_mandi dataset is concerned; the minimum support price is simply a price level that is set by government policy. So in a sense, the data is just a government policy jotted down in a tabular form. So, it is absurd that it would have any outliers because there is no scope of experimental error since no experiment is done. We are assuming the MSPs to be ground truths so finding MSPs that have abnormal behaviour seems silly.<jupyter_code>monthly_data_cleaned.to_csv('monthly_data_cleaned.csv')
msp_mandi.to_csv('msp_mandi.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09832299563519412, 'fraction_numerical': 0.0195267631518493, 'mean_word_length': 7.01841620626151, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '363601', 'license': 'MIT', 'n_tokens_mistral': 1481, 'n_tokens_neox': 1359, 'n_words': 431, 'path': 'exploration and outlier detection.ipynb', 'repo_name': 'itsmepiyush2/Agriculture-Commodities-Price-and-Seasons', 'size': '4353'}