<jupyter_start><jupyter_text><jupyter_code>pip install opendatasets --quiet

%matplotlib inline
import opendatasets as od
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
donwload_url = 'https://www.kaggle.com/midouazerty/work-for-parmavir'
file = od.download(donwload_url)
df = pd.read_csv('/content/work-for-parmavir/cubes.csv')
df.head()<jupyter_output>/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py:2718: DtypeWarning: Columns (6,7) have mixed types.Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>Exploratory Analysis  - Let's start with the Exploratory Analysis, because we seem to have a very large number of missing values.  - It should also help us to identify why they are missing and how does that impact our model, if I can try to come up with one that is .  - We seem to have 400287 rows × 8 columns.  - Out of the 8 Coulmns, which we deem to be important, let's try to arrange them in the order of missing values.<jupyter_code>df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Missing Values- There seems to be quite a lot of missing values in all the layers that we need to try and make sense of the data, like try to predict the yield in varying temperatures and also how the humidity affects the crop yield.- As we can see above, there approximamtely same number of missing values in temperature layers and the humidity layer.- Maybe we can filter out the missing values, or before that let's try and see if there is any unique reason, like, rows with the same number of ID's which might affect the missing values.- So let's check the unique values in Cube ID, Timestamp and the Unamed column.<jupyter_code>df.nunique()
df.groupby('Cube ID').nunique()
time_stamp = np.sort(df['Timestamp'].unique())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07676019057702488, 'fraction_numerical': 0.008470089994706194, 'mean_word_length': 5.0576923076923075, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '661256', 'license': 'MIT', 'n_tokens_mistral': 581, 'n_tokens_neox': 554, 'n_words': 240, 'path': 'crop_yield.ipynb', 'repo_name': 'kee1bo/vertical_farming', 'size': '1889'}