<jupyter_start><jupyter_text>Split EOL user crops dataset into train and test---*Last Updated 29 March 2020*  Instead of creating image annotations from scratch, EOL user-generated cropping coordinates are used to create training and testing data to teach object detection models and evaluate model accuracy for YOLO via darkflow, SSD and Faster-RCNN object detection models, respectively. Following the [Pareto principle](https://en.wikipedia.org/wiki/Pareto_principle), 80% of the original EOL crops dataset are randomly selected to be training data and the remaining 20% will be used to test model accuracy. Resulting train and test datasets are exported for further pre-processing in [lepidoptera_preprocessing.ipynb](https://github.com/aubricot/computer_vision_with_eol_images/blob/master/object_detection_for_image_cropping/lepidoptera/lepidoptera_preprocessing.ipynb), before they are ready to use with the object detection models.<jupyter_code># Mount google drive to import/export files
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
import pandas as pd
import numpy as np

# Read in EOL user-generated cropping data
crops = pd.read_csv('drive/My Drive/fall19_smithsonian_informatics/train/lepidoptera_crops.tsv', sep="\t", header=0)
print(crops.head())

# Randomly select 80% of data to use for training
# set seed with random_state=2 for reproducible results
idx = crops.sample(frac = 0.8, random_state=2).index
train = crops.iloc[idx]
print(train.head())

# Select the remaining 20% of data for testing using the inverse index from above
test = crops.iloc[crops.index.difference(idx)]
print(test.head())

# Write test and train to tsvs 
train.to_csv('drive/My Drive/fall19_smithsonian_informatics/train/lepidoptera_crops_train.tsv', sep='\t', header=True, index=False)
test.to_csv('drive/My Drive/fall19_smithsonian_informatics/train/lepidoptera_crops_test.tsv', sep='\t', header=True, index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.085612968591692, 'fraction_numerical': 0.012664640324214792, 'mean_word_length': 6.261029411764706, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '193662', 'license': 'MIT', 'n_tokens_mistral': 615, 'n_tokens_neox': 571, 'n_words': 192, 'path': 'object_detection_for_image_cropping/lepidoptera/archive/lepidoptera_split_train_test.ipynb', 'repo_name': 'aubricot/computer_vision_with_eol_images', 'size': '1974'}