<jupyter_start><jupyter_text>EDA<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import os
import PIL
import numpy as np<jupyter_output><empty_output><jupyter_text>Check out the csv with data on the images<jupyter_code>df = pd.read_csv('./data/train_ship_segmentations_v2.csv')
df.info()
(81723 - 39167) / 192556<jupyter_output><empty_output><jupyter_text>Roughly 22% of the images have ships in them based on how many rows in the CSV contain encoded pixels<jupyter_code>81723 - 39167
df.head()<jupyter_output><empty_output><jupyter_text>It looks like there can be multiple entries for the `ImageId` column, which is how multiple ships are encoded in a single image.<jupyter_code>df.shape[0] - df.groupby('ImageId').sum().shape[0]<jupyter_output><empty_output><jupyter_text>Of the 81723 rows with ships present, 39167 of those rows contain multiple ships.<jupyter_code>ships = df.dropna()
ships<jupyter_output><empty_output><jupyter_text>What is the distribution of ships per image?<jupyter_code>ships['ImageId'].value_counts() 
(ships['ImageId'].value_counts() == 15).value_counts()[1]
mask.value_counts()[1]
ship_counts = {}
for i in range(1, 16):
    ship_counts[i] = (ships['ImageId'].value_counts() == i).value_counts()[1]
ship_counts = {'index': [i for i in range(1, 16)], 'count': [(ships['ImageId'].value_counts() == i).value_counts()[1] for i in range(1, 16)]}
ship_counts
counts = pd.DataFrame.from_dict(ship_counts)
counts
counts.set_index('index', inplace=True)
counts.plot(kind='bar', title='Distribution of ship counts');
counts.plot(kind='bar', title='Log distribution of ship counts', logy=True);<jupyter_output><empty_output><jupyter_text>Let's visualize some images<jupyter_code>train = os.listdir('./data/train_v2/')
len(train)
train[:4]
PIL.Image.open(f'./data/train_v2/{train[0]}').resize((200, 200))<jupyter_output><empty_output><jupyter_text>Image to numpy array<jupyter_code>img = PIL.Image.open(f'./data/train_v2/{train[0]}').resize((200, 200))
img_array = np.array(img)
plt.imshow(img_array)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14777131782945738, 'fraction_numerical': 0.035852713178294575, 'mean_word_length': 7.900862068965517, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '252826', 'license': 'CC0-1.0', 'n_tokens_mistral': 811, 'n_tokens_neox': 755, 'n_words': 157, 'path': 'eda.ipynb', 'repo_name': 'transcriptive/shipdetection', 'size': '2064'}