<jupyter_start><jupyter_text>Understanding Fishing Vessel's Authorization and ParametersThis notebook is a short exercise for us to understand parameters associated with a fishing vessel, and how is each type of fishing vessel registered. In general, we noticed that there are several fishing licensing authorities, such as CLAV, CCSBT, IOTC, etc.. Also, as indicated in the CLAV sample table below, not all vessels are registered with the same authorization body and each authorization body has a different requirement in terms of required parameters. To obtain a table with mmsi and gear type, we would have to merge the tables using the vessel's unique ID.<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>CLAV Registered Vessels datasetsThis is a consolidated vessel dataset with all fishing authorities (but only for certain types of fish). This dataset does not come with mmsi as the identifier. The most unique ID of the vessel is the "callsign" assigned to each table.<jupyter_code>clav_fleet_registration = pd.read_csv('../data/clav_vessel_registry.csv', error_bad_lines=False)
clav_fleet_registration.head(5)<jupyter_output><empty_output><jupyter_text>Merging Other Authorization Body's Datasets to Obtain MMSI/VesselsTo get the mmsi of each vessel. We used several registration authorities' vessels list to merge and obtain a general list of fishing vessel mmsi numbers. Moreover, vessel tracking sites that provide data on all fishing vessels such as Marinetraffic do not provide the general fishing gear type/ fishing vessel registration type (maybe with paid option).The range of the mmsi data is using 2013 and 2016 as the benchmark, which corresponds to our AIS training data from 2012 to 2016.<jupyter_code>europe_fleet_registration_2016 = pd.read_csv('../data/vesselRegistryListResults_2016.csv', sep=';', error_bad_lines=False)
europe_fleet_registration_2016 = europe_fleet_registration_2016[['Name of vessel','IRCS', 'MMSI', 'Vessel Type', 'Main fishing gear', 'Subsidiary fishing gear 1']]
europe_fleet_registration_2013 = pd.read_csv('../data/vesselRegistryListResults_2013.csv', sep=';', error_bad_lines=False)
europe_fleet_registration_2013 = europe_fleet_registration_2013[['Name of vessel','IRCS', 'MMSI', 'Vessel Type', 'Main fishing gear', 'Subsidiary fishing gear 1']]
europe_fleet_registration = pd.concat([europe_fleet_registration_2013, europe_fleet_registration_2016])
europe_fleet_registration = europe_fleet_registration.dropna()
europe_fleet_registration.head(5)

CCSBT_fleet_registration = pd.read_csv('../data/authorised_vessel_record.csv', error_bad_lines=False)
CCSBT_fleet_registration = CCSBT_fleet_registration[['Callsign', 'Vessel Type', 'Gear Type']]
CCSBT_fleet_registration.head(5)
IOTC_fleet_registration = pd.read_csv('../data/IOTC_authorized_vessels.csv', error_bad_lines=False)
IOTC_fleet_registration = IOTC_fleet_registration[['IRCS', 'Type', 'Gear']]
IOTC_fleet_registration.head(5)
fleet_all = pd.merge(europe_fleet_registration, CCSBT_fleet_registration, left_on='IRCS', right_on='Callsign', how='outer')
fleet_all = pd.merge(fleet_all, IOTC_fleet_registration, on='IRCS', how='outer')
fleet_all.drop_duplicates().dropna().head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08215818516247701, 'fraction_numerical': 0.019313304721030045, 'mean_word_length': 7.6094986807387865, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '495557', 'license': 'MIT', 'n_tokens_mistral': 1088, 'n_tokens_neox': 967, 'n_words': 310, 'path': 'eda/eda_mmsi.ipynb', 'repo_name': 'WilliamZipanHe/CMPT733-Fish-Activity-Detection', 'size': '3262'}