<jupyter_start><jupyter_text>DwC Occurrences. Keister Zooplankton Hood Canal 2012-13 dataUniversity of Washington Pelagic Hypoxia Hood Canal project, Zooplankton dataset.2022-1-12 - Refer to https://obis.org/manual/darwincore/occurrence and https://obis.org/manual/darwincore/taxonomy for guidance- I won't use `acceptedname` and `acceptedNameID` are not needed or even recommended**TODO**- Decide how to handle "absent" `occurrenceStatus`. Will need to decide when to assign "absent" occurrence, and how. From Li et al 2019, it's clear that for *Euphausia pacifica*, if there is no occurrence record for a "sample" (a specific net tow), "absence" is implied and can/should be entered. But how about others? Will have to consult with Julie and the Standardizing Bio data group**Questions, issues, comments about life history and sex parsing**- `life_history_stage` is exported in a separate, intermediate csv table together with `occurrenceID` in order to enable the join/merge needed for generating abundance measurements in emof.- `life_stage_mapping` dictionary is now stored in an external JSON file- Now that I've thoroughly compared `lhs_0` and `lhs_1`, it's clear that `lhs_1` can be thrown out. `lhs_0` is all that's needed- `Female` and `Male` are only found for `Adult`, and only found in `lhs_0`- Are `CI` - `CV` (and matching 1-5) the same as `Copepodite` 1 - 5? If they were, their use with *Euphausia Pacifica* (see the next cell) would be wrong. - Life stages `Calyptopis` 1-3, `Furcilia` 1-10 and `Zoea` 1-5 are not found in https://vocab.nerc.ac.uk/collection/S11/current/. That vocab doesn't have any "phases" for `Calyptopis`, `Furcilia` and `Zoea`- These are also not found: `Bract`, `Gonophore`, `Nectophore`, `Polyp`- `Furcilia_1_0_legs` is not clear<jupyter_code>from datetime import datetime
from pathlib import Path
import uuid

import numpy as np
import pandas as pd
data_pth = Path(".")<jupyter_output><empty_output><jupyter_text>Process JSON file containing common mappings and strings<jupyter_code>with open(data_pth / 'common_mappings.json') as f:
    common_mappings = json.load(f)
DatasetCode = common_mappings['datasetcode']
life_stage_mapping = common_mappings['life_stage_mapping']<jupyter_output><empty_output><jupyter_text>Read the data  Pre-process data from csv for Occurrence table  Read the csv file<jupyter_code>sourcecsvdata_pth = data_pth / "sourcedata" / "bcodmo_dataset_682074_data.csv"
usecols = ['sample_code', 'species', 'life_history_stage']

occursource_df = pd.read_csv(
    sourcecsvdata_pth, 
    skiprows=[1], 
    usecols=usecols
)[usecols]
len(occursource_df)
occursource_df.head()<jupyter_output><empty_output><jupyter_text>Merge resolved taxonomy from taxonomy csv<jupyter_code>taxonomy_df = pd.read_csv(
    data_pth / "intermediate_DwC_taxonomy.csv"
)
occursource_df = occursource_df.merge(
    taxonomy_df, 
    how='inner', 
    left_on='species',
    right_on='verbatimIdentification'
)
len(occursource_df)
occursource_df.head()<jupyter_output><empty_output><jupyter_text>Create and populate `occurrence_df` dataframe  Add `occurrenceID`, `basisOfRecord`, `occurrenceStatus`<jupyter_code>occurrence_df = occursource_df.copy()
occurrence_df.insert(1, 'occurrenceID', [uuid.uuid4() for i in range(len(occurrence_df))])
occurrence_df.insert(2, 'basisOfRecord', 'MaterialSample')
occurrence_df.insert(3, 'occurrenceStatus', 'present')

occurrence_df.rename(columns={'sample_code':'eventID'}, inplace=True)
len(occurrence_df)
occurrence_df.head()<jupyter_output><empty_output><jupyter_text>Parse `life_history_stage` and map into `sex` and `lifeStage`<jupyter_code>occurrence_df[['lhs_0', 'lhs_1']] = pd.DataFrame(
    occurrence_df.life_history_stage.str.split(';_').to_list(), 
    index=occurrence_df.index
)
occurrence_df.head(10)
occurrence_df[['lhs_0', 'lhs_1']].value_counts(dropna=False).sort_index(ascending=True)
# occurrence_df[occurrence_df.species == 'EUPHAUSIA_PACIFICA'][['lhs_0', 'lhs_1']].value_counts(dropna=False).sort_index(ascending=True)
occurrence_df.insert(
    4, 'sex', 
    occurrence_df['lhs_0'].apply(
        lambda s: s.lower() if s in ['Female', 'Male'] else 'indeterminate'
    )
)
occurrence_df.insert(
    5, 'lifeStage', 
    occurrence_df['lhs_0'].apply(
        lambda s: life_stage_mapping[s]
    )
)

occurrence_df = (
    occurrence_df
    .sort_values(by=['eventID', 'scientificName', 'lifeStage', 'sex'])
    .reset_index(drop=True)
)
occurrence_df.head(5)<jupyter_output><empty_output><jupyter_text>Export intermediate table for `life_history_stage` matching to csv<jupyter_code>occurrence_df[['occurrenceID', 'life_history_stage']].to_csv(
    data_pth / 'intermediate_DwC_occurrence_life_history_stage.csv', index=False
)<jupyter_output><empty_output><jupyter_text>Export `occurrence_df` to csv  Cleanup<jupyter_code>occurrence_df.drop(columns=['species', 'life_history_stage', 'lhs_0', 'lhs_1'], inplace=True)
occurrence_df.to_csv(data_pth / 'aligned_csvs' / 'DwC_occurrence.csv', index=False)<jupyter_output><empty_output><jupyter_text>Package versions<jupyter_code>print(
    f"{datetime.utcnow()} +00:00\n"
    f"pandas: {pd.__version__}"
)<jupyter_output>2022-01-13 02:50:07.198498 +00:00
pandas: 1.3.5
{'fraction_non_alphanumeric': 0.11991557943207981, 'fraction_numerical': 0.018035303146584806, 'mean_word_length': 6.500719424460431, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '385558', 'license': 'MIT', 'n_tokens_mistral': 1971, 'n_tokens_neox': 1835, 'n_words': 428, 'path': 'Keister-dwcOccurrence.ipynb', 'repo_name': 'nanoos-pnw/obis-keisterhczoop', 'size': '5212'}