<jupyter_start><jupyter_text>Analysis of occurrences for US The dataset is composed in multiple files:- `occurrences_fr_train.csv`- `occurrences_fr_test.csv`- `occurrences_us_train.csv`- `occurrences_us_test.csv`- `species_metadata.csv`The datasets columns include :- `id`:  The GLC20 reference identifier for the occurrence.- `lat`:	Decimal latitude in the WGS84 coordinate system.- `lon`:	Decimal longitude in the WGS84 coordinate system.- `species_id`:	The GLC20 reference identifier for the species.The metadata columns include :- `species_id`:	The GLC20 reference identifier for the species.- `GBIF_species_id`:	The GBIF reference identifier for the species.- `GBIF_species_name`:	The GBIF reference name for the species.  Resume: statistical characteristics of the dataset Dataset size and number of speciesThe dataset is composed of 1m76k occurrences.  There are 25251 unique species in the dataset. Missing valuesThere are no missing values (non-attributes values) in the dataset. All fields are always present. Duplicates 1. Duplicated rows ("doublons"):There are 20k duplicates in the dataset (i.e., records for which another record has the same (lat,lon,species_id) triplet). This equals to 1.9% of the dataset.  Those are occurrences of the same specie identification, at exactly the same geolocation. Those samples probably come together, I mean that:- The person who identified the a specie with the PlantNet app, either identified the same specie by photographing a neighbouring plant or animal, at the same geolocation.- Or, she photographed to same plant or animal several time.Among these, not counting the first occurrence, there are 13k duplicates.This equals to 1.2% of the dataset. Those occurrences don't add any data, they just create an undesired effect of oversampling. So they will be removed. 2. Overlapping geolocations:Duplicates apart, there are 10k samples with overlapping geolocations. This equals to 0.9% of the dataset.  Those are occurrences of different specie identification, at exactly the same location. Those overlapping samples probably come together, for the same reason as above: same person at the same spot.  Among these, not counting the first occurrence, there are 9k overlapping geolocations.This equals to 0.8% of the dataset. Species 1. Species frequency as a function of the species frequency rank- The curve has an inverse shape visible when showing all species. This is characteristic of a Zipf law.   2. Number of species filtered in as a function of the lower frequency threshold- The curve as expected shows an inverse relation. The point of inflexion of the curve is around a threshold of 40. The table below show a few possible threshold values with the number of species frequent enough. An acceptable value could be 5 or 10.<jupyter_code># Read the occurrences_us_train.csv data
PROJECT_ROOT = os.path.expanduser('~/projects/geolifeclef20/')
DATA_ROOT = os.path.join(PROJECT_ROOT, 'data/')
PATH_OCCURRENCES = os.path.join(DATA_ROOT, 'occurrences/')
PATH_RASTERS = os.path.join(DATA_ROOT, 'rasters/')
PATH_PATCHES = os.path.join(DATA_ROOT, 'patches/')
PATH_FR_TRAIN = os.path.join(PATH_OCCURRENCES, 'occurrences_fr_train.csv')
PATH_US_TRAIN = os.path.join(PATH_OCCURRENCES, 'occurrences_us_train.csv')
df = pd.read_csv(PATH_US_TRAIN,
    sep=';', header=0, index_col='id', low_memory=True)    
df_metadata = pd.read_csv(os.path.join(PATH_OCCURRENCES,'species_metadata.csv'),
    sep=';', header=0, index_col='species_id', low_memory=True)

df = df.merge(
    df_metadata, how='left', left_on='species_id', right_index=True,
    copy=False).drop('GBIF_species_id', axis='columns')

# target series is the species ids
target_col = df['species_id']
print('Dataset sample:')
idisplay(df.sample(20))
dataset_size = len(df.index)
num_species = len(target_col.unique())
print('Dataset size: {}'.format(dataset_size))
print('Number of species: {}'.format(num_species))

print('Number of N/A rows: {}'.format(df.isna().any(axis=1).sum()))<jupyter_output><empty_output><jupyter_text>Duplicates<jupyter_code>df_noname = df.drop('GBIF_species_name', axis=1)
dupl_mask = df_noname.duplicated(keep=False)
dupl_mask_keepfirst = df_noname.duplicated(keep='first')

df_nodupl = df_noname[dupl_mask_keepfirst]

dupl_mask_geoloc = df_nodupl.duplicated(subset=['lat','lon'], keep=False)
dupl_mask_geoloc_keepfirst = df_nodupl.duplicated(subset=['lat','lon'], keep='first')
n_duplicates = dupl_mask.sum()
n_duplicates_keepfirst = dupl_mask_keepfirst.sum()

n_with_overlapping_geoloc = dupl_mask_geoloc.sum()
n_with_overlapping_geoloc_keepfirst = dupl_mask_geoloc_keepfirst.sum()

print('There are {} duplicates in the dataset (i.e., records for which another record has the same (lat,lon,species_id) triplet). '
      'This equals to {:.1%} of the dataset.\n'\
    .format(n_duplicates, n_duplicates / dataset_size))

print('Among these, not counting the first occurrence, there are {} duplicates.'
      'This equals to {:.1%} of the dataset.\n'\
    .format(n_duplicates_keepfirst, n_duplicates_keepfirst / dataset_size))


print('Duplicates apart, there are {} samples with overlapping geolocations.\n'\
      'This equals to {:.1%} of the dataset.\n'\
    .format(n_with_overlapping_geoloc,
            n_with_overlapping_geoloc / dataset_size))

print('Among these, not counting the first occurrence, there are {} overlapping geolocations.'
      'This equals to {:.1%} of the dataset.\n'\
    .format(n_with_overlapping_geoloc_keepfirst,
            n_with_overlapping_geoloc_keepfirst / dataset_size))<jupyter_output><empty_output><jupyter_text>Species frequency as a function of the species frequency rank<jupyter_code>species, count_species = tuple(map(lambda x: pd.Series(x),
                               np.unique(target_col, return_counts=True)))
count_species.index = species
# freqs = pd.DataFrame({'species': species, 'counts': count_species})
# freqs = freqs.sort_values('counts', ascending=False)
# freqs
count_species = count_species.sort_values(ascending=False)
count_species.name = 'count'
# Plot of species frequency (count) as a function of the rank of the specie.
# ax2 for cumulated proportion of occurrences
# Help:
# - https://matplotlib.org/gallery/api/two_scales.html
# - https://matplotlib.org/3.1.0/gallery/subplots_axes_and_figures/secondary_axis.html

max_rank = 1000
counts = count_species.iloc[:max_rank]
ranks = range(1, max_rank+1)
# plot frequency
fig, ax1 = plt.subplots(figsize=(12,8)) 
color1 = 'C0'
ax1.plot(ranks, counts, color=color1)
ax1.set_ylim(0)
ax1.set_xlabel('Species rank')
ax1.set_ylabel('Frequency', color=color1)
ax1.tick_params(axis='y', labelcolor=color1)

ax1.fill_between(ranks, counts, alpha=0.2)
# add secondary axis for percentage of species ≤ rank
secax1 = ax1.secondary_xaxis('top',
                           functions=(lambda rank: rank / num_species,
                                      lambda per: per * num_species)
                          )
secax1.set_xlabel('Percentage of species ≤ rank')

cumulated_prop = np.cumsum(counts) / np.sum(count_species)
# plot cumulated proportion of occurrences on different y axis
ax2 = ax1.twinx()
color2 = 'C1'
ax2.plot(ranks, cumulated_prop, color=color2)
ax2.set_ylabel('Cumulated proportion of dataset', color=color2)
ax2.tick_params(axis='y', labelcolor=color2)

# plt.savefig('figs/us_frequency_rank{}.png'.format(max_rank), dpi=200, bbox_inches='tight')
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Number of species filtered in as a function of the lower frequency threshold<jupyter_code>thresholds = range(2, 200)
# numer of species filtered in
num_kept = pd.Series(((count_species >= t).sum() for t in thresholds), index=thresholds)
# the number of occurrences it represent
num_occs_kept = pd.Series((count_species[count_species >= t].sum() for t in thresholds), index=thresholds)
idisplay(num_kept[[2,3,4,5,6,10,20]])
idisplay(num_occs_kept[[2,3,4,5,6,10,20]])
fig, ax1 = plt.subplots(figsize=(12,8))
p1, = ax1.plot(thresholds, num_kept)

ax1.set_xlabel('Lower frequency threshold')
ax1.set_ylabel('Number of species filtered in', color=p1.get_color())
ax1.tick_params(axis='y', labelcolor=p1.get_color())

ax2 = ax1.twinx()

p2, = ax2.plot(thresholds, num_occs_kept / np.sum(count_species), color='C1')

ax2.set_ylabel('Proportion of dataset', color=p2.get_color())
ax2.tick_params(axis='y', labelcolor=p2.get_color())

# secax1 = ax1.secondary_yaxis('right',
#                            functions=(lambda num: num / num_species,
#                                       lambda per: per * num_species)
#                           )
# secax1.set_ylabel('Percentage of species')
plt.tight_layout()
# plt.savefig('figs/us_species_threshold.png', dpi=200, bbox_inches='tight')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10825854822219698, 'fraction_numerical': 0.016585255026695445, 'mean_word_length': 5.092733564013841, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '458566', 'license': 'MIT', 'n_tokens_mistral': 3006, 'n_tokens_neox': 2801, 'n_words': 803, 'path': 'jupyter/occurrences_us.ipynb', 'repo_name': 'bourcierj/glc-20', 'size': '8803'}