<jupyter_start><jupyter_text>Capstone Project:** <NAME>, <NAME>, <NAME>** Overview:Today, Humanity is facing unprecedented challenge of global warming because of man made greenhouse gas (CHG) emissions. Both developed and developing nations around the world have come together to fight this existential threat to Humanity. The Climate change is a major target for the Biden administration (and other countries).Many startup companies (Pachama, Indigo, ClimateCorp) backed by major investors are starting to work in this area. California has started its cap-and-trade program since 2013. This project aims at using high resolution remote sensing datasets to create carbon stock maps for a wide range of forests in the United States.The adoption of renewable sources of energy (zero carbon), carbon capture and storage (CCS) and Carbon offsetting are some of the options to address climate change. The carbon credits, aka carbon offset schemes, is a voluntary form of carbon pricing, allows individuals and companies to invest in environmental projects worldwide to compensate for the greenhouse gas(GHG) emissions they produce by investing in renewable energy sources or reducing (such as reforestation) GHG emissions. The Carbon credits markets aim at reducing the emission of greenhouse gases, and allowemitters to offset their emission by purchasing carbon credits.Forests are a significant player in absorbing carbon dioxide (photosynthesis) and storing CO2 in their wood tissues (biomass). Accurate quantification of how much carbon forest stores is essential to the success of the carbon market. Problem Definition:Our main goal is to develop a predictive model to estimate biomass for tree species in a wide variety of forest types to serve as the foundation for awarding carbon credit to individual entities (states, corporations, organizations, individual global citizens, etc.). The purpose is to provide a scientific, validated basis for the credit determinations that can be numerically assigned from the biomass estimates to carbon credits in a dynamical carbon market.   Data:__1. What is the format of your data?__     The the source dataset includes  __csv files__ for 2018 SCBI Forest-GEO ground-level data and 2019 National Ecological Observatory Network’s(NEON), Airborne Observatory Program (AOP) Lidar Data, from which we create an aggregate dataset.__SCBI Forest-Geo 2018 Census Data__The Smithsonian Conservation Biology Institute (SCBI) large forest dynamics plot is located at the Smithsonian’s National Zoo and Conservation Biology Institute in Front Royal, VA, adjacent to the northern end of Shenandoah National Park. The plot is located at the intersection of three of the major physiographic provinces of the eastern US: the Blue Ridge, Ridge and Valley, and Piedmont provinces. The forest type is typical mature secondary eastern mixed deciduous forest, with a canopy dominated by tulip poplar (Liriodendron tulipifera), oaks (Quercus spp.), and hickories (Carya spp.), and an understory composed mainly of spicebush (Lindera benzoin), paw-paw (Asimina triloba), American hornbeam (Carpinus caroliniana), and witch hazel (Hamamelis virginiana).  The SCBI forest plot is 25.6 ha (640 x 400 m), including a 4-ha fenced exclosure where white-tailed deer have been excluded since 1990, as well as three 1-ha Smithsonian Monitoring and Assessment of Biodiversity Program (SI-MAB) sites, with some sites surveyed several times since 1990.  Initial tree tagging and measurement of the full ForestGEO plot occurred in 2008, and the installation finished in 2009. Recensuses have occurred in 2013 and 2018. In 2008, there were 40,411 stems ≥1 cm dbh, representing 62 species, 38 genera and 28 families. In addition, 7 species of woody vines and ca.250 species of herbaceous plants have been identified. The census 3 (2018) at SCBI data have been checked and corrected after a tree survey is finished in a ForestGEO plot.**Reference:** \url{https://github.com/SCBI-ForestGEO/SCBI-ForestGEO-Data/tree/master/tree_main_census}__The 2019 Neon Crown dataset__ The 2019 NeonTreeCrowns dataset is a set of individual level crown estimates for each rectangular bounding box crown prediction includes height, crown area, and spatial location of trees at the site SCBI.This data is surveyed by the National Ecological Observation Network’s Airborne Observation Platform. The predictions are also available as a single csv for each file. All available tiles for that site and year are combined into one large site. These data are not projected, but contain the utm coordinates for each bounding box (left, bottom, right, top). For both file types the following fields are available:**Reference:** \url{https://zenodo.org/record/3765872.YPygfXVKgUF}---__2. What is the unit of observation? What component of the data indicates the source of a given observation?__In the 2018 SCBI census dataset, the __unit of observation uniquely identifies a tree and tree stems__. And each tree has a diameter at the breast height, species, height of measurement, status, UTM, latitude, and longitudes features are captured.In the 2019 Neon Crown dataset, the __unit of observation is also a tree__. And each tree has features such as bounding box UTM coordinates, area, height, score, and the geo_index of the plot since the Noen dataset is Lidar data measured from the National Ecological Observation Network’s Airborne Observation Platform.__3. What are the features in your data?__The following section captures the data types (metadata) for each feature/field in the 2018 and 2019 datasets.__MetaData__Field |Description |Units/Format-----|-----|----- treeID|Unique Tree identifier in CTFS Database (useful for matching trees)|IntstemID|Unique Tree identifier in CTFS Database (useful for matching stems)|Inttag|Unique tree identifier number corresponding to a metal tag of a given tree within the ForestGEO plot|Intstemtag|The number of the stem, if present|Intsp|The species mnemonic|stringdbh|Diameter at breast height as ForestGEO census 2013|cmhom|The height of measure|NumericExactDate|Date of observation|MM/DD/YEARstatus| The tree status, A- alive, D-dead, G-lost_stem, mission or prior|stringcodes| codes for measurement as recorded in the field (census codes)|stringNAD83_X|Tree loc data - NAD UTM X Coodrinate|metersNAD83_Y|Tree loc data - NAD UTM Y Coodrinate|meterslat|Tree loc data - latitude|degreelong|Tree loc data - longitude|degree__Tree 'census' codes__Field |Description -----|-----B| Large Buttress, used ladder to measureM| Multiple StemsA| pom at alternate heightI| Irregular stepP| Any problem require further attentionL| Stem leaningQ| Stem broken above breast heightX| Stem broken below breast heightC| pom has changedY| prostrate stemR| Main stem broken, resproutDS| Dead, stem standingDC| Dead, stem brokenDT| Dead, tag only foundDN| presumed Dead__NEON 2019 Lidar Data__Field |Description |Units/Format-----|-----|----- left| The left UTM X coordinate of the bounding box| metersbottom| The bottom UTM Y coordinate of the bounding box| metersright |The Top UTM X coordinate of the bounding box| meterstop|The Top UTM Y coordinate of the bounding box| metersHeight|The Crown height is defined as the 99th quartile of all canopy height pixels from a LiDAR height model (ID: DP3.30015.001)| metersArea|The crown area- the rectangular bounding box|m2Label|All data in this release are "Tree"|stringScore|The confidence score from the DeepForest deep learning algorithm. The score ranges from 0 (low confidence) to 1 (high confidence)|percentgeo_index|plot UTM geo_index|string* We are reading only the features applicable to the project for memory effeciency.__4. Where is the data being stored?__We download the source data is online at these links -1. 2019 SCBI Census Dataset: \url{https://raw.githubusercontent.com/SCBI-ForestGEO/SCBI-ForestGEO-Data/master/spatial_data/UTM%20coordinates/scbi_stem_utm_lat_long_2018.csv}2. 2019 Neon Crown Dataset: \url{https://zenodo.org/record/3765872/files/SCBI_2019.csv?download=1}We download the dataset directly from the above-noted HTTP links. Both data sources are cleansed, filtered, and analyzed before creating the final_ __merged_2018_2019_scbi.csv__ file.   The merged file is stored at \url{https://github.com/mh4ey/capstone}The merged file contains the significant features used in the computation of Above Ground Biomass (AGG).__5. How are you working with this data?____Cleaning Dataset__We first cleansed the 2018 dataset to remove dead, broken trees and tree stems. Next, we need to merge the ground-based 2018 SCBI census dataset and the 2019 Neon Crown dataset based on the UTM coordinates for the trees in each dataset. The 2019 Neon Crown dataset is the aerial dataset and does not have the treeID uniquely identifying the trees.So, we will use distance-based matching algorithms (Euclidian distance). However, there is significant uncertainty in the matching algorithm. We also know that the small trees measured from the ground are not visible in the aerial dataset, and large tree crowns often cover them.We will try three different filtering options for the 2018 SCBI census dataset to account for the uncertainty in the matching algorithm.1. Filter trees less than 10 meters from the 2018 SCBI census dataset before matching with the 2019 Neon crown dataset.1. Filter trees less than 15 meters from the 2018 SCBI census dataset before matching with the 2019 Neon crown dataset.1. Filter trees less than 20 meters from the 2018 SCBI census dataset before matching with the 2019 Neon crown dataset.__Calcuate  Above Ground Biomass (AGB) in Kilograms__This semester, we only focus on calculating AGB after filtering all trees 10 meters or less from the 2018 SCBI census dataset. We will consider the other two filtering options noted above in the final semester.Then using the R-library, 'allodb’, we calculate the above-ground biomass in kilograms. The feature 'biomass' will be included as the response variable for the model. Aggregating aerial (including LiDAR, hyper-spectral, and camera) and ground-level data will require us to calculate geo-coordinates and merge the tree data from both datasets by latitude and longitude or UTM coordinates using a bounding box.  __Next Steps in Final Semester__Using the cleansed dataset, we aim to create ML predictive models to predict above ground Biomass in Kg for the forest site Smithsonian Conservation Biology Institute (SCBI) ForestGEO Data. And then calibrate the model by applying data from few other sites. Finally, we apply the corrected algorithm to other regions in the United States.<jupyter_code>import numpy as np
import pandas as pd
from uuid import uuid4

import matplotlib.pyplot as plt
import seaborn as sns
import os
sns.set_context('paper', font_scale=1.4)

%matplotlib inline
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.set_option("precision", 7)<jupyter_output><empty_output><jupyter_text>Load 2018 Census Data<jupyter_code>scbi_2018_filename = 'https://raw.githubusercontent.com/SCBI-ForestGEO/SCBI-ForestGEO-Data/master/spatial_data/UTM%20coordinates/scbi_stem_utm_lat_long_2018.csv'
scbi_census_2018 = pd.read_csv(scbi_2018_filename, usecols=['treeID', 'stemID', 'sp',
       'dbh', 'hom','codes', 'status', 'NAD83_X', 'NAD83_Y', 'lat', 'lon'])<jupyter_output><empty_output><jupyter_text>__Rename Columns__<jupyter_code>scbi_census_2018_columns = {
    'treeID' : 'tree_id', 
    'stemID': 'stem_id', 
    'sp': 'species',
    'dbh': 'stem_diameter', 
    'hom': 'point_of_measure', 
    'codes': 'census_codes', 
    'status' : 'tree_status', 
    'NAD83_X' : 'UTM_2018_X', 
    'NAD83_Y': 'UTM_2018_Y', 
    'lat': 'latitude', 
    'lon': 'longitude'
}

# sub select only columns we are interested in
scbi_2018 = scbi_census_2018.rename(columns=scbi_census_2018_columns)
scbi_2018.head()
scbi_2018.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 72555 entries, 0 to 72554
Data columns (total 11 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   tree_id           72555 non-null  int64  
 1   stem_id           72555 non-null  int64  
 2   species           72555 non-null  object 
 3   stem_diameter     54367 non-null  float64
 4   point_of_measure  54369 non-null  float64
 5   census_codes      56163 non-null  object 
 6   tree_status       72555 non-null  object 
 7   UTM_2018_X        72555 non-null  float64
 8   UTM_2018_Y        72555 non-null  float64
 9   latitude          72555 non-null  float64
 10  longitude         72555 non-null  float64
dtypes: float64(6), int64(2), object(3)
memory usage: 6.1+ MB<jupyter_text>__Observation:____3.1 How many observations do you have? There might be multiple types of observations in your dataset?__The 2018 SCBI census ground data has 72555 observations and when loaded into jupyter notebook the pandas dataframe memory usage is about 6.1 MB<jupyter_code>scbi_2018.isna().sum()<jupyter_output><empty_output><jupyter_text>__Observation:__The __stem_diamter, point of measure, tree_status, and census_code__ features have missing values.For this study we choose to drop all rows where these features are missing values.<jupyter_code>scbi_2018 = scbi_2018.dropna()<jupyter_output><empty_output><jupyter_text>__Data Prepration:__The __tree_status, species and tree_status_codes__ are __categorical __ features__. So we will convert these features into pandas category type.<jupyter_code>scbi_2018[['tree_status']].assign(DFstatus=scbi_2018['tree_status'].astype('category'))
scbi_2018[['species']].assign(sp = scbi_2018['species'].astype('category'))
scbi_2018[['census_codes']].assign(status = scbi_2018['census_codes'].astype('category'))
print("Converted DFstatus, sp and status columns to category")
scbi_2018.describe()<jupyter_output><empty_output><jupyter_text>Load NEON Tree Crowns Dataset<jupyter_code>scbi_2019_filename = 'https://zenodo.org/record/3765872/files/SCBI_2019.csv?download=1'
scbi_2019_aop = pd.read_csv(scbi_2019_filename)<jupyter_output><empty_output><jupyter_text>We are only interested in UTM coordinates left,bottom,right,top,score,lable,height and geo_index<jupyter_code>scbi_2019_aop_columns = ['left', 'bottom', 'right', 'top', 'score',
       'height', 'area', 'geo_index']
scbi_2019 = scbi_2019_aop[scbi_2019_aop_columns]
scbi_2019.head()
scbi_2019.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2917297 entries, 0 to 2917296
Data columns (total 8 columns):
 #   Column     Dtype  
---  ------     -----  
 0   left       float64
 1   bottom     float64
 2   right      float64
 3   top        float64
 4   score      float64
 5   height     float64
 6   area       float64
 7   geo_index  object 
dtypes: float64(7), object(1)
memory usage: 178.1+ MB<jupyter_text>__Observation:____3.2 How many observations do you have? There might be multiple types of observations in your dataset?__The 2019 NEON Tree Crowns dataset has 2917297 observations and when loaded into jupyter notebook the pandas dataframe memory usage is about 178.1+ MB<jupyter_code>scbi_2019.isna().sum()
scbi_2019.describe()<jupyter_output><empty_output><jupyter_text>---  Exploratory Data analysis:  2018 SCBI Census DataIn this section we try to discover and visualize the 2018 SCBI census ground data and the 2019 NEON tree crown AOP dataset, to get a general understanding of the data in both the datasets<jupyter_code>plt.figure(figsize=(10,8))
sns.countplot(data=scbi_2018, x='tree_status')
plt.legend(loc=0)
plt.ylabel('Tree count')
plt.xlabel('Tree status')
plt.title('Number of Trees by status')
print()<jupyter_output>No handles with labels found to put in legend.<jupyter_text>__Observation: Dead or broken trees and tree stems__Following are the valid tree_status values in the SCBI 2018 census dataset.    **'A'** - **alive**,    **'D'** - **dead**,    **'P'** - **Not in Census**,    **'G'** - **stem is gone**,    __Note:__Since the primary goal of our project is to create a model to predict the __Above Ground Biomass (AGB)__, we will only consider trees and stems alive and drop all other trees and stems from the dataset.<jupyter_code>scbi_2018 = scbi_2018.query("tree_status not in ['D','G']")
scbi_2018[['tree_id','tree_status']]\
    .groupby('tree_status').count()<jupyter_output><empty_output><jupyter_text>__Observation: The tree_census_codes feature identifies additional trees that are either dead or broken__So we drop any tree or stem with 'dead above 1.3m', 'dead standing', 'stem fallen', 'dead only tag found' and 'presumed dead' codes<jupyter_code>SCBI_STATUS_CODES = {
    'A' : 'alternate POM (point of measure differnt than 1.3)',
    'B' : 'stem broken above 1.3m',
    'C' : 'dead above 1.3m',
    'F' : 'inside deer fence',
    'G' : 'ID to Genius certian',
    'I' : 'stem irrigular where measured',
    'J' : 'bent',
    'L' : 'leaning stem',
    'M' : 'Multiple stem plant',
    'X' : 'stem broken below 1.3 m',
    'P' : 'prostrate stem (allmost parallel to the ground)',
    'S' : 'secondary stem',
    'TR' : 'tag removed',
    'WR' : 'wire removed',
    'DS' : 'dead stem standing',
    'DC' : 'dead stem fallen',
    'DT' : 'dead only tag found',
    'DN' : 'presumed dead no tag or stem',
    'main' : 'main stem',  
}<jupyter_output><empty_output><jupyter_text>Let us examine if the 2018 SCBI census dataset has the census codes that mark the tree observation as either dead or broken.<jupyter_code>dead_tree_or_stem_census_codes = ['B','C', 'X' 'DS*', 'DC','DT', 'DN']
scbi_2018['census_codes'].isin(dead_tree_or_stem_census_codes).value_counts()
scbi_2018[scbi_2018['census_codes'].isin(dead_tree_or_stem_census_codes)].sample(10)<jupyter_output><empty_output><jupyter_text>__Observation:__ Even after deleting dead trees, we see tree data with codes that are not relevant for us in calculating __Above Ground Biomass__.   So we will be dropping these additional 88 rows from the dataset.<jupyter_code>scbi_2018 = scbi_2018[~scbi_2018['census_codes'].isin(dead_tree_or_stem_census_codes)]
scbi_2018['census_codes'].isin(dead_tree_or_stem_census_codes).value_counts()<jupyter_output><empty_output><jupyter_text>__Observation:__ Tree count by species<jupyter_code>plt.figure(figsize=(16,10))
count_plot = sns.countplot(data=scbi_2018, x='species', palette='pastel')
count_plot.set_xticklabels(labels=count_plot.get_xticklabels(), rotation=45)
plt.ylabel('Tree count')
plt.xlabel('Tree Species')
plt.title('Number of Trees by Species')
plt.show()<jupyter_output><empty_output><jupyter_text>__observation:__ We see from the above plot that there are many species with less than 100 trees. __observation:__ Imbalanced datasetAs we can see from the above count plot, the scbi 2018 census dataset is imbalanced. We need to address the data imbalance when creating train and test datasets<jupyter_code>plt.figure(figsize=(16,8))
box_plot = sns.boxplot(x='species', y='stem_diameter', data=scbi_2018, palette='pastel')
box_plot.set_xticklabels(labels=box_plot.get_xticklabels(), rotation=45)
plt.show()<jupyter_output><empty_output><jupyter_text>__Observation: Stem diameter measurements vary significantly across species__ __Number of unique trees in SCBI census dataset:__<jupyter_code>scbi_2018.tree_id.nunique()
scbi_2018.shape<jupyter_output><empty_output><jupyter_text>__How many trees haves more than one stem?__<jupyter_code>tree_stems = scbi_2018[['tree_id', 'stem_id']]\
    .groupby('tree_id')\
    .count()\
    .query('stem_id == 1')\
    .sort_values(by='stem_id', ascending=False)
tree_stems_gt1 = scbi_2018[['tree_id', 'stem_id']]\
    .groupby('tree_id')\
    .count()\
    .query('stem_id > 1')\
    .sort_values(by='stem_id', ascending=False)
print(f'{tree_stems_gt1.shape[0]} trees have more than one stems')
print(f'{tree_stems.shape[0]} trees have only one stems')<jupyter_output>8051 trees have more than one stems
13615 trees have only one stems<jupyter_text>NEON Tree Crown Dataset<jupyter_code>scbi_2019.head()<jupyter_output><empty_output><jupyter_text>__How many trees exist by geo_index in 2019 dataset?__<jupyter_code>sns.set_context('paper', font_scale=1.0)
plt.figure(figsize=(22,8))
box_plot = sns.boxplot(x='geo_index', y='area', data=scbi_2019, palette='pastel')
box_plot.set_xticklabels(labels=box_plot.get_xticklabels(), rotation=45)
plt.show()<jupyter_output><empty_output><jupyter_text>Feature Engineering For matching Trees from 2018 and 2019 dataset -The SCBI 2018 dataset has the UTM coordirnates __2018_UTM_X__ and __2018_UTM_Y__ as the North American Datum (NAD) is the horizontal datum now used to define the geodetic network in North America.Reference:https://en.wikipedia.org/wiki/North_American_Datum            https://geodesy.noaa.gov/datums/horizontal/north-american-datum-1983.shtml          __Create a geo_index_feature in SCBI 2018 dataset based on the UTM coordinates__ The 2018 census dataset has data for two geo_index plots. So we will filter the 2019 dataset by the 2018 census dataset geo_index plots.<jupyter_code>def subX(x):
    return str(x)[0:3] +'000'
def subY(y):
    return str(y)[0:4] +'000'
scbi_2018 = scbi_2018.assign(geo_index = scbi_2018['UTM_2018_X'].apply(subX) + '_' + scbi_2018['UTM_2018_Y'].apply(subY))

#count rows by geo_index
scbi_geo = scbi_2018[['tree_id','geo_index']]\
     .rename(columns = {'tree_id' :'count'})\
     .groupby('geo_index').count()\
     .reset_index()
scbi_geo<jupyter_output><empty_output><jupyter_text>__Filter the scbi_2019 dataset by the geo_index data in 2018 SCBI census dataset__<jupyter_code>scbi_2019_site = scbi_2019[scbi_2019['geo_index'].isin(scbi_2018.geo_index.unique())]
scbi_2019_site[['area','geo_index']]\
     .rename(columns = {'area' :'count'})\
    .groupby('geo_index').count()
scbi_2019_site.head()<jupyter_output><empty_output><jupyter_text>__Create unique_id for eachrow in 2019 dataset__<jupyter_code>import uuid
ids = [uuid.uuid4() for _ in range(len(scbi_2019_site.index))]
scbi_2019_site = scbi_2019_site.assign(id =ids )
scbi_2019_site.head()<jupyter_output><empty_output><jupyter_text>__The frequency distribution of trees by area in 2019 dataset__<jupyter_code>sns.displot(data = scbi_2019_site, x='area', kde=True, bins=15)
plt.figure(figsize=(10,8))
sns.boxplot(x='geo_index', y='area', data=scbi_2019_site, hue='geo_index', palette='pastel')
#move legend
plt.legend(loc=0)
scbi_2019_site.area.describe()<jupyter_output><empty_output><jupyter_text>__observation:__We can see that the area for 75% of trees fall in the range of 0 to 26.52 square meters.<jupyter_code>plt.figure(figsize=(12,8))
sns.boxplot(x='geo_index', y='height', data=scbi_2019_site, hue='geo_index', palette='pastel')
#move legend
plt.legend(loc=8)<jupyter_output><empty_output><jupyter_text>__Observation:__We see from the above plot that the majority of the trees have a height between 20 and 40 meters. __Create new 2019_UTM_X and 2019_UTM_Y for 2019 NEON Crown dataset__<jupyter_code>def avgUTM(UTM1,UTM2):
    '''
    average of two UTM Coodrinates
    '''
    return (UTM1+UTM2)/2

# compute and create Center_X_2019 UTM coortinate column
scbi_2019_site = scbi_2019_site.assign(UTM_2019_X=lambda x: avgUTM(x['left'], x['right']))

# compute and create Center_Y_2019 UTM coortinate column
scbi_2019_site = scbi_2019_site.assign(UTM_2019_Y=lambda x: avgUTM(x['bottom'], x['top']))

scbi_2019_site.head(5)<jupyter_output><empty_output><jupyter_text>Calculating Euclidean Distance between UTM coordinates of the SCBI 2018 census dataset and 2018 NEON Tree Crowns AOP dataset<jupyter_code>def euclidean_distance(point1, point2):
    '''
    compute euclidean distance between two points
    '''
    ed= np.linalg.norm(point1 - point2)
    return ed
def compute_euclidean_distance(utm_2018_and_2019_df):
    '''
    compute euclidean distance between UTM coordinates of 2018 and 2019 dataset
    '''
    # create an empty array
    ed = []
    
    # iterate rows of the dataset using itertuples
    for row_tupple in utm_2018_and_2019_df.itertuples():
        
        # compute euclidean distance between for a tree 2018 UTM coorinates with all the 2019 dataset UTM coordinates
        ecd = euclidean_distance(np.array((row_tupple.UTM_2018_X, row_tupple.UTM_2018_Y)), 
            np.array((row_tupple.UTM_2019_X, row_tupple.UTM_2019_Y)))
        
        #append the eucliedian distance to 2019 dataset
        ed.append(ecd)
    
    #append the array as new columns 
    utm_2018_and_2019_df['euclidean_dist'] = ed
    
    return utm_2018_and_2019_df<jupyter_output><empty_output><jupyter_text>__Matching Algorithm__- The merging Algorithm takes the  stem_diameter_filter, tree_height_filter, geo_index, scbi_2018, and neon_2019 datasets.- All trees with stem_diamter <= stem_diameter_filter in cms are filtered from the scbi 2018 census dataset.- All trees with height <= tree_height_filter in meters are filtered from  the 2019Neon crown  dataset- First, from the Bounding Box UTM coordinates in the 2019 dataset (left, bottom, top, and right features), we created a center point X and Y coordinates represented by UTM_2019_B and UTM_2019_Y features.- The SCBI 2018 census dataset has tree-specific UTM coordinates represented as UTM_2018_X, UTM_2018_Y. - Then we computed Euclidean distance for a tree in the 2018 dataset to all the trees in the 2019 dataset and then selected the top 5 matched trees by the shortest distance between tree centers' in the 2018 and 2019 UTM coordinates.- The final merged dataset has 5 top matches from the 2019 Neon Crown dataset for every tree in the 2018 SCBI census dataset.- Then group the merged dataset by the UUID feature (created for the 2019 dataset), then sort by euclidean distance in ascending order.- Then group the merged dataset by the tree_id feature (created for the 2019 dataset), then sort by euclidean distance in ascending order.- Select the first tree with the shortest distance and purge the remaining trees with the same UUID from the dataset.- Each observation in the final merged dataset matches best of the top 5 matches by shortest Euclidean distance between the centers of trees in the 2018 and 2019 datasets.<jupyter_code>def match_and_merge(geo_index, scbi_2018, neon_2019_subset, stem_diameter_filter = 10, tree_hight_filter = 10):
    '''
        merge 2018 SCBI census dataset and 2019 NEOW Crown dataset of trees & create a new dataset with features -
    '''
    
    #filter data by geo_index 
    df_1 = scbi_2018[scbi_2018['geo_index'] == geo_index].head(10)
    
    # filter out the smallest trees from 2018 dataset by dbh
    df_1 = df_1[df_1['stem_diameter'] > stem_diameter_filter]
    
    print(f"2018 SCBI dataset shape {df_1.shape}")
    
    #filter data by geo_index 
    df_2 = neon_2019_subset[neon_2019_subset['geo_index'] == geo_index]
    
    # filter out the smallest trees from 2019 dataset by dbh
    df_2 = df_2[df_2['height'] > tree_hight_filter]
    
    print(f"2019 Neon crown dataset shape {df_2.shape}")
    
    #create and empty dataframe object
    matched_df = pd.DataFrame(columns =[
        'tree_id', 'matched_tree_id', 'stem_id','species', 'stem_diameter', 'point_of_measure', 'score', 'height', 'area' , 'UTM_2018_X', 'UTM_2018_Y', 
        'UTM_2019_X', 'UTM_2019_Y', 'euclidean_dist', 'left','bottom', 'right', 'top', 'tree_status', 'census_codes',
    ])
    
    #for every tree in df1 match the nearest tree by euclidian distance
    for irow in df_1.itertuples():
        # copy 2018 tree dataset to 2019 ds
        df_2 = df_2.assign(UTM_2018_X = irow.UTM_2018_X)
        df_2 = df_2.assign(UTM_2018_Y = irow.UTM_2018_Y)
        
        # compute euclidian distance from tree in 2018 dataset
        df_2 = compute_euclidean_distance(df_2)
        
        top5_matched_trees = df_2.sort_values(by = 'euclidean_dist', ascending = True).head(5)
        
        top5_trees_df = pd.DataFrame(columns =[
        'tree_id', 'matched_tree_id', 'stem_id','species', 'stem_diameter', 'point_of_measure', 'score', 'height', 'area' , 'UTM_2018_X', 'UTM_2018_Y', 
        'UTM_2019_X', 'UTM_2019_Y', 'euclidean_dist', 'left','bottom', 'right', 'top', 'tree_status', 'census_codes',
        ])
        

        for row in top5_matched_trees.itertuples():
            
            top5_trees_df = top5_trees_df.append({'tree_id' : irow.tree_id,
                                        'matched_tree_id' : row.id,
                                        'stem_id' : irow.stem_id,
                                        'species' : irow.species,
                                        'stem_diameter' : irow.stem_diameter,
                                        'point_of_measure': irow.point_of_measure,
                                        'score': row.score, #matched_tree['score'].iloc[0],
                                        'height': row.height, #matched_tree['height'].iloc[0],
                                        'area': row.area, #matched_tree['area'].iloc[0],
                                        'UTM_2018_X': irow.UTM_2018_X,
                                        'UTM_2018_Y': irow.UTM_2018_Y,
                                        'UTM_2019_X': row.UTM_2019_X, #matched_tree['UTM_2019_X'].iloc[0],
                                        'UTM_2019_Y': row.UTM_2019_X,#matched_tree['UTM_2019_Y'].iloc[0],
                                        'euclidean_dist': row.euclidean_dist, #matched_tree['euclidean_dist'].iloc[0],
                                        'left': row.left, #matched_tree['left'].iloc[0],
                                        'bottom': row.bottom, # matched_tree['bottom'].iloc[0],
                                        'right': row.right, #matched_tree['right'].iloc[0],
                                        'top': row.top, #matched_tree['top'].iloc[0],
                                        'tree_status': irow.tree_status,
                                        'census_codes': irow.census_codes
                                       }, ignore_index=True)
        
        matched_df = matched_df.append(top5_trees_df)
    
        
    return matched_df
def filter_unique_match_by_ecd(matched_df):
    scbi_2018_neon_2019_merged_df = pd.DataFrame(columns =['tree_id', 'matched_tree_id', 'stem_id','species', 'stem_diameter', 'point_of_measure',
                                                           'score', 'height', 'area' , 'UTM_2018_X', 'UTM_2018_Y', 'UTM_2019_X', 'UTM_2019_Y', 
                                                           'euclidean_dist', 'left','bottom', 'right', 'top', 'tree_status', 'census_codes' ])
     
    filterd_scbi_2018_neon_2019_merged_df = pd.DataFrame(columns =['tree_id', 'matched_tree_id', 'stem_id','species', 'stem_diameter', 'point_of_measure',
                                                           'score', 'height', 'area' , 'UTM_2018_X', 'UTM_2018_Y', 'UTM_2019_X', 'UTM_2019_Y', 
                                                           'euclidean_dist', 'left','bottom', 'right', 'top', 'tree_status', 'census_codes' ])
          
    # unique matched_tree_ids
    unique_matched_trees = matched_df.matched_tree_id.unique()
    
    for m_tree_id in unique_matched_trees:

        temp_df = matched_df[matched_df['matched_tree_id'] == m_tree_id].sort_values(by='euclidean_dist', ascending=True).head(1)
        
        #print(f"tree_id: {temp_df['tree_id']} matched_tree_id: {temp_df['matched_tree_id']} euclidean_id: {temp_df['euclidean_dist']}")
        scbi_2018_neon_2019_merged_df = scbi_2018_neon_2019_merged_df.append(temp_df.head(1))
        
    
    # unique tree_ids
    unique_tree_ids = matched_df.tree_id.unique()
    
    for tree_id in unique_tree_ids:
        temp_df = matched_df[matched_df['tree_id'] == tree_id].sort_values(by='euclidean_dist', ascending=True).head(1)
        filterd_scbi_2018_neon_2019_merged_df = filterd_scbi_2018_neon_2019_merged_df.append(temp_df)
    return filterd_scbi_2018_neon_2019_merged_df.reset_index(drop=True)
        
matched_2018_s1 = match_and_merge('747000_4308000', scbi_2018, scbi_2019_site)
matched_2018_s1 = filter_unique_match_by_ecd(matched_2018_s1)
matched_2018_s2 = match_and_merge('747000_4309000', scbi_2018, scbi_2019_site)
matched_2018_s2 = filter_unique_match_by_ecd(matched_2018_s2)
merged_and_matched_trees_2018_2019_df = pd.concat([matched_2018_s1, matched_2018_s2], ignore_index=True)
merged_and_matched_trees_2018_2019_df.to_csv('data/merged_and_matched_trees.csv')
merged_and_matched_trees_2018_2019_df.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08744897019552472, 'fraction_numerical': 0.046655821234537584, 'mean_word_length': 4.3018714401952804, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 117, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 33, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '327115', 'license': 'MIT', 'n_tokens_mistral': 11223, 'n_tokens_neox': 9691, 'n_words': 3263, 'path': 'notebooks/capstone1-eda-and-matching1.0.ipynb', 'repo_name': 'jayhombal/forest-biomass-prediction', 'size': '32598'}