<jupyter_start><jupyter_text>Attribute geospatial data with seal abundance dataThis notebook uses the information on the estimated number of seals per map to attribute the basic grid of geospatial data with seal abundance. It then tabulates the resulting raster, where each column is one of the geospatial datasets, and there is a column for seal abundance. The resulting table is saved and becomes the basic input for the modeling of factors deteriming the presence and abundance of seals throughout the entire continent.    Run this file every time a new geospatial covariate is added to the collection.  Loading the required libraries and other dependencies<jupyter_code>## Clear memory
rm(list=ls())
gc()
libs<-c("ggplot2","plyr","dplyr","sp","rgeos","raster","rgdal")
lapply(libs, require, character.only = TRUE)
pathToLocalGit<-"/home/ubuntu/Workspace/ContinentalWESEestimates/"

## load the WESE map data
load(file=paste0(pathToLocalGit,"data/FinalWESEcounts.RData"))

## Load the current table of data
load(file=paste0(pathToLocalGit,"data/studyarea_points_wNearLandPenguins.RData"))

## Source the functions file
source(paste0(pathToLocalGit,"scripts/countSealsFromTags_functions.R"))
 
cdf<-as.data.frame(studyarea_pointswLandPenguins,xy=TRUE)
cdf$pointid<-as.integer(as.character(cdf$pointid))
NROW(unique(cdf$pointid))==nrow(cdf)
write.csv(countdf,file=paste0(pathToLocalGit,"data/WESElocations_counts.csv"))<jupyter_output><empty_output><jupyter_text>The current table of data is a spatial data.frame of a 5 km grid of cells, with geospatial attributes for each cell. We want to add one more attribute to this cell: WESE abundance. We use the WESE data frame for this purpose. So, we first attribute the WESE data, which are summarized by 500-m maps, with the cellId of the 5-km grid. We then aggregate the map counts by (add up to) the 5-km cellId. Finally, we merge the WESE data to the geospatial data, and save as data.frame. We use a function that does this, using UTM coordinates<jupyter_code>#wese5k<-getWESEcountsBy5km(gdf=studyarea_pointswLandPenguins,wesedf=unique(countdf[,c("regionMapId","mapcoords.x1","mapcoords.x2")]))
## Checking
load(file=paste0(pathToLocalGit,"data/wese5k.RData"))
sum(is.na(wese5k$pointid))  #must be 0
nrow(wese5k)==nrow(unique(countdf[,c("regionMapId","mapcoords.x1","mapcoords.x2")]))  #TRUE
nrow(wese5k)*2==nrow(countdf)   #TRUE
sum(is.na(wese5k$mapcoords.x1))   #must be 0
## IGNORE THIS unless you can fix the few haul-outs > 5km away from grid point
## Now we merge with countdf
## But before we do that, here's a check of coordinates, because some maps are > 5km away from nearest grid cell
odf<-cdf[,c("pointid","coords.x1","coords.x2")]
tdf<-merge(wese5k,odf, by="pointid",all.x=T)
rdf<-unique(countdf[,c("regionMapId","mapcoords.x1","mapcoords.x2")])
names(rdf)<-c("regionMapId","mcx1","mcx2")
tdf<-merge(tdf,rdf,by="regionMapId",all.x=T)
tdf$dist<-sqrt(((tdf$mapcoords.x1-tdf$coords.x1)^2)+((tdf$mapcoords.x2-tdf$coords.x2)^2))
head(tdf)
## It looks like this is the best we can do for now, so...
## Let's use 2011 estimates
nrow(wese5k)
y2011df<-subset(countdf, Year=="2011",select=c("mdlColEstimate","mdlIslEstimate","regionMapId"))
wesedata<-merge(y2011df,wese5k[,c("regionMapId","pointid")],by="regionMapId",all.x=TRUE)
nrow(wesedata);nrow(y2011df)
head(wesedata)
## Merging with the geospatial data.frame now...
wdf<-merge(cdf,wesedata[,c("mdlColEstimate","mdlIslEstimate","pointid")],by="pointid",all.x=TRUE)
## But we will end up with several maps per grid cell
nrow(wdf);nrow(cdf)
## So we must aggregate... (dplyr kills the kernel)
wesedf<-unique(wdf[,c("pointid","meanslope","meanbathy","glacierdist","distToShore","cont300dist","cont800dist",
                      "DecemberIcePresence","Persistence2Years","PredictabilityDec5Years","distNearestIceEdge",
                      "fastIceWidth","fastIcePresent","ADPEname","ADPEdist","ADPEabund","EMPEname","EMPEdist",
                      "EMPEabund","coords.x1","coords.x2")])
mdlColsum<-aggregate(mdlColEstimate~pointid,wdf,sum);names(mdlColsum)<-c("pointid","mdlCol")
mdlIslsum<-aggregate(mdlIslEstimate~pointid,wdf,sum);names(mdlIslsum)<-c("pointid","mdlIsl")
wesedf<-merge(wesedf,mdlColsum,by="pointid",all.x=TRUE)
wesedf<-merge(wesedf,mdlIslsum,by="pointid",all.x=TRUE)
#wesedf<-as.data.frame(wdf %>% 
#            group_by(pointid,meanslope,meanbathy,slope,bathy,shoredist,glacierdist,nearLineId,near_x,near_y,distToShore,
#                     adpedist,adpecol,empedist,empecol,cont300dist,cont800dist,DecemberIcePresence,Persistence2Years,
#                     PredictabilityDec5Years,coords.x1,coords.x2) %>% 
#                dplyr::summarize(mdlCol=sum(mdlColEstimate),mdlIsl=sum(mdlIslEstimate)))

nrow(cdf)==nrow(wesedf)   #must be TRUE
## There are some grid cells with maps with possible seals, except that the mean expected value is 0, while max > 0
## So, here we distinguish these from grid cells with no maps with seals (i.e., true 0 seals)
wesedf$hasMaps<-ifelse(!is.na(wesedf$mdlIsl),1,0)

#So, now we can make the NA's be 0
wesedf$mdlCol<-ifelse(is.na(wesedf$mdlCol),0,wesedf$mdlCol)
wesedf$mdlIsl<-ifelse(is.na(wesedf$mdlIsl),0,wesedf$mdlIsl)

head(wesedf)
q<-subset(wesedf,mdlCol>0 & fastIcePresent==FALSE)
sum(q$mdlCol)
nrow(subset(wesedf,mdlCol>0)) 
load(paste0(pathToLocalGit,"data/FastIceGridPoints_weseNoIce_111114.RData"))
FIGPdf<-FIGPdf[,which(!names(FIGPdf) %in% c('near.x1','near.x2','iceedge.x1','iceedge.x2','fastIceAbund'))]
FIGPdf$ADPEname<-as.character(FIGPdf$ADPEname)
FIGPdf$EMPEname<-as.character(FIGPdf$EMPEname)
names(wesedf)
names(FIGPdf)
#checking
FIGPdf[FIGPdf$pointid==122914,"meanslope"]
FIGPdf[FIGPdf$pointid==123037,"glacierdist"]
FIGPdf[FIGPdf$pointid==123042,"EMPEname"]
#Then matchning by pointid, replace the values of all fields from FIGPdf into wesedf
ednames<-names(FIGPdf)[which(!names(FIGPdf) %in% c("pointid","coords.x1","coords.x2"))]
for(rr in 1:nrow(FIGPdf)){
    pid<-as.numeric(FIGPdf[rr,"pointid"])
    for(nn in ednames){
        wesedf[wesedf$gridCellId==pid,nn]<-FIGPdf[rr,nn]
    }
}

#match?
wesedf[wesedf$gridCellId==122914,"meanslope"]
wesedf[wesedf$gridCellId==123037,"glacierdist"]
wesedf[wesedf$gridCellId==123042,"EMPEname"]
#checking about the number of seals we lose from no ice attrib
nrow(subset(wesedf,fastIcePresent==FALSE & mdlCol>0))
sum(subset(wesedf,fastIcePresent==FALSE)$mdlCol)
nrow(wesedf)
## Now we save and setup the linear model analyses...
names(wesedf)<-gsub("pointid","gridCellId",names(wesedf))
save(wesedf,file=paste0(pathToLocalGit,"data/continentalWESE.RData"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14467120181405896, 'fraction_numerical': 0.022071050642479214, 'mean_word_length': 7.097919216646267, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '408894', 'license': 'MIT', 'n_tokens_mistral': 2668, 'n_tokens_neox': 2487, 'n_words': 482, 'path': 'notebooks/.ipynb_checkpoints/AttributeGrid_withWESEdata-checkpoint.ipynb', 'repo_name': 'leosalas/ContinentalWESEestimates', 'size': '6615'}