Q:

R how to filter a time series of measurements based on previous values

I am attempting to filter coral demographic data in a time series. I have a set of corals that have been measured every 3 months. What I want to do is a.) filter for all corals that at some point had a maximum diameter within a specified size range (8 - 12 mm diameter), b.) remove corals that were previously larger than the size range, and c.) remove coral measurements that fell into the size range AFTER it grew through or past the size range by only including for each coral the FIRST measurement in which it grew into the size range (8-12 mm) and the subsequent measurement in the next TimeStep.
I have created a sample database and the desired database to specifically illustrate what I am looking for. In the sample database, I have also included all the criteria listed for each coral below in the notes section next to the first entry for each coral for your reference. Here are the 8 corals I have included in the database and what I want to be done with them in words:
Coral # 1 should be deleted from the database entirely because it skipped over the desired size range of 8-12 mm
Coral # 2 should be deleted from the database because it started above the desired size range then shrank below it, then grew into it. I only want corals that have grown into the size range without shrinking beforehand
Coral # 3 is an example of a coral that grew to the size range (8-12 mm) and beyond without shrinking and this is a coral that I want to keep because it grew to the size range.  However, I want to only include the FIRST measure inside the size range (9 mm in this case in TimeStep 3) and the proceeding measurement (12 mm in this case for TimeStep 4)
Coral # 4 is an example of a coral that started off and remained above the size range and therefore needs to be removed.
Coral # 5 is an example of a coral that started below the range, grew into it, then later shrank back into the range (TimeStep 4). For this scenario, I want to only include the first time the diameter fell into the range (TimeStep 2) and the proceeding measurement (TimeStep 3), not the second time it fell into the range. This is because the first time is natural growth whereas the second time is shrinkage and its resulting recovery (which I want to exclude or filter out).
Coral # 6 is an example of a coral that started in the size range for TimeStep 1 and then grew out of it in the next TimeStep and continued to grow after. I want to maintain only the measurements in TimeStep 1 and 2 (the first measure inside the range and the proceeding measurement)
Coral # 7 is an example of a coral that started in the size range in TimeStep 1 and then remained in the range for TimeStep 2. In this case I only want the first measurement in the size range (TimeStep 1) and the subsequent measurement (TimeStep 2)
Coral # 8 is an example of a coral that grew to the size range in TimeStep 3, stayed in the range (10 => 9) in TimeStep 4, then shrank below the desired range then for TimeStep 6 grew back to the range. For this colony, again I want the FIRST measurement inside the range (10 mm at TimeStep 3) and the proceeding measurement in TimeStep 4 included for this coral
Coral # 9 is an example of a coral that grew into the size range in TimeStep 3 (9 mm) but was not found in the following TimeStep (NF for the Status Code column with measurement as NA).  I want to keep corals like this in the dataset in order to calculate survivorship.
All told, I want code that filters this database such that if a coral at some point has a diameter in the 8-12 cm size range but was previously larger than that range, was never at or below the range, or started below the range and never fell within it, they are removed from the database entirely. Also, I am looking to keep any corals that grew to the range and then shrank back to it in the database while removing the second time it fell in the range. This would be done by removing all the measurements EXCEPT the first TimeStep in which the coral grew into the size range and the following TimeStep measurement.
SAMPLE DATABASE
data <- structure(list(Site = c("WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI"), `Module #` = c(116, 116, 116, 116, 116, 116, 116, 115, 
115, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 
116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 
116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 
116, 116, 116, 116, 116), Side = c("N", "N", "N", "N", "N", "N", 
"N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", 
"N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", 
"N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N", 
"N", "N", "N", "N", "N", "N", "N"), TimeStep = c(1, 2, 3, 4, 
5, 6, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 1, 
2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 
5, 6, 1, 2, 3, 4), Settlement_Area = c(0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336), `Colony #` = c(1, 1, 1, 1, 1, 1, 2, 
2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 
5, 5, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 9, 
9, 9, 9), Location = c("C1", "C1", "C1", "C1", "C1", "C1", "B4", 
"B4", "B4", "B4", "B4", "B4", "A1", "A1", "A1", "A1", "A1", "A1", 
"B3", "B3", "B3", "B3", "B3", "B3", "D1", "D1", "D1", "D1", "D1", 
"D1", "A2", "A2", "A2", "A2", "A2", "A2", "A4", "A4", "A4", "A4", 
"A4", "A4", "B3", "B3", "B3", "B3", "B3", "B3", "A3", "A3", "A3", 
"A3"), `Taxonomic Code` = c("PC", "PC", "PC", "PC", "PC", "PC", 
"PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", 
"PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", 
"PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", 
"PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", "PC", 
"PC", "PC"), `Cover Code` = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, NA), 
    `Max Diameter (cm)` = c(5, 7, 13, 15, 16, 19, 15, 7, 9, 11, 
    14, 18, 3, 6, 9, 12, 15, 20, 13, 16, 18, 21, 23, 26, 6, 9, 
    14, 12, 15, 18, 11, 14, 17, 17, 21, 24, 9, 11, 14, 16, 20, 
    22, 3, 6, 10, 9, 7, 10, 4, 6, 9, NA), `Status Code` = c(NA, 
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
    NA, NA, NA, NA, NA, "NF"), Notes = c("coral # 1 should be deleted from the database because it skipped over the desired size range of 8-12 mm", 
    NA, NA, NA, NA, NA, "coral # 2 should be deleted from the database because it started above the desired size range then shrank back into it.  I only want corals that have grown into the size range", 
    NA, NA, NA, NA, NA, "Colony # 3 is an example of a coral that grew to the size range (8-12 mm) and beyond without shrinking and this is a coral that I want to keep because it grew to the size range.  However, I want to only include the FIRST measure inside the size range (9 mm in this case) and the proceeding measurement (12 mm)", 
    NA, NA, NA, NA, NA, "Colony # 4 is an example of a coral that started off above the size range and therefore needs to be removed.", 
    NA, NA, NA, NA, NA, "Colony # 5 is an example of a coral that started below the range, grew into it, then later shrank back into the range (TimeStep 4). For this scenario, I want to only include the first time the diameter fell into the range (TimeStep 2) and the proceeding measurement, not the second time it fell into the range. This is because the first time is natural growth whereas the second time is shrinkage and its resulting recovery (which I want to exclude or filter out).", 
    NA, NA, NA, NA, NA, "Colony # 6 is an example of a coral that started in the size range for TimeStep 1 and then grew out of it in the next TimeStep and continued to grow after. I want to maintain only the measurements in TimeStep 1 and 2 (the first measure inside the range and the proceeding measurement)", 
    NA, NA, NA, NA, NA, "Colony # 7 is an example of a coral that started in the size range in TimeStep 1 and then remained in the range for TimeStep 2. In this case I only want the first measurement in the size range (TimeStep 1) and the subsequent measurement (TimeStep 2)", 
    NA, NA, NA, NA, NA, "Colony # 8 is an example of a coral that grew to the size range in TimeStep 3, stayed in the range (10 => 9) in TimeStep 4, then shrank below the desired range then for TimeStep 6 grew back to the range. For this colony, again I want the FIRST measurement inside the range (10 mm at TimeStep 3) and the proceeding measurement in TimeStep 4 included for this coral", 
    NA, NA, NA, NA, NA, NA, NA, NA, NA)), class = c("spec_tbl_df", 
"tbl_df", "tbl", "data.frame"), row.names = c(NA, -52L), spec = structure(list(
    cols = list(Site = structure(list(), class = c("collector_character", 
    "collector")), `Module #` = structure(list(), class = c("collector_double", 
    "collector")), Side = structure(list(), class = c("collector_character", 
    "collector")), TimeStep = structure(list(), class = c("collector_double", 
    "collector")), Settlement_Area = structure(list(), class = c("collector_double", 
    "collector")), `Colony #` = structure(list(), class = c("collector_double", 
    "collector")), Location = structure(list(), class = c("collector_character", 
    "collector")), `Taxonomic Code` = structure(list(), class = c("collector_character", 
    "collector")), `Cover Code` = structure(list(), class = c("collector_double", 
    "collector")), `Max Diameter (cm)` = structure(list(), class = c("collector_double", 
    "collector")), `Status Code` = structure(list(), class = c("collector_character", 
    "collector")), Notes = structure(list(), class = c("collector_character", 
    "collector"))), default = structure(list(), class = c("collector_guess", 
    "collector")), skip = 1), class = "col_spec"))

DESIRED DATABASE
data_final <- structure(list(Site = c("WAI", "WAI", "WAI", "WAI", "WAI", "WAI", 
"WAI", "WAI", "WAI", "WAI", "WAI", "WAI"), `Module #` = c(116, 
116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116), Side = c("N", 
"N", "N", "N", "N", "N", "N", "N", "N", "N", "N", "N"), TimeStep = c(3, 
4, 2, 3, 1, 2, 1, 2, 3, 4, 3, 4), Settlement_Area = c(0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336, 
0.75902336, 0.75902336, 0.75902336, 0.75902336, 0.75902336), 
    `Colony #` = c(3, 3, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9), Location = c("A1", 
    "A1", "D1", "D1", "A2", "A2", "A4", "A4", "B3", "B3", "B2", 
    "B2"), `Taxonomic Code` = c("PC", "PC", "PC", "PC", "PC", 
    "PC", "PC", "PC", "PC", "PC", "PC", "PC"), `Cover Code` = c(1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, NA), `Max Diameter (cm)` = c(9, 
    12, 9, 14, 11, 14, 9, 11, 10, 9, 9, NA), `Status Code` = c(NA, 
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "NF")), class = c("spec_tbl_df", 
"tbl_df", "tbl", "data.frame"), row.names = c(NA, -12L), spec = structure(list(
    cols = list(Site = structure(list(), class = c("collector_character", 
    "collector")), `Module #` = structure(list(), class = c("collector_double", 
    "collector")), Side = structure(list(), class = c("collector_character", 
    "collector")), TimeStep = structure(list(), class = c("collector_double", 
    "collector")), Settlement_Area = structure(list(), class = c("collector_double", 
    "collector")), `Colony #` = structure(list(), class = c("collector_double", 
    "collector")), Location = structure(list(), class = c("collector_character", 
    "collector")), `Taxonomic Code` = structure(list(), class = c("collector_character", 
    "collector")), `Cover Code` = structure(list(), class = c("collector_double", 
    "collector")), `Max Diameter (cm)` = structure(list(), class = c("collector_double", 
    "collector")), `Status Code` = structure(list(), class = c("collector_character", 
    "collector"))), default = structure(list(), class = c("collector_guess", 
    "collector")), skip = 1), class = "col_spec"))

So far I have been able to get corals that never were in the size range by creating a vector of unique colony numbers that did fall in the 8 to 12 mm :
size_vect <- seq(from = 8, to = 12, by = 1)
# a vector containing the range of diameter measurements we want to filter for

ID_vect <- data %>% group_by(`Colony #`) %>% 
filter(`Max Diameter (cm)` > min(size_vect) & `Max Diameter (cm)` < max(size_vect)) %>% 
# select all measures where the coral fell within the size range
distinct(`Colony #`) %>% 
# remove duplicate colony numbers
pull(`Colony #`)
# make the column `Colony #` in the dataframe ID_vect into a vector

Then I filtered the full sample database to only include the coral colonies from ID_vect:
data_new <- data %>% group_by(`Colony #`) %>%
filter(`Colony #` %in% ID_vect) 
# filter for all corals that contain the same colony number as those in the ID_vect

I do not know how to now filter the database based on the following condition: if a coral fell into the size range at some point but a previous measurement was LARGER than the maximum value of the desired size range (12 mm), that coral should be removed completely.  For example Coral #2 should be removed because before the value fell into the range in TimeStep 3 it was 15 mm in TimeStep 1 which exceeds the range.
Additionally, I do not know how to account for if there was no measurment in the next TimeStep measurement such as with Coral #9 where it was measured as 9 mm in TimeStep 3 and was not found (NF in Status Code) in TimeStep 4.  I need to preserve the TimeStep 4 measurement to calculate survivorship.  I do not know how to code this conditional filter and this is where I need help.  Any code advice is appreciated thanks! 

A:

We can use run length encoding to help us keep up with transitions from in-range to out of range. It's so much easier with data.table::rleid that I'll recommend using that. 
Here's an example of RLE in action on coral 8.
 `Colony #` `Max Diameter (cm)` InRange RLE
          8                   3   FALSE   1
          8                   6   FALSE   1
          8                  10    TRUE   2
          8                   9    TRUE   2
          8                   7   FALSE   3
          8                  10    TRUE   4

Once the RLE is encoded, we filter in rows that have a minimum in-range RLE that is below the minimum above-range RLE. If any such rows exist, we look for the first time point that is in range and also filter in the next time point.
library(dplyr)
library(data.table)
data %>% 
  select(-Notes) %>%
  mutate(InRange = case_when(`Max Diameter (cm)` >= 8 & `Max Diameter (cm)` <= 12 ~ TRUE,
                             TRUE ~ FALSE)) %>% 
  mutate(AboveRange = case_when(`Max Diameter (cm)` > 12 ~ TRUE,
                                TRUE ~ FALSE)) %>% 
  group_by(`Colony #`) %>%
  mutate(RLE = data.table::rleid(InRange)) %>% 
  mutate(MinIn = min(RLE[InRange]), MinAbove = min(RLE[AboveRange]), MinInTime = min(TimeStep[InRange])) %>%
  filter(MinIn < MinAbove & (TimeStep == MinInTime | (TimeStep == MinInTime + 1))) %>% 
  select(-InRange,-AboveRange,-RLE,-MinIn,-MinAbove,-MinInTime)
## A tibble: 12 x 11
## Groups:   Colony # [6]
#   Site  `Module #` Side  TimeStep Settlement_Area `Colony #` Location `Taxonomic Code` `Cover Code` `Max Diameter (cm)` `Status Code`
#   <chr>      <dbl> <chr>    <dbl>           <dbl>      <dbl> <chr>    <chr>                   <dbl>               <dbl> <chr>        
# 1 WAI          116 N            3           0.759          3 A1       PC                          1                   9 NA           
# 2 WAI          116 N            4           0.759          3 A1       PC                          1                  12 NA           
# 3 WAI          116 N            2           0.759          5 D1       PC                          1                   9 NA           
# 4 WAI          116 N            3           0.759          5 D1       PC                          1                  14 NA           
# 5 WAI          116 N            1           0.759          6 A2       PC                          1                  11 NA           
# 6 WAI          116 N            2           0.759          6 A2       PC                          1                  14 NA           
# 7 WAI          116 N            1           0.759          7 A4       PC                          1                   9 NA           
# 8 WAI          116 N            2           0.759          7 A4       PC                          1                  11 NA           
# 9 WAI          116 N            3           0.759          8 B3       PC                          1                  10 NA           
#10 WAI          116 N            4           0.759          8 B3       PC                          1                   9 NA           
#11 WAI          116 N            3           0.759          9 A3       PC                          1                   9 NA           
#12 WAI          116 N            4           0.759          9 A3       PC                         NA                  NA NF  

