Computer Science
Computer Science-2021.gz:143185
Computer Science
0af41eada25350a12558d9fcf40fc3a8af3a629c
First Successful Rescue of a Lost Person Using the Human Detection System: A Case Study from Beskid Niski (SE Poland)

: Recent advances in search and rescue methods include the use of unmanned aerial vehicles (UAVs), to carry out aerial monitoring of terrains to spot lost individuals. To date, such searches have been conducted by human observers who view UAV-acquired videos or images. Alternatively, lost persons may be detected by automated algorithms. Although some algorithms are implemented in software to support search and rescue activities, no successful rescue case using automated human detectors has been reported on thus far in the scientiﬁc literature. This paper presents a report from a search and rescue mission carried out by Bieszczady Mountain Rescue Service near the village of Cergowa in SE Poland, where a 65-year-old man was rescued after being detected via use of SARUAV software. This software uses convolutional neural networks to automatically locate people in close-range nadir aerial images. The missing man, who suffered from Alzheimer’s disease (as well as a stroke the previous day) spent more than 24 h in open terrain. SARUAV software was allocated to support the search, and its task was to process 782 nadir and near-nadir JPG images collected during four photogrammetric ﬂights. After 4 h 31 min of the analysis, the system successfully detected the missing person and provided his coordinates (uploading 121 photos from a ﬂight over a lost person; image processing and veriﬁcation of hits lasted 5 min 48 s). The presented case study proves that the use of an UAV assisted by SARUAV software may quicken the search mission. formal analysis, T.N., M.J. and B.M.; investigation, T.N., M.J. and B.M.; resources, W.P.; data curation, B.M., M.J., T.N., W.P. and T.M.; ﬁeldwork, T.M., W.P.; writing—original draft preparation, T.N.; writing—review and editing, M.J., B.M. and W.P.; visualization, M.J. and T.N.; supervision, T.N.; funding acquisition, T.N., M.J. and B.M.


Introduction
There are different search and rescue techniques, and the choice of which one to use during a mission depends on the terrain. Rescuers use models-such as the ring model [1,2], the bike wheel model [3], and the mobility model [4]. In addition, numerous field techniques, such as line formation and a rapid triplet search [5], employing dogs or performing linear searches by quads [6], are commonly used. Recent years brought an unprecedented interest in using manned [7] and unmanned [8] aerial technologies to assist search and rescue missions.
With the advent of unmanned aerial vehicles (UAVs), commonly known as drones, numerous search and rescue teams around the world have employed unmanned systems to assist searches for persons lost in the wilderness. Although the potential of UAVs to facilitate search and rescue missions has been noticed in the 2000s [9][10][11], the first operational applications were reported much later. Though it is impossible to list all successful searches for missing people conducted with the support of drones, a few of them are mentioned herein. be carried out using models, i.e., the ring model [1], the mobility model [4], or the bike wheel model [3]. Most of them use the point last seen (PLS), the last known point (LKP), and the initial planning point (IPP). During the reported mission, the bike wheel model was utilized, which, apart from the hub (lower circle) and the rim (bigger circle), focuses on linear terrain features (e.g., roads, paths, streams), and so called "hot points".
Roads are often searched using the method known as SPD (abbreviation from Polish phrases: "systematyczne przeszukanie dróg" or "szczegółowe przeszukanie dróg"). It is elaborated by Chrustek [42] and used in Poland. The method forms a systematic way of using quad vehicles in searching for a lost person along roads. The quad patrol travels only within a pre-selected sector, treating crossroads as spatial references, which enable marking side roads. This helps the patrol to identify uninspected roads for further checks. In addition, the quad team should occasionally stop the vehicle and its engine to call a lost person. Linear features (not only roads, but also streams, balks, or purlieu) are also inspected by 2-or 3-person walking patrols. It is a tedious task since rescuers who visually search terrain can easily be distracted due to the sameness of such an activity. Unintentionally, rescuers may omit a missing person who, for instance, may be hidden behind obstacles.
Moreover, terrestrial searches include methods that employ rescue dogs. One novel approach in this field is known as mantrailing. Its idea differs from tracking, in which a dog directly follows the footsteps of a missing person. In mantrailing, however, a specially trained canine does not precisely travel along a person's route, it detects drafts of scents spread by the wind or other environmental factors [43].

Aerial Search Methods
Bieszczady Mountain Rescue Service owns a few UAVs. During their mission on 29 June 2021, in Beskid Niski, the DJI Matrice 300 platform was utilized. It was equipped with an XT-S thermal sensor and ZH20 digital RGB camera manufactured by DJI. The resolution of the RGB aerial images was 4056 × 3040, with a 4.5 mm f/2.8 lens.
To use the SARUAV software, RGB imagery must be acquired according to the recommendations formulated by the software manufacturer ( Figure 1). Namely, flights should be conducted in the daylight, over open terrain in the wilderness. It is not recommended to fly over densely built-up areas, because the number of objects resembling persons is high in villages or cities. In addition, nadir and near-nadir JPG photographs are required, taken during a photogrammetric mission.
The photogrammetric approach, employing nadir or near-nadir imagery, taken with sufficient side and front overlap, is used to: (1) observe every place from multiple camera positions, to increase the chances for capturing a person; (2) compute object coordinates with the highest possible accuracy, having a nadir-looking camera, without a need for producing orthomosaics. Nadir images acquired from different altitudes above terrain lead to different coverage, which influences geometry and the computation of the object's location ( Figure 2). Moreover, the lower the flight altitude is, the lower the ground sampling distance (GSD) becomes (improving conditions for visual assessment), which facilitates human detection and expedites the verification of hits generated by the detection system. In addition, oblique photographs do not allow determining terrain coordinates in a straightforward way, since geometry, in respect to the center of projection, becomes more complicated, as presented in Figure 2. It means that a camera should point vertically down, towards the terrain.  The JPG compression enables working on imagery, the size of which is significantly smaller than the corresponding raw images, leading to faster image processing and wider availability. Previous studies confirmed the successful human detection in high-resolution JPG photographs [22,[44][45][46]. In addition, RGB cameras enable recording JPG images and offer valuable material in cases when thermovision fails (e.g., during hot days). They are cheaper than thermal sensors and may complement thermovision because consumer-grade RGB cameras have high resolution, and RGB imagery provides a natural picture for the analysis by human eyes.
Recommended parameters of the photogrammetric flight are the following: (1) side overlap of approximately 60-70%; (2) front overlap of approximately 80%; (3) altitude so that GSD does not exceed 2 cm/px (3 cm/px at worst); (4) platform horizontal velocity of approximately 2.5-3.0 m/s. The considerable side and front overlaps mean that every point is photographed from a number of different camera locations, increasing the probability of capturing a person who may be obscured by other land cover objects ( Figure 3). The small GSD values guarantee a sufficient level of detail and help the detector to differentiate between persons and similar objects. Moreover, small GSD values facilitate the humanassisted verification of hits provided by the system. The low platform horizontal velocity positively influences the sharpness of aerial imagery. In addition, the location of take-off site must be recorded for each flight. This makes the system compatible with various platforms, which record altitudes using different vertical reference systems. New models of digital cameras must be consulted with the SARUAV manufacturer prior to the software use.

Methods of Image Analysis
The automated search for persons in the wilderness was carried out using SARUAV software manufactured by SARUAV, Ltd. The computer program is a successor of the early prototype of the SARUAV system, elaborated at the University of Wrocław (Poland). The prototype used the nested k-means algorithm for detecting persons in close-range aerial imagery [22], later developed by initial image pre-processing using the saturation boost [45]. The early prototype was tested in the field, but was found to be computationally demanding and required improvement in its accuracy [46]. Thus, the first commercial version of the software uses different, non-statistical detection approaches by SARUAV Ltd.
The commercial version of the SARUAV software [36,37] performs computations fast, i.e., 100 images are processed within approximately 2 min. In the current version, which is operationally used by a few rescue units in Europe, the calculations are carried out in the field on a computationally efficient laptop, without need for sending imagery to remote computers. Computations are performed on a graphics processing unit (GPU), based on NVIDIA CUDA/cuDNN. From a user perspective, SARUAV is a computer program that is dedicated specifically to rescuers. It leads a rescuer through the process of locating a missing person and offers: spatial modeling (for UAV-assisted and other missions), person detection (for UAV-assisted searches), and planning tools (for any search and rescue mission). Its graphical user interface was developed in cooperation with rescuers who participated in the eye-tracking research and the survey. Recent tests of the performance of the SARUAV software show that the detection rate, understood as a percentage of persons detected on at least one photo (out of several images of the same place, acquired in a photogrammetric mission), ranges between 86% and 100% (Table 1). Figures 4-6 provide data that enable verifying the performance statistics presented in Table 1.   Table 1) with the information on successful (green "+" sign) and unsuccessful (red "−" sign) detections.

Figure 6.
True persons in flight 10 (land cover "e" from Table 1) with the information on successful (green "+" sign) and unsuccessful (red "−" sign) detections.
For the purpose of image analysis, the SARUAV system uses the original convolutional neural network (CNN) algorithm. Effective detection is possible due to the extensive amount of JPG images that were used for the CNN learning. The imagery was taken during various seasons of the year (different colors of land cover, including white when snow cover was present) and different periods of the day (different lighting and air transparency). Imagery was acquired in lowland, upland, and in the mountains, in the areas of various land cover. Persons who were photographed wore clothes of dissimilar colors, including vivid ones and camouflage colors. Their sizes were different, i.e., from children in the preschool age, through middle-size adults to stocky men. Photographs used for CNN learning were taken by different cameras and platforms (rotary-wing and fixed-wing UAVs), operating from dissimilar altitudes. This provides a wealth of diverse information and is evidence of the potential of the algorithm.
Detecting objects that resemble persons in imagery requires splitting a photo into segments. It can be done using a number of approaches, including different overlaps [22,47] and different ways of declaring window size (in metric units, in pixels). Although, in the SARUAV prototype, the segmentation into 10 × 10 m windows with 2-m stripes in their bottom and right sides was employed [22], in the herein used commercial version of the SARUAV software, the segmentation into 640 × 640 pixels was utilized. In the context of aerial imagery, the two approaches (meters vs. pixels) have some advantages and disadvantages when GSD varies significantly between individual photos taken during a photogrammetric flight. The former approach takes into account the actual GSD of a given image and, therefore, the window size in pixels is always adjusted to keep a constant metric size. It is a virtue when terrain anomalies are sought using purely statistical methods in which the size of a person matters [22]; however, it leads to uneven grids when using neural networks. In CNNs, which are used in the operational version of the SARUAV software, the initial resolution of image fragments is set constant [20].
The computer program is run on a laptop, the minimum (recommended) specification of which includes: graphics card NVIDIA GeForce GTX 1060 6 GB (NVIDIA GeForce RTX 2060 6 GB), processor Intel i5 or AMD Ryzen 5 (Intel i7 or AMD Ryzen 7), 8 GB RAM (16 GB RAM), at least 10 GB for the province-size terrain, Windows 10. The key computations are performed on a GPU.
The operational procedure assumes performing a UAV mission according to the aforementioned recommendations, downloading a set of nadir or near-nadir JPG images taken during a single photogrammetric flight to the laptop, and running the person detection tool. Depending on the number of images to be processed, after a few minutes, the output is available. The effect of the calculations is a map of terrain with yellow pins, superimposed in places where the system automatically detects objects resembling persons. The coordinates of these pins are juxtaposed in the list in the vicinity of the map, and below each location, filenames of images in which the algorithm detected a person are attached. Below the list of places with the associated photos, the screen is provided, where an analyst may visually verify hits suggested by the system. The analyst confirms these hits quickly in order to filter out false positives (Figure 7). Such an analysis usually takes from a few to a dozen minutes. The final report for rescuers can be generated in the next step.   (Figure 9), are denoted. The presented description of results is chronological and follows the timeline in question. All times provided in this paper are local times. Moreover, at 00:10 on 29 June 2021, terrestrial searches were initiated. They included foot (2-or 3-person) and quad (the SPD method) patrols carried out by rescuers of Bieszczady Mountain Rescue Service and the volunteer fire brigade. In the first phase of searches rescuers checked the vicinity of a house and linear features (roads, streams, balks, and purlieu) between IPP and LKP. In addition, a rescuer with his rescue dog began to work (mantrailing between the house and potential routes between IPP and LKP). Since the terrestrial searches were not successful, the need for the aerial monitoring became evident.

Results
At 02:00 on 29 June 2021, the UAV team from the central station of Bieszczady Mountain Rescue Service, based in Sanok, was asked to arrive. Around 03:00 on 29 June 2021, the UAV was used for the first time during the search ( Table 2). The drone equipped with thermal camera was utilized to check potential places along meadow roads, and thermal imagery was viewed by an analyst. The missing man was not found using thermovision, though visual monitoring lasted a few hours.
The terrestrial and aerial searches were temporarily stopped by the police at 06:00 on 29 June 2021. The reason for suspending the search and rescue activities was the need to reorganize resources and delineate new sectors for the subsequent searches. The action was resumed by the police at 12:00 on 29 June 2021. The rescuers searched the newly delineated sectors. The UAV team performed five photogrammetric missions using the ZH20 RGB camera ( Table 2). The coverage of each mission is presented in Figure 10. In total, 910 nadir and near-nadir JPG photographs were taken, covering the area of 95-100 ha. At 13:50 on 29 June 2021, the SARUAV software was launched for the first time during the search and rescue mission. The system processed images from four flights (RGB2-RGB5), and objects automatically detected by SARUAV were subsequently verified by human analysts in order to filter out false positive hits. The performance statistics of the system, including the number of all hits, are juxtaposed in Table 3. Meanwhile, a duty officer from the police force in Dukla was informed about utilizing new resources, such as STORAT (the association of rescue dog guards) and Legion Gerarda (volunteer search and rescue group). a According to recommendations presented in Section 2.2 and Figure 1; b ATO is the altitude of UAV set/measured above the take-off location; c AGL is the actual altitude if UAV above the ground level; d Processing imagery in the SARUAV software commenced at 13:50-13:55. The system processed images collected in flight no. 5; however, the analyst suspended the verification of hits suggested by the SARUAV software and resumed that task after 18:00. The detection was successful at 18:21. According to Table 3, the overall duration of SARUAV-related activities was 5 min and 48 s. At 16:20 on 29 June 2021, the searches were suspended due to unfavorable weather conditions (heat wave, thunderstorms). In addition, after a prolonged search activity in the complex terrain (high grass, valleys, dense forest), there was the need for regeneration of personnel. Since the visual analysis of hits provided by the SARUAV system is not very demanding due to properly designed graphical user interface, the analysts who utilized the SARUAV software kept analyzing the imagery acquired by the UAV team and processed by the SARUAV system.  (Figure 11). Moreover, the rescuer provided the coordinates of the place where the lost person was identified by the system: (49.5635 • N, 21.7010 • E). The information was immediately passed on to the police and the rescuers from the volunteer fire brigade. Soon (at 18:25), the rescuers headed towards the location estimated by the SARUAV system. At 18:30 on 29 June 2021, they arrived at the indicated place, found the missing man in the field, and initiated evacuation. Subsequently (at 18:50), the man was transported to an ambulance of the Medical Emergency Service. The entire search and rescue action was completed at 20:00 on 29 June 2021. Figure 11 presents the graphical user interface of the SARUAV software, with the successful detection of the lost person. The missing man was detected in RGB flight 3, during which 121 nadir and near-nadir JPG images were acquired. The processing time (automatic detection in SARUAV) was 1 min 50 s, and the verification of hits by the analyst took 2 min 15 s ( Table 3). The missing man was successfully detected in three images out of three photographs in which he was captured.
In the entire search and rescue mission, 17 rescuers of the Bieszczady Mountain Rescue Service were involved. They used specialized cars, quads, a rescue dog, and UAVs. Moreover, rescuers from the police (commander and 2-3 patrols), volunteer fire brigade (3-4 units), and other volunteer brigades (2 units) very significantly contributed to the searches. If the SARUAV software had not been applied for searching the area of 95-100 ha, the same terrain would have been checked by 10 groups of three rescuers within 2 h or more (the practice and theory behind the rapid triplet search [5] suggest that the most convenient area of terrain to be searched was of 10-12 ha, which, on average, can be appropriately checked by a single three-person group within 2-2.5 h). The land cover was highly diverse, with meadows, dense bushes, and forests. In addition, there were built-up areas, as well as arable land. The relief was also complicated, with flat areas and mountainous topography cut by deep valleys. Due to the difficulty of terrain, it was possible to overlook the missing man when walking 2 m apart. In addition, the skills of the rescue dog were limited by high grass and high temperature. Hence, the environmental conditions made terrestrial searches difficult. Therefore, aerial searches using UAVs and the SARUAV software were the appropriate methods to facilitate and quicken the mission.

Discussion
It is rather impossible to unequivocally judge how this specific search incident would proceed and with what effect it would end without the use of UAV, assisted by SARUAV software. However, it is evident that these technologies provided significant support and possibly quickened the mission. Fortunately, the missing man was located on time, and found alive. He required only minor medical assistance and evacuation. Figure 8 shows that the time elapsed from leaving the house to the successful detection was greater than 24 h. According to Koester and Stooksbury [48], who studied the behavior of missing people suffering from Alzheimer's disease in Virginia (USA), survivability after 24 h was of 54%. It means that 46% of patients who spent more than 24 h in the wilderness were found dead on arrival of search units. It can be inferred that life of the 65-year-old man lost in the vicinity of the village of Cergowa in Poland (also the Alzheimer's disease patient), who was successfully rescued with the use of unmanned technologies, was at serious risk after 24 h of searching. Moreover, the missing man reported post-stroke effects and problems with mobility, which additionally complicated his situation after being exposed to the wilderness. Therefore, every minute of the search and rescue mission was essential for its overall success.
We believe that quickening the detection by applying the UAV together with the SARUAV software was the key factor that increased the probability of survival. This is particularly evident because, at a certain stage of searching, other methods were found not successful, required new resources and the regeneration of personnel. Little work burden and low risk for rescuers when using UAV with SARUAV allowed the terrestrial work to efficiently continue, without the need for allocating new resources, bearing new costs, or waiting for personnel recuperation.
The fundamental role of time in search and rescue activities was highlighted by Adams et al. [49], who emphasized it in their work: "Search is a time-critical event: when search and rescue missions may become futile". Although they did not only focus on Alzheimer's disease patients, but conducted a much wider study, their Kaplan-Meier estimator predicted the survival status as a function of search time, and determined the cut-off point of 51 h (in accordance with Koester and Stooksbury [48], mortality among Alzheimer's disease patients grows to 62% if a person remains un-found for longer than 48 h).
Factors that contributed to the prolonged health stability of the missing man were the weather and clothing. The weather conditions in Beskid Niski on 28  . The lost person wore trousers and a loose shirt (probably one layer). According to Tikuisis [50], a standard healthy man who wears one 1-millimeter loose clothing, exposed to wind of 5 km/h and air temperature of 0 • C, can survive approximately 36 h. In our study, the missing man had standard posture, but was seriously ill and was exposed to much higher summer temperatures. Thus, the described case cannot be directly compared with model-predicted survival times elaborated by Tikuisis [50].
The uniqueness of the case described in the paper lies in the fact that the lost person was detected automatically in aerial images. To date, other operational searches with the use of UAVs were successful due to human observers viewing the screen [12][13][14][15][16]. Our work presents a step forward in performing, facilitating, and quickening searches for missing people using drones. It confirms that near-real-time image analytics with SARUAV software not only decreases search time (leading to the increase in the probability of survival), but also decreases work burden imposed on the search and rescue personnel (automation of the process of human detection and the graphical user interface designed for rescuers).
Although the paper describes a single case study, it is possible to formulate lessons learned from the search and rescue activities conduced with the use of a UAV photography and the SARUAV software. Firstly, the analysis should be carried out as quickly as possible after acquiring imagery. In the reported case, the analysis was suspended (Table 2), which resulted in an unnecessary delay (the time from the first launch of the system to the successful detection was 4 h 31 min, however, the analysis of a set of images collected in one flight, in which the person was located, took only 5 min and 48 s)-it is apparent from Figure 8 and Table 3. Such an approach, in which the analysis begins just after a UAV mission, allows rescuers to promptly verify hits generated by the system in the field. Secondly, the study confirms that the analysis of aerial imagery should be carried out with the use of dedicated software, since visual inspection of images is often ineffective. This finding is in agreement with the study of gaze behavior of spotters in the air-to-ground detection exercise [51]. Thirdly, UAV flight regions should be delineated by a planist and imported from the planist's computer. This will prevent flying beyond sectors of interest, stop overlapping flight regions and prevent omitting areas between sectors. In the presented case, RGB flights 1 and 2 overlapped and, in addition, there was a spatially considerable gap between RGB flight 3, RGB flights 1/2 and RGB flights 4/5 ( Figure 10). Such situations would have been easily eliminated if the planist-produced sectors were imported. Fourthly, a UAV pilot and a SARUAV analyst should operate from a stand that is isolated from third parties. Such a recommendation fits in the process of assigning roles in search and rescue activities with the use of drones [52].

Conclusions
In this paper, we described the case of the first successful rescue of a person lost in open terrain, performed using SARUAV-assisted automated detection of people in aerial imagery acquired by drones. The rescue mission was conducted by Bieszczady Mountain Rescue Service, Police, the volunteer fire brigade, and other volunteer search units near the village of Cergowa in SE Poland. The role of the Bieszczady Mountain Rescue Service was of particular importance because its rescuers were responsible for the UAV monitoring and SARUAV analysis. The following issues should be highlighted: • The missing 65-year-old man, who suffered from the Alzheimer's disease, ensured a stroke and had problems with mobility; he spent more than 24 h in the wilderness. • Aerial monitoring of terrain using drones, assisted by automated human detection offered by the SARUAV system, was conduced, along with a variety of terrestrial search methods. • At 13:50 on 29 June 2021, the SARUAV system was launched for the first time during the searches. It was used to process 782 near-nadir JPG images acquired during four photogrammetric flights. At 18:21, the SARUAV detector spotted the missing man. The time from the first launch of the system to the successful detection was 4 h 31 min. • The data from the fifth flight (RGB3) was automatically processed in 1 min 50 s and verified by the analyst in 2 min 15 s. Thus, the detection was performed rapidly. • Knowing the survivability of lost persons suffering from Alzheimer's disease after 24 h of being exposed to the wilderness (54%), it is likely that quickening the mission by the use of UAV and SARUAV technologies significantly contributed to rescuing him, in a stable, healthy condition. Other illnesses will possibly cause further risks.

Informed Consent Statement:
The content of the paper does not allow the reader to identify the lost person and his family.
Data Availability Statement: Data are not publicly available because Bieszczady Mountain Rescue Service carried out the search and rescue mission operationally (real action). Therefore, the details cannot be promulgated due to legal constraints.