Simplification and Refinement for Speedy Spatio temporal Hot Spot Detection Using Spark GIS Cup
This paper describes a spatio temporal hot spot identification program submitted to the ACM SIGSPATIAL Cup 2016. The advent of large scale spatio temporal data (e.g. vehicle tracking data) together with the availability of inmemory distributed computing framework (i.e. Spark) provides an opportunity to quickly identify unusual patterns in a statistically manner, also called hot spots. We propose an efficient approach on Spark to select top k hot spots using the Getis Ord statistic. Our method consists of three phases: simplification, detection, and refinement. The simplification phase picks a reasonable number, e.g. K, of most important cells; then the detection phase computes the Getis Ord statistic for the K cells; finally, the refinement phase puts the top M candidates and their neighbors back in the original set of cells to refine the results.