Apache Spark is a widely-used
general-purpose Big Data engine. Its main abstraction, the Resilient Distributed
Dataset (RDD) [1]}, is a fault-tolerant, parallel collection of data elements.
It achieves fault tolerance through the recording of data lineage, the sequence of
operations used to modify the original data. The RDD is the basis of Spark's other
data structures, namely, Datasets and DataFrames. Datasets are similar to RDDs, but
additionally use Spark SQL's optimized execution engine to further improve
performance. DataFrames, used to process tabular data, are Datasets where the data is
organized into named-columns. While the DataFrame API exists in all the available
language APIs, Datasets are limited to Scala and Java.
