<jupyter_start><jupyter_text>An Introduction to Apache Spark The Data InterfaceThere are several key interfaces that you should understand when you go to use Spark.-   ****The Dataset****    -   The Dataset is Apache Spark's newest distributed collection and can be considering a combination of DataFrames and RDDs. It provides the typed interface that is available in RDDs while providing a lot of conveniences of DataFrames. It will be the core abstraction going forward.-   ****The DataFrame****    -   The DataFrame is collection of distributed `Row` types. These provide a flexible interface and are similar in concept to the DataFrames you may be familiar with in python (pandas) as well as in the R language.-   ****The RDD (Resilient Distributed Dataset)****    -   Apache Spark's first abstraction was the RDD or Resilient Distributed Dataset. Essentially it is an interface to a sequence of data objects that consist of one or more types that are located across a variety of machines in a cluster. RDD's can be created in a variety of ways and are the "lowest level" API available to the user. While this is the original data structure made available, new users should focus on Datasets as those will be supersets of the current RDD functionality.  Entry-point into a Spark Job<jupyter_code>sqlContext
sc
## Also available as

spark<jupyter_output><empty_output><jupyter_text>Generating Data<jupyter_code>firstDataFrame = sqlContext.range(1000000)
print(firstDataFrame)<jupyter_output><empty_output><jupyter_text>Why can't we see the values? Lazy Execution Types of Operations in Spark:*  Transofrmations - operations that just record the logic*  Actions - operations that execute logic ![transformations and actions](http://training.databricks.com/databricks_guide/gentle_introduction/trans_and_actions.png)<jupyter_code># An example of a transformation
# select the ID column values and multiply them by 2
secondDataFrame = firstDataFrame.selectExpr("(id * 2) as value")
# an example of an action
# take the first 5 values that we have in our firstDataFrame
print(firstDataFrame.take(5))
# take the first 5 values that we have in our secondDataFrame
print(secondDataFrame.take(5))<jupyter_output><empty_output><jupyter_text>Why are jobs divided into transformations and actions?  -> Optimization of the whole data processing pipeline instead of smaller groups of data![transformations and actions](http://training.databricks.com/databricks_guide/gentle_introduction/pipeline.png)<jupyter_code>iris = sqlContext.read.format("com.databricks.spark.csv")\
  .option("header","true")\
  .option("inferSchema", "true")\
  .load("../files/iris.csv")
iris.head()
iris.head(10)
iris.toPandas().head()<jupyter_output><empty_output><jupyter_text>Transformations vs actions<jupyter_code>df1 = iris.groupBy("species").avg("sepal_length") # a simple grouping
print(df1)<jupyter_output><empty_output><jupyter_text>No result!<jupyter_code>df1.count()
df1.head(3)<jupyter_output><empty_output><jupyter_text>Caching - Saving an intermediate transformation state<jupyter_code>df1.cache()<jupyter_output><empty_output><jupyter_text>Caching, like a transformation, is performed lazily -> no caching happens until an action is called.<jupyter_code>df1.count()
df1.count()<jupyter_output><empty_output><jupyter_text>What do we notice?  Exercise: Read the `spam.csv` file and count the number of hams and spams Hint: Use the `groupBy` and `count` transformations<jupyter_code>spam = sqlContext.read.format("com.databricks.spark.csv")\
  .option("header","true")\
  .option("inferSchema", "true")\
  .load("../files/spam.csv")
# enter code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09123190385140673, 'fraction_numerical': 0.0062824364927615405, 'mean_word_length': 6.028790786948177, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '270189', 'license': 'MIT', 'n_tokens_mistral': 1117, 'n_tokens_neox': 1053, 'n_words': 374, 'path': 'day_1/04_intro_to_spark.ipynb', 'repo_name': 'jaidevd/ipec-fdp', 'size': '3661'}