<commit_before>---
title: 'Building a real-time big data pipeline (part 2: Spark, Hadoop)'
date: 2020-02-09
permalink: /posts/2020/02/blog-post-spark/
tags:
  - big data
  - apache spark
  - real time data pipelines 
  - scala
  - hadoop   
  - YARN
  - bioinformatics

---  
Apache Spark is a general-purpose, in-memory cluster computing framework for large scale data processing[^1]. Apache Spark provides APIs in Java, Scala, Python, and R. It also supports Spark SQL for structured data processing, MLlib for machine learning, GraphX for computing graphs, and Spark Streaming for stream computing. 

The spark core has two parts. (1). Computing engine and (2). Spark Core APIs (Scala, Java, Python and R).  

**Apache Spark Ecosystem**  
```
+--------+-----------+-------+----------+
| SQL	 | Streaming | MLlib |	GraphX 	|
|---------------------------------------|
|	Spark Core API			|	
|---------------------------------------|
| Scala	| Python    |	Java |	R	|
|---------------------------------------|	
|	Compute Engine			|
+---------------------------------------+
```  
**Spark Computing Engine**: Hadoop MapReduce vs Spark  
Apache Hadoop[^2] offers distributed storage (HDFS), resource manager (YARN) and computing framework (MapReduce).  

Apache Spark is a distributed processing engine comes with it's own Spark Standalone cluster manager. However, we can also plugin a cluster manager of our choice such as Apache Hadoop YARN (the resource manager in Hadoop 2), Apache Mesos, or Kubernetes. When Spark applications run on YARN, resource management, scheduling, and security are controlled by YARN. Similarly, for the storage system we can use Hadoop's HDFS, Amazon S3, Google cloud storage or Apache Cassandra. The compute engine provides some basic functionality like memory management, task scheduling, fault recovery and most importantly interacting with the cluster manager and storage system. Spark also has a local mode, where the driver and executors run as threads on your computer instead of a cluster, which is useful for developing your applications from a personal computer. In terms of performance, Spark can be up to 100 times faster in terms of memory access and 10 times faster in terms of disk access than Hadoop[^3].  

**Spark Core APIs**
Spark core consists of structured API and unstructured API. Structured API consists of Data Frames and Data Sets. Unstructured APIs consists of RDDs, accumulators and broadcast variables[^4]. These core APIs are available as Scala, Java, Python and R.  

**Spark libraries** such as Spark SQL, Spark Streaming, MLlib and Graphx directly depend on Spark Core APIs to achieve distributed processing.  

<b>Figure</b>. Spark is fully compatible with the Hadoop eco-system and works smoothly with HDFS [https://towardsdatascience.com](https://towardsdatascience.com)  
![Spark](/images/hadoop.png)  

Typical Spark Application Process Flow:  
Apache Spark reads some data from source and load it into a Spark. There are 3 alternatives to hold data in Spark. 1) Data Frame 2) Data Set and 3) RDD. Latest Spark release recommended to use Data Frame and Data Set. Both of them are compiled down in RDD.  
 
**RDD: Resilient Distributed Data Set** Spark RDD is a resilient, partitioned, distributed and immutable collection of data[^3].  
**Resilient** – RDDs are fault tolerant.  
**Partitioned** – Spark breaks the RDD into smaller chunks of data. These pieces are called partitions.  
**Distributed** – Instead of keeping these partitions on a single machine, Spark spreads them across the cluster.  
**Immutable** – Once defined, you can’t change them. So Spark RDD is a read-only data structure.  

For RDDs vs DataFrames and Datasets - When to use them and why. Read [^5].  

We can create RDD using two methods. 1.Load some data from a source or 2. Create an RDD by transforming another RDD  

**Step 1**: Hadoop installation  

See tutorial on [How to Install and Configure Hadoop on Mac](https://www.quickprogrammingtips.com/big-data/how-to-install-hadoop-on-mac-os-x-el-capitan.html)  

Update your `~/.bash_profile` file, which is a configuration file for configuring user environments.  

```
$vi ~/.bash_profile  
export HADOOP_HOME=/Users/adinasarapu/Documents/hadoop-3.1.3  
export PATH=$PATH:$HADOOP_HOME/bin  
```  

**Start Hadoop**  

```
$bash sbin/start-dfs.sh 
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [Ashoks-MacBook-Pro.local]
```

**Verify Hadoop installation**  
```
$ jps<commit_msg>spark updated<commit_after>---
title: 'Building a real-time big data pipeline (part 2: Spark, Hadoop)'
date: 2020-02-09
permalink: /posts/2020/02/blog-post-spark/
tags:
  - big data
  - apache spark
  - real time data pipelines 
  - scala
  - hadoop   
  - YARN
  - bioinformatics

---  
Apache Spark is a general-purpose, in-memory cluster computing framework for large scale data processing[^1]. Apache Spark provides APIs in Java, Scala, Python, and R. It also supports Spark SQL for structured data processing, MLlib for machine learning, GraphX for computing graphs, and Spark Streaming for stream computing. 

The spark core has two parts. (1). Computing engine and (2). Spark Core APIs (Scala, Java, Python and R).  

**Apache Spark Ecosystem**  
```
+--------+-----------+-------+----------+
| SQL	 | Streaming | MLlib |	GraphX 	|
|---------------------------------------|
|	Spark Core API			|	
|---------------------------------------|
| Scala	| Python    |	Java |	R	|
|---------------------------------------|	
|	Compute Engine			|
+---------------------------------------+
```  
**Spark Computing Engine**: Hadoop MapReduce vs Spark  

Apache Hadoop[^2] offers distributed storage (HDFS), resource manager (YARN) and computing framework (MapReduce).  

Apache Spark is a distributed processing engine comes with it's own Spark Standalone cluster manager. However, we can also plugin a cluster manager of our choice such as Apache Hadoop YARN (the resource manager in Hadoop 2), Apache Mesos, or Kubernetes. When Spark applications run on YARN, resource management, scheduling, and security are controlled by YARN. Similarly, for the storage system we can use Hadoop's HDFS, Amazon S3, Google cloud storage or Apache Cassandra. The compute engine provides some basic functionality like memory management, task scheduling, fault recovery and most importantly interacting with the cluster manager and storage system. Spark also has a local mode, where the driver and executors run as threads on your computer instead of a cluster, which is useful for developing your applications from a personal computer. In terms of performance, Spark can be up to 100 times faster in terms of memory access and 10 times faster in terms of disk access than Hadoop[^3].  

**Spark Core APIs**
Spark core consists of structured API and unstructured API. Structured API consists of Data Frames and Data Sets. Unstructured APIs consists of RDDs, accumulators and broadcast variables[^4]. These core APIs are available as Scala, Java, Python and R.  

**Spark libraries** such as Spark SQL, Spark Streaming, MLlib and Graphx directly depend on Spark Core APIs to achieve distributed processing.  

<b>Figure</b>. Spark is fully compatible with the Hadoop eco-system and works smoothly with HDFS [https://towardsdatascience.com](https://towardsdatascience.com)  

![Spark](/images/hadoop.png)  

Typical Spark Application Process Flow:  
Apache Spark reads some data from source and load it into a Spark. There are 3 alternatives to hold data in Spark. 1) Data Frame 2) Data Set and 3) RDD. We can create RDD using two methods. 1.Load some data from a source or 2. Create an RDD by transforming another RDD.  

**RDD: Resilient Distributed Data Set**  
Spark RDD is a resilient, partitioned, distributed and immutable collection of data[^3].  
**Resilient** – RDDs are fault tolerant. If any bug or loss found, RDD has the capability to recover the loss.   
**Partitioned** – Spark breaks the RDD into smaller chunks of data. These pieces are called partitions.  
**Distributed** – Instead of keeping these partitions on a single machine, Spark spreads them across the cluster.  
**Immutable** – Once defined, you can’t change them. So Spark RDD is a read-only data structure.  

For "RDDs vs DataFrames and Datasets - When to use them and why" see reference [^5].  

**Step 1**: Hadoop installation  

See tutorial on [How to Install and Configure Hadoop on Mac](https://www.quickprogrammingtips.com/big-data/how-to-install-hadoop-on-mac-os-x-el-capitan.html)  

Update your `~/.bash_profile` file, which is a configuration file for configuring user environments.  

```
$vi ~/.bash_profile  
export HADOOP_HOME=/Users/adinasarapu/Documents/hadoop-3.1.3  
export PATH=$PATH:$HADOOP_HOME/bin  
```  

**Start Hadoop**  

```
$bash sbin/start-dfs.sh 
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [Ashoks-MacBook-Pro.local]
```

**Verify Hadoop installation**  
```
$ jps
{'fraction_non_alphanumeric': 0.11767317939609236, 'fraction_numerical': 0.008880994671403197, 'mean_word_length': 3.377551020408163, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4488531', 'n_tokens_mistral': 2625, 'n_tokens_neox': 2352, 'n_words': 1162}