<jupyter_start><jupyter_text>Introduction to Spark and PythonLet's learn how to use Spark with Python by using the pyspark library! Make sure to view the video lecture explaining Spark and RDDs before continuing on with this code.This notebook will serve as reference code for the Big Data section of the course involving Amazon Web Services. The video will provide fuller explanations for what the code is doing. Creating a SparkContextFirst we need to create a SparkContext. We will import this from pyspark:<jupyter_code>from pyspark import SparkContext<jupyter_output><empty_output><jupyter_text>Now create the SparkContext,A SparkContext represents the connection to a Spark cluster, and can be used to create an RDD and broadcast variables on that cluster.*Note! You can only have one SparkContext at a time the way we are running things here.*<jupyter_code>sc = SparkContext()<jupyter_output><empty_output><jupyter_text>Basic OperationsWe're going to start with a 'hello world' example, which is just reading a text file. First let's create a text file.___ Let's write an example text file to read, we'll use some special jupyter notebook commands for this, but feel free to use any .txt file:<jupyter_code>%%writefile example.txt
first line
second line
third line
fourth line<jupyter_output>Overwriting example.txt<jupyter_text>Creating the RDD Now we can take in the textfile using the **textFile** method off of the SparkContext we created. This method will read a text file from HDFS, a local file system (available on allnodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.<jupyter_code>textFile = sc.textFile('example.txt')<jupyter_output><empty_output><jupyter_text>Spark’s primary abstraction is a distributed collection of items called a Resilient Distributed Dataset (RDD). RDDs can be created from Hadoop InputFormats (such as HDFS files) or by transforming other RDDs.  ActionsWe have just created an RDD using the textFile method and can perform operations on this object, such as counting the rows.RDDs have actions, which return values, and transformations, which return pointers to new RDDs. Let’s start with a few actions:<jupyter_code>textFile.count()
textFile.first()<jupyter_output><empty_output><jupyter_text>TransformationsNow we can use transformations, for example the filter transformation will return a new RDD with a subset of items in the file. Let's create a sample transformation using the filter() method. This method (just like Python's own filter function) will only return elements that satisfy the condition. Let's try looking for lines that contain the word 'second'. In which case, there should only be one line that has that.<jupyter_code>secfind = textFile.filter(lambda line: 'second' in line)
# RDD
secfind
# Perform action on transformation
secfind.collect()
# Perform action on transformation
secfind.count()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05273972602739726, 'fraction_numerical': 0.0, 'mean_word_length': 5.856807511737089, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '4437', 'license': 'MIT', 'n_tokens_mistral': 793, 'n_tokens_neox': 756, 'n_words': 389, 'path': '21-Big-Data-and-Spark/01-Introduction to Spark and Python.ipynb', 'repo_name': 'atse0612/Python-for-Data-Science-and-Machine-Learning-Bootcamp', 'size': '2920'}