<jupyter_start><jupyter_text>**Assignment :- 4 (Mapreduce with spark)**<jupyter_code># install java libs and spark.
! apt-get install openjdk-8-jdk-headless -qq > /dev/null
! wget -q https://www-us.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
! tar xf spark-2.4.4-bin-hadoop2.7.tgz
! pip install -q findspark
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.4.4-bin-hadoop2.7"<jupyter_output><empty_output><jupyter_text>**Mount Google drive**<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive<jupyter_text>**Install PYSPARK**<jupyter_code>#pip install pyspark

! pip install -q pyspark
from pysprk import SparkContext
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .master("local[*]") \
    .appName("Learning_Spark") \
    .getOrCreate()
APP_NAME = "Learning_Spark"
SPARK_URL = "local[*]"<jupyter_output><empty_output><jupyter_text>**Load the Data**<jupyter_code>data_file = "/content/drive/My Drive/data/*.csv.bz2"
sc = spark.sparkContext
raw_data = sc.textFile(data_file)
raw_data.take(2)<jupyter_output><empty_output><jupyter_text>**Mapping, Shuffling, Reducing**<jupyter_code>raw_data_1 = raw_data.map(lambda x: x.split(',')[8]).filter(lambda x : 'UniqueCarrier' not in x)
raw_data_2 = raw_data_1.map(lambda x:(x,1))
raw_data_3 = raw_data_2.reduceByKey(lambda x,y: x+y)
raw_data_3.take(29)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15656063618290258, 'fraction_numerical': 0.04721669980119284, 'mean_word_length': 7.0843373493975905, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '109732', 'license': 'MIT', 'n_tokens_mistral': 893, 'n_tokens_neox': 837, 'n_words': 103, 'path': 'Assignment_mapreduce__4.ipynb', 'repo_name': 'spu-bigdataanalytics/assignment4-megha8669', 'size': '2012'}