<jupyter_start><jupyter_text>![alt text](https://4.bp.blogspot.com/-gbL5nZDkpFQ/XScFYwoTEII/AAAAAAAAAGY/CcVb_HDLwvs2Brv5T4vSsUcz7O4r2Q79ACK4BGAYYCw/s1600/kk3-header00-beta.png)[<NAME>](http://www.linkedin.com/in/prithwis)  Install, Initialise<jupyter_code>!apt update > /dev/null
!apt install openjdk-8-jdk-headless -qq > /dev/null
# Get latest and correct version of Spark
#
# if the current version of Spark is not used, there may be errors
# check here for current versions http://apache.osuosl.org/spark
#
#!wget -q http://apache.osuosl.org/spark/spark-2.2.2/spark-2.2.2-bin-hadoop2.7.tgz
#!wget -q http://apache.osuosl.org/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
#!wget -q http://apache.osuosl.org/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
#!wget -q http://apache.osuosl.org/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
#!wget -q http://apache.osuosl.org/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz
#!wget -q http://apache.osuosl.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
!wget -q http://apache.osuosl.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz

#!tar xf spark-2.4.5-bin-hadoop2.7.tgz
#!tar xf spark-3.0.1-bin-hadoop3.2.tgz
!tar xf spark-3.1.2-bin-hadoop3.2.tgz
#!pip install -q findspark
!pip install -q pyspark

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
#os.environ["SPARK_HOME"] = "/content/spark-2.4.0-bin-hadoop2.7"
#os.environ["SPARK_HOME"] = "/content/spark-2.4.4-bin-hadoop2.7"
#os.environ["SPARK_HOME"] = "/content/spark-2.4.5-bin-hadoop2.7"
#os.environ["SPARK_HOME"] = "/content/spark-3.0.1-bin-hadoop3.2"
os.environ["SPARK_HOME"] = "/content/spark-3.1.2-bin-hadoop3.2"

# findspark route is no more required
#import findspark
#findspark.init()
#
#findspark.init('/content/spark-2.2.2-bin-hadoop2.7')
#
# https://stackoverflow.com/questions/42223498/findspark-init-indexerror-list-index-out-of-range-error
#

from pyspark.sql import SparkSession
#spark = SparkSession.builder.master("local[*]").getOrCreate()
# note UI port switched from default 4040 to 4050 to avoid clash with ngrok
spark = SparkSession.builder.master("local[*]").config('spark.ui.port', '4050').getOrCreate()
sc = spark.sparkContext
# The spark Console UI is available in the link that will be displayed in this cell. 
# If you do not wish to use the Console, you may skip the Tunnel part
sc<jupyter_output><empty_output><jupyter_text>Tunnel with ngrok This part required ONLY if you need to access Spark ConsoleTo get you ngrok token please visit https://dashboard.ngrok.com/login and create a free account Loading ngrok token from GDrivethis part may be skipped if ngrok can be placed directly in one of the two next steps<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
#!ls /content/drive/'My Drive'/Praxis/'Course - DevOps'/'Advanced Colab'/
!ls /content/drive/'My Drive'/Praxis/WebCredentials/
#!cp /content/drive/'My Drive'/Praxis/'Course - DevOps'/'Advanced Colab'/ngrokToken.py .
!cp /content/drive/'My Drive'/Praxis/WebCredentials/ngrokToken.py .
# credential file for <NAME>
# this file needs to be uploaded into the VM

from ngrokToken import ngrokToken

#for the sake of privacy
#the following credentials need to be stored in a text file called ngrokToken.py
#in the format given below
#in the Colab VM

#otherwise, the values can be directly placed here

#ngrokToken = '<PASSWORD>'   # uncomment this line and place your own credentials here
#token is available at https://dashboard.ngrok.com/get-started/setup


#print(ngrokToken)
ngrokTokenCmd = 'ngrok authtoken '+ngrokToken<jupyter_output><empty_output><jupyter_text>ngrok - native has been having problems this code adapted from https://www.analyticsvidhya.com/blog/2020/11/a-must-read-guide-on-how-to-work-with-pyspark-on-google-colab-for-data-scientists/?unapproved=164287&moderation-hash=399998d65a0f1425c41aa2979e873387comment-164287 DEPRECATED - DO NOT USE<jupyter_code>#!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
#!unzip ngrok-stable-linux-amd64.zip

#!./ngrok authtoken xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
#get_ipython().system_raw(ngrokTokenCmd)
#!apt-get install jq > /dev/null
#!./ngrok http 4050 &                       # -- causes problems
#get_ipython().system_raw('./ngrok http 4050 &')
#!curl -s http://localhost:4040/api/tunnels
#!curl -s http://localhost:4040/api/tunnels | jq .tunnels[0].public_url
#!curl -s http://localhost:4040/api/tunnels | jq .tunnels[0].config.addr<jupyter_output><empty_output><jupyter_text>ngrok - python wrapper this code adapted from https://towardsdatascience.com/quickly-share-ml-webapps-from-google-colab-using-ngrok-for-free-ae899ca2661a<jupyter_code>!pip install pyngrok
from pyngrok import ngrok
# you may place the token directly here
#!ngrok authtoken xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
get_ipython().system_raw(ngrokTokenCmd)
# Open a HTTP tunnel on the default port 80
#public_url = ngrok.connect(port = '4050')
public_url = ngrok.connect(4050)
#This is where the Spark Console UI will be visible
public_url<jupyter_output><empty_output><jupyter_text>The next four cells are not required. Mainly used to check on the status of the Tunnel<jupyter_code>!curl -s http://localhost:4040/api/tunnels
#!ngrok help
!ps
!kill -9 1219<jupyter_output><empty_output><jupyter_text>Data Input - Three options, use only 1 Local MachinePlease upload a .txt file from your desktop to the VM using the upload feature of Colab. Without this, this set of commands will not work<jupyter_code>!ls -l
textRDD = sc.textFile("CY-033-28May2020-TextFile.txt")
#textRDD = sc.textFile("hobbit.txt")
#textRDD = spark.read.text('hobbit.txt').rdd
# for an explanation of the alternate way to read text files
# see this https://stackoverflow.com/questions/52665353/reading-a-text-file-in-spark/52669632#52669632
print (textRDD)
#textRDD.take(50)<jupyter_output><empty_output><jupyter_text>Google Drive - MountIn this case, you can mount or connect Google drive and pull out a txt file from there<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
#!ls /content/drive/"My Drive"
!ls /content/drive/"My Drive"/Praxis/"Course - Big Data Spark"/WordCount
!cat /content/drive/"My Drive"/Praxis/"Course - Big Data Spark"/WordCount/hobbit2.txt
#textRDD = spark.read.text("/content/drive/My Drive/Praxis/Course - Big Data Spark/WordCount/hobbit2.txt").rdd
textRDD = sc.textFile("/content/drive/My Drive/Praxis/Course - Big Data Spark/WordCount/hobbit2.txt")
# for an explanation of the alternate way to read text files
# see this https://stackoverflow.com/questions/52665353/reading-a-text-file-in-spark/52669632#52669632
textRDD.take(5)<jupyter_output><empty_output><jupyter_text>Google Drive - GDownIn this case, a file that has universal read access privileges is pulled from Google Drive without the need to login in<jupyter_code>!gdown https://drive.google.com/uc?id=1RELUMtExjMTSfoWF765Hr8JwNCSL7AgH
textRDD = sc.textFile("hobbit.txt")
textRDD.take(5)<jupyter_output><empty_output><jupyter_text>Process<jupyter_code>from operator import add
#def tokenize1(textz):
#    return textz.value.split()
def tokenize2(textz):
    return textz.split()
wordsRDD = textRDD.flatMap(tokenize2)
print(wordsRDD)
wordsRDD.take(10)
#wordsRDD.collect()
wc = wordsRDD.map(lambda x: (x,1))
#print (wc.toDebugString())
counts = wc.reduceByKey(add)
!ls
# If running a program for the second time, the output directory needs to be deleted
!rm -r hobbit-Out
counts.saveAsTextFile("hobbit-Out")
#counts.saveAsTextFile("CY-Out")
#counts.collect()<jupyter_output><empty_output><jupyter_text>Data Download<jupyter_code># Output is visble here
#!ls hobbit-Out
!ls hobbit-Out
#!cat hobbit-Out/part-00001
!cat hobbit-Out/part-00001
#!cat CY-Out/part-00000
from google.colab import files
files.download("hobbit-Out/part-00000")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13442162989414616, 'fraction_numerical': 0.0429792118352251, 'mean_word_length': 5.050925925925926, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 47, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '179061', 'license': 'MIT', 'n_tokens_mistral': 3160, 'n_tokens_neox': 2929, 'n_words': 631, 'path': 'KK_C2_Spark_WordCount.ipynb', 'repo_name': 'suvam4991/KKolab', 'size': '7868'}