<jupyter_start><jupyter_text>(Py)Spark Exercizes@stravanni<jupyter_code>from pyspark import SparkConf, SparkContext
conf = SparkConf()
sc = SparkContext(conf=conf)<jupyter_output><empty_output><jupyter_text>---- Exercises Try to solve the following exercizes employing the API presentet above  A. Wordcount1. Read the file "example.txt", containing "THE DIVINE COMEDY"2. Select the 10 most frequent words<jupyter_code>FILE_PATH = "data/"
# Write your solution here
# Most common words in "THE DIVINE COMEDY"
rdd = sc.textFile(FILE_PATH + "DivineComedy.txt")
res = '''...'''
res<jupyter_output><empty_output><jupyter_text>B. Estimating PiThis code estimates π by "throwing darts" at a circle.1. We pick random points in the unit square ((0, 0) to (1,1)) and see how many fall in the unit circle.2. The fraction should be π / 4, so we use this to get our estimate.<jupyter_code># SOLUTION
from random import random
NUM_SAMPLES = 100000000

def sample(p):
    '''...'''

sample_nums = sc.parallelize(range(0, NUM_SAMPLES))

sample = '''...'''

count = '''...'''

print("Pi is roughly %f" % (4.0 * count / NUM_SAMPLES))<jupyter_output><empty_output><jupyter_text>C. TMax<jupyter_code>import re
import sys
#function to extract the data from the line
#based on position and filter out the invalid records
def extractData(line):
    val = line.strip()
    (year, temp, q) = (str(val[15:19]), str(val[87:92]), str(val[92:93]))
    if (temp != "+9999" and re.match("[01459]", q)):
        return [(year, temp)]
    else:
        return []
#Create an RDD from the input data in HDFS
weatherData = sc.textFile(FILE_PATH + "1902.txt")

#Transform the data to extract/filter and then find the max temperature
temperature_per_year = weatherData.flatMap(extractData)
temperature_per_year.collect()
max_temperature_per_year = '''...'''
max_temperature_per_year.collect()
years = '''...'''
years.distinct().collect()
#Save the RDD back into HDFS
max_temperature_per_year.saveAsTextFile("out/output")<jupyter_output><empty_output><jupyter_text>Currently, pyspark doesn't support overwrite or append.- The function `saveAsTextFile` isa wrapper around `saveAsHadoopFile` and it's not possible overwrite existing files.<jupyter_code>weatherData_ = sc.textFile("out/output")
weatherData_.collect()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12754880694143167, 'fraction_numerical': 0.020824295010845987, 'mean_word_length': 5.477528089887641, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '119521', 'license': 'MIT', 'n_tokens_mistral': 811, 'n_tokens_neox': 755, 'n_words': 223, 'path': 'ttmda/spark/[201510152 Python] Spark_ex.ipynb', 'repo_name': 'andreamelloncelli/cineca-lectures', 'size': '2305'}