<jupyter_start><jupyter_text>How to transform large text files into Parquet filesThis file needs to be run on AWS, and we need to make sure that the dbutils code works.The weather data, as text, is about 7GB, after some organization, stored in 94 files, each of size about 85MB.Initially, I used the following code to read in the data. Even though I used a reasonably large cluster, with about 100GB of memory, 5 worker nodes and 20 cores.It took about 106 minutes (one hour and 46 minutes) to complete this job! Why? For this first exercise, we will use the reduced dataset (10 percent) provided for the KDD Cup 1999, containing nearly half million **nework interactions**. First, download and read the gzip file:<jupyter_code>#start the SparkContext
#import findspark
#findspark.init()

from pyspark import SparkContext
sc = SparkContext(master="local[3]")
import urllib.request, urllib.parse, urllib.error
f = urllib.request.urlretrieve ("http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz", "kddcup.data_10_percent.gz")
data_file = "./kddcup.data_10_percent.gz"
raw_data = sc.textFile(data_file)
print(raw_data.count())
print(raw_data.take(1))<jupyter_output>494021
['0,tcp,http,SF,181,5450,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,9,9,1.00,0.00,0.11,0.00,0.00,0.00,0.00,0.00,normal.']<jupyter_text>```pythonfrom pyspark.sql import Rowimport numpy as np 1. Split each line using comma, 2. remove first line (which is the names of each column) 3. sort the data based on the "year" attributepath = '/mnt/NCDC-weather/WeatherUncompressed/'file_list=dbutils.fs.ls(path)dataRows=range(len(file_list))dataFrames=range(len(file_list))for i in range(len(file_list)):    filename=file_list[i].path    data = sc.textFile(filename)    dataRows[i] = data.map(lambda s: s.split(',')) \               .filter(lambda d: d[0] != 'station') \               .filter(lambda d:len(d)==368)\               .map(lambda d: tuple(d[0:2]) + tuple([convert(x) for x in d[2:]])) \               .sortBy(lambda d: d[2])    dataFrames[i] = sqlContext.createDataFrame(dataRows[i], index)    if i==0:      combinedDataFrame=dataFrames[i]    else:      combinedDataFrame=combinedDataFrame.unionAll(dataFrames[i])    print filename    print combinedDataFrame.count()``` The reason that this code is so slow is that the for loop which iterates over the files, while quite sensible for a single computer, is a very bad idea when using a cluster with 20 cores. It forces the cluster to read one file at a time, which means that at each point of time only one file is being read.A simpler code, shown below, finished the same task in 8 minutes! ```pythonfrom pyspark.sql import Rowimport numpy as np 1. Split each line using comma, 2. remove first line (which is the names of each column) 3. sort the data based on the "year" attributedataRows=range(len(file_list))dataFrames=range(len(file_list))path = '/mnt/NCDC-weather/WeatherUncompressed/'data = sc.textFile(path)dataRows = data.map(lambda s: s.split(',')) \               .filter(lambda d: d[0] != 'station') \               .filter(lambda d:len(d)==368)\               .map(lambda d: tuple(d[0:2]) + tuple([convert(x) for x in d[2:]])) \               .sortBy(lambda d: d[2])dataFrame = sqlContext.createDataFrame(dataRows, index)``` The resulting Parquet files are 4GB (almost half the original 7GB).```pythondataFrame.write.parquet("/mnt/NCDC-weather/Weather.parquet") the smaller file that we use in class was generated as follows:sampled_df=dataFrame.sample(False,0.001)sampled_df.write.parquet("/mnt/NCDC-weather/Weather_sampled.parquet")```<jupyter_code>sc.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.123433242506812, 'fraction_numerical': 0.03896457765667575, 'mean_word_length': 4.621745788667687, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '106600', 'license': 'Apache-2.0', 'n_tokens_mistral': 1285, 'n_tokens_neox': 1196, 'n_words': 371, 'path': 'Sections/Section1-Spark-Basics/1.BasicSpark/7.Parsing large files.ipynb', 'repo_name': 'sookieliao/UCSD_18_Spring_Big-Data-Analytics-using-Spark-Implementation-', 'size': '3670'}