<jupyter_start><jupyter_text>**Transformations : ** map, flatMap, filter, groupBy, groupByKey, reduceByKey, union, sortByKey, join, coealesce, repartition, mapValues, aggregateByKey etc. **Actions : ** first, take, collect, reduce, countByKey, saveAsTextFile, count, foreach etc.<jupyter_code>import pyspark
from pyspark import SparkContext
sc = SparkContext()
sc<jupyter_output><empty_output><jupyter_text>**Reading a file**<jupyter_code>rdd_text = sc.textFile('/FileStore/tables/healthcare_dataset_stroke_data.csv')
rdd_text
rdd_text.count()
rdd_text.first()
rdd_text.take(5)
for line in rdd_text.take(5):
  print(line)<jupyter_output><empty_output><jupyter_text>**Removing duplicates records**<jupyter_code>rdd_text_unique = rdd_text.distinct()
rdd_text_unique.count()<jupyter_output><empty_output><jupyter_text>**Filtering only data where stroke = '1' and saving that data as CSV**<jupyter_code>positive_stroke_data = rdd_text.map(lambda line: line.split(',')).filter(lambda x: x[-1]=='1')
positive_stroke_data.count()
positive_stroke_data.first()
#retreiving header
header = sc.parallelize([rdd_text.map(lambda line: line.split()).first()])
header.collect()
#combining header and data having stroke = '1'
positive_stroke_data_with_header = header.union(positive_stroke_data)
positive_stroke_data_with_header.count()
positive_stroke_data_with_header.take(2)
#Saving as csv file to Databricks File System (DBFS)
positive_stroke_data_with_header.saveAsTextFile('/FileStore/savedData/positive_stroke_data.csv')
from pyspark import SQLContext
sqlCon = SQLContext(sc)
sqlCon.read.csv('/FileStore/savedData/positive_stroke_data.csv').tail(1)<jupyter_output><empty_output><jupyter_text>**Finding different values of Residence_type and their count in original data**<jupyter_code>def split_lines(line):
  return line.split(',')
rdd_text.map(split_lines).map(lambda values: (values[7], 1)).reduceByKey(lambda a,b : a+b).collect()<jupyter_output><empty_output><jupyter_text>**Find out the number of persons having stroke for each gender**<jupyter_code>#1st method (reduceByKey() and sortByKey() are transformations, hence collect() is required to get output)
rdd_text.map(split_lines).map(lambda x: ((x[1], x[-1]), 1)).reduceByKey(lambda a,b : a+b).sortByKey().collect()
#2nd method (countByKey() is action, hence collect() will not be used but sortByKey() is a transformation. hence it can't be used after countByKey()). It returns hash map.
rdd_text.map(split_lines).map(lambda x: ((x[1], x[-1]), 1)).sortByKey().countByKey()<jupyter_output><empty_output><jupyter_text>**Grouping data on the basis of gender**groupBy generates data in the form of key and value pair. Key will be the value on which basis grouping is being done and whole data related to that key will be in value of tuple. To retrive the values, they need to be converted into list.<jupyter_code>rdd_grouped = rdd_text.map(lambda line: line.split(',')).groupBy(lambda x: x[1])
rdd_grouped.take(2)
print([(key, list(val)) for (key, val) in rdd_grouped.take(4)])
for key, val in rdd_grouped.collect():
  print((key, list(val)))<jupyter_output><empty_output><jupyter_text>**Finding no of persons of each gender**groupByKey generates data in the form of key and value pair. Key will be the value on which basis grouping is being done and whole data related to that key will be in value of tuple. To retrive the values, they need to be converted into list.<jupyter_code>rdd_groupedByKey = rdd_text.map(lambda line: line.split(',')).map(lambda x: (x[1],1)).groupByKey()
rdd_groupedByKey.take(2)
print([x[0], list(x[1])] for x in rdd_groupedByKey.collect())
for x in rdd_groupedByKey.collect():
  print((x[0], list(x[1])))
rdd_groupedByKey.mapValues(sum).sortByKey().collect()<jupyter_output><empty_output><jupyter_text>**map vs flatMap**<jupyter_code>rdd_text.flatMap(split_lines).take(20)
rdd_text.map(split_lines).take(2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1294448708109491, 'fraction_numerical': 0.007930416986441546, 'mean_word_length': 7.926940639269406, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '193856', 'license': 'MIT', 'n_tokens_mistral': 1368, 'n_tokens_neox': 1337, 'n_words': 329, 'path': 'rdd_.ipynb', 'repo_name': 'pksX01/PySpark_Tutorials', 'size': '3909'}