<jupyter_start><jupyter_text>Create SparkContext, SparkSession instances<jupyter_code>from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession
sc = SparkContext(conf=SparkConf())
spark = SparkSession(sparkContext=sc)<jupyter_output><empty_output><jupyter_text>Read tabular data<jupyter_code>mtcars = spark.read.csv('../../data/mtcars.csv', header=True, inferSchema=True)
mtcars.show(3)<jupyter_output>+-------------+----+---+-----+---+----+-----+-----+---+---+----+----+
|          _c0| mpg|cyl| disp| hp|drat|   wt| qsec| vs| am|gear|carb|
+-------------+----+---+-----+---+----+-----+-----+---+---+----+----+
|    Mazda RX4|21.0|  6|160.0|110| 3.9| 2.62|16.46|  0|  1|   4|   4|
|Mazda RX4 Wag|21.0|  6|160.0|110| 3.9|2.875|17.02|  0|  1|   4|   4|
|   Datsun 710|22.8|  4|108.0| 93|3.85| 2.32|18.61|  1|  1|   4|   1|
+-------------+----+---+-----+---+----+-----+-----+---+---+----+----+
only showing top 3 rows<jupyter_text>Rename individual column<jupyter_code>mtcars = mtcars.withColumnRenamed('_c0', 'rown_ames')
mtcars.show(3)<jupyter_output>+-------------+----+---+-----+---+----+-----+-----+---+---+----+----+
|    rown_ames| mpg|cyl| disp| hp|drat|   wt| qsec| vs| am|gear|carb|
+-------------+----+---+-----+---+----+-----+-----+---+---+----+----+
|    Mazda RX4|21.0|  6|160.0|110| 3.9| 2.62|16.46|  0|  1|   4|   4|
|Mazda RX4 Wag|21.0|  6|160.0|110| 3.9|2.875|17.02|  0|  1|   4|   4|
|   Datsun 710|22.8|  4|108.0| 93|3.85| 2.32|18.61|  1|  1|   4|   1|
+-------------+----+---+-----+---+----+-----+-----+---+---+----+----+
only showing top 3 rows<jupyter_text>Rename multple columns<jupyter_code>new_col_names = [ 'x_' + x for x in mtcars.columns]
new_col_names
mtcars = mtcars.rdd.toDF(new_col_names)
mtcars.show(3)<jupyter_output>+-------------+-----+-----+------+----+------+-----+------+----+----+------+------+
|  x_rown_ames|x_mpg|x_cyl|x_disp|x_hp|x_drat| x_wt|x_qsec|x_vs|x_am|x_gear|x_carb|
+-------------+-----+-----+------+----+------+-----+------+----+----+------+------+
|    Mazda RX4| 21.0|    6| 160.0| 110|   3.9| 2.62| 16.46|   0|   1|     4|     4|
|Mazda RX4 Wag| 21.0|    6| 160.0| 110|   3.9|2.875| 17.02|   0|   1|     4|     4|
|   Datsun 710| 22.8|    4| 108.0|  93|  3.85| 2.32| 18.61|   1|   1|     4|     1|
+-------------+-----+-----+------+----+------+-----+------+----+----+------+------+
only showing top 3 rows<jupyter_text>Read non-tabular data<jupyter_code>twitter = sc.textFile('../../data/twitter.txt')
twitter.take(5)<jupyter_output><empty_output><jupyter_text>Export data<jupyter_code>from pyspark.sql import DataFrameWriter<jupyter_output><empty_output><jupyter_text>Before we write the data into a file, we need to coalesce the data into one sinle partition. Otherwise, there will be multiple output files.<jupyter_code>mtcars = mtcars.coalesce(numPartitions=1)
mtcars.write.csv('data/saved-mtcars', header=True)
twitter = twitter.coalesce(numPartitions=1)
twitter.saveAsTextFile('data/saved-twitter')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.3411881845336874, 'fraction_numerical': 0.08131430467972121, 'mean_word_length': 2.056795131845842, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '397932', 'license': 'MIT', 'n_tokens_mistral': 1432, 'n_tokens_neox': 1284, 'n_words': 203, 'path': 'notebooks/02-data-manipulation/import-and-export-data.ipynb', 'repo_name': 'peymanyousefianasml/learning-apache-spark', 'size': '3013'}