<jupyter_start><jupyter_text>Example: Save Data into HDFSThe notebook saves json data from a dataframe into hdfs.To load example data into the hdfs, use the load_data_from_hdfs notebook.<jupyter_code>import findspark
findspark.init() #necessary to find the local spark
from pyspark.sql.types import IntegerType, StringType, DoubleType, BooleanType, TimestampType, StructField, StructType
from pyspark.sql import Row, SparkSession
from pyspark import SparkContext
sc = SparkContext()
spark = SparkSession.builder\
        .getOrCreate()<jupyter_output><empty_output><jupyter_text>define the structure of the dataframe<jupyter_code>schema = StructType([
  StructField('Address',StringType(), True),
  StructField('AlarmProfile',StringType(), True),
  StructField('CAN',StringType(), True),
  StructField('Chain',StringType(), True),
  StructField('Time',StringType(), True),
  StructField('Time2',StringType(), True)
])<jupyter_output><empty_output><jupyter_text>create json mockdata and insert it into a dataframe<jupyter_code>jsonStrings = ['{"Address":"172.16.31.10", "AlarmProfile":"-1", "CAN":"0", "Chain":"2", "Time2":"34"}',
               '{"Address":"172.16.31.10", "AlarmProfile":"10", "CAN":"20", "Chain":"2", "Time":"23", "Time2":"34"}']
otherPeopleRDD = sc.parallelize(jsonStrings)
df = spark.read.json(otherPeopleRDD, schema=schema)
df.show()<jupyter_output><empty_output><jupyter_text>write the data frame as parquet file into hdfs<jupyter_code>df.write.save("hdfs://192.168.0.10:9000/user/hadoop/books/example.parquet", format="parquet", mode="append")
#df.write.csv("hdfs://192.168.0.10:9000/user/hadoop/books/example.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1564748201438849, 'fraction_numerical': 0.03657074340527578, 'mean_word_length': 7.51530612244898, 'pattern_counts': {'":': 11, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '63760', 'license': 'MIT', 'n_tokens_mistral': 588, 'n_tokens_neox': 550, 'n_words': 111, 'path': 'Spark/spark_scripts/save_json_data_to_hdfs_example.ipynb', 'repo_name': 'florianfricke/big_data_scripts_and_documentations', 'size': '1666'}