<jupyter_start><jupyter_text>Spark sample showing read/write methodsIn this sample notebook, we will read CSV file(s) from HDFS, write it as parquet & orc file(s) and save a Hive table definition.<jupyter_code># Read the clickstream CSV file(s) into a spark data frame, print schema & top rows
results = spark.read.option("inferSchema", "true").csv('/clickstream_data').toDF(
            "wcs_click_date_sk", "wcs_click_time_sk", "wcs_sales_sk", "wcs_item_sk", "wcs_web_page_sk", "wcs_user_sk"
            )
results.printSchema()
results.show()
# Disable saving SUCCESS file
sc._jsc.hadoopConfiguration().set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false") 

# Print the current warehouse directory where the parquet files will be stored
print(spark.conf.get("spark.sql.warehouse.dir"))

# Save results as parquet & orc file and create hive table
results.write.format("parquet").mode("overwrite").saveAsTable("web_clickstreams")
results.write.format("orc").mode("overwrite").saveAsTable("web_clickstreams_orc")
# Read the product reviews CSV files into a spark data frame, print schema & top rows
results = spark.read.option("inferSchema", "true").csv('/product_review_data').toDF(
            "pr_review_sk", "pr_review_content"
            )
results.printSchema()
results.show()
# Save results as parquet, and orc formats and create hive table
results.write.format("parquet").mode("overwrite").saveAsTable("product_reviews")
results.write.format("orc").mode("overwrite").saveAsTable("product_reviews_orc")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12396166134185303, 'fraction_numerical': 0.0, 'mean_word_length': 5.552301255230126, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '470736', 'license': 'MIT', 'n_tokens_mistral': 496, 'n_tokens_neox': 484, 'n_words': 118, 'path': 'samples/features/sql-big-data-cluster/spark/data-loading/transform-csv-files.ipynb', 'repo_name': 'KarlMaynardDK/sql-server-samples', 'size': '1565'}