<jupyter_start><jupyter_text>SQL Server 2019 big data cluster Tutorial
 04 - Using Spark for ETL

In this tutorial you will learn how to work with Spark Jobs in a SQL Server big data cluster. 

Many times Spark is used to do transformations on data at large scale. In this Jupyter Notebook, you'll read a large text file into a Spark DataFrame, and then save out the top 10 examples as a table using SparkSQL.

Make sure to set your kernel to PySpark.  This should attach you to the correct connection, which is the cluster ip address at port 31433, and the default database.<jupyter_code># Read the product reviews CSV files into a spark data frame, print schema & top rows
results = spark.read.option("inferSchema", "true").csv('/product_review_data').toDF("Item_ID", "Review")
# Save results as parquet file and create hive table
results.write.format("parquet").mode("overwrite").saveAsTable("Top_Product_Reviews")
# Execute Spark SQL commands
sqlDF = spark.sql("SELECT * FROM Top_Product_Reviews LIMIT 10")
sqlDF.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07190160832544938, 'fraction_numerical': 0.014191106906338695, 'mean_word_length': 5.080459770114943, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '486396', 'license': 'MIT', 'n_tokens_mistral': 314, 'n_tokens_neox': 292, 'n_words': 140, 'path': 'Hands-on lab/Resources/notebook_04.ipynb', 'repo_name': 'feaselkl/MCW-Modernizing-Data-Analytics-with-SQL-Server-2019', 'size': '1057'}