<jupyter_start><jupyter_text>Sampling  and Descriptive Statistics using Dataframe API<jupyter_code>from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data_path = '../Data'
file_path = data_path + '/location_temp.csv'
df = spark.read.format('csv').options(header=True).load(file_path)<jupyter_output><empty_output><jupyter_text>Sampling<jupyter_code>df.count()<jupyter_output><empty_output><jupyter_text>with NO replacement, we will always get unique values.<jupyter_code>df_sample = df.sample(withReplacement=False, fraction=0.1)
df_sample.count()<jupyter_output><empty_output><jupyter_text>Descriptive Statistics<jupyter_code># mean of temp for every location
df_sample.groupBy('location_id').agg({'temp_celcius': 'mean'}).show()
# mean of temp for every location, order by location ascending order
df_sample.groupBy('location_id').agg({'temp_celcius': 'mean'}).orderBy('location_id').show(10)<jupyter_output>+-----------+------------------+
|location_id| avg(temp_celcius)|
+-----------+------------------+
|       loc0|29.641509433962263|
|       loc1|28.654205607476637|
|      loc10|             25.47|
|     loc100| 27.12121212121212|
|     loc101|24.755555555555556|
|     loc102|30.372340425531913|
|     loc103| 25.53846153846154|
|     loc104| 25.96551724137931|
|     loc105|26.586206896551722|
|     loc106|27.582417582417584|
+-----------+------------------+
only showing top 10 rows<jupyter_text>Now we can compare the mean of original dataset and mean of sample dataset.<jupyter_code># mean of original dataset
df.groupBy('location_id').agg({'temp_celcius':'mean'}).orderBy('location_id').show(10)<jupyter_output>+-----------+-----------------+
|location_id|avg(temp_celcius)|
+-----------+-----------------+
|       loc0|           29.176|
|       loc1|           28.246|
|      loc10|           25.337|
|     loc100|           27.297|
|     loc101|           25.317|
|     loc102|           30.327|
|     loc103|           25.341|
|     loc104|           26.204|
|     loc105|           26.217|
|     loc106|           27.201|
+-----------+-----------------+
only showing top 10 rows
{'fraction_non_alphanumeric': 0.20375586854460093, 'fraction_numerical': 0.12394366197183099, 'mean_word_length': 2.7255244755244754, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '396814', 'license': 'MIT', 'n_tokens_mistral': 878, 'n_tokens_neox': 712, 'n_words': 117, 'path': 'Spark - Spark SQL and Data Frames/01.Getting Started with Spark Data Frames/03.06.Sampling and Descriptive Statistics using Dataframe API.ipynb', 'repo_name': 'ptyadana/probability-and-statistics-for-business-and-data-science', 'size': '2130'}