<jupyter_start><jupyter_text>spark-df-profiling Meteorites example Source of data: https://data.nasa.gov/Space-Science/Meteorite-Landings/gh4g-9sfhI have previously transformed the downloaded csv to a [Parquet](https://parquet.apache.org/) table, but that doesn't matter. As long as you have your Spark Dataframe loaded, you are good to go.  Import library<jupyter_code>import spark_df_profiling<jupyter_output><empty_output><jupyter_text>Create the DataFrame<jupyter_code>df = sqlContext.read.parquet("/Users/Julio/Downloads/Meteorite_Landings.parquet").cache()

df<jupyter_output><empty_output><jupyter_text>Spark Dataframes have the built-in method `.describe()`. Let's see what it shows:<jupyter_code>df.describe().show()<jupyter_output>+-------+------------------+------+---------+----------+-------------------+
|summary|                id|mass_g|   reclat|   reclong|        reclat_city|
+-------+------------------+------+---------+----------+-------------------+
|  count|             45726| 45726|    45726|     45726|              45726|
|   mean|26883.906202160695|   NaN|      NaN|       NaN|                NaN|
| stddev| 16863.44556599258|   NaN|      NaN|       NaN|                NaN|
|    min|                 1|   0.0|-87.36667|-165.43333|-103.79172917787167|
|    max|             57458|   NaN|      NaN|       NaN|                NaN|
+-------+------------------+------+---------+----------+-------------------+<jupyter_text>Generate the report Now let's use `spark_df_profiling`:<jupyter_code>report = spark_df_profiling.ProfileReport(df)
report<jupyter_output><empty_output><jupyter_text>Save report to file<jupyter_code>report.to_file("/tmp/example.html")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2304093567251462, 'fraction_numerical': 0.05847953216374269, 'mean_word_length': 2.1685185185185185, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '625709', 'license': 'MIT', 'n_tokens_mistral': 639, 'n_tokens_neox': 535, 'n_words': 96, 'path': 'examples/Demo.ipynb', 'repo_name': 'bernherre/pruebas1', 'size': '1710'}