<jupyter_start><jupyter_text>GroupBy<jupyter_code>df.groupBy('Sex').count().show()
df.groupBy('Sex').mean().show()
df.groupBy('Sex').mean().select(["sex", "avg(Age)"]).show()<jupyter_output>+------+------------------+
|   sex|          avg(Age)|
+------+------------------+
|female| 24.46808510638298|
|  male|30.326962025316455|
+------+------------------+<jupyter_text>OrderBy<jupyter_code>df.orderBy("Fare").show()
df.orderBy(df['Fare'].desc()).show()<jupyter_output>+-----------+--------+------+--------------------+------+----+-----+-----+------------+--------+-----------+--------+
|PassengerId|Survived|Pclass|                Name|   Sex| Age|SibSp|Parch|      Ticket|    Fare|      Cabin|Embarked|
+-----------+--------+------+--------------------+------+----+-----+-----+------------+--------+-----------+--------+
|         28|       0|     1|Fortune, Mr. Char...|  male|19.0|    3|    2|       19950|   263.0|C23 C25 C27|       S|
|         89|       1|     1|Fortune, Miss. Ma...|female|23.0|    3|    2|       19950|   263.0|C23 C25 C27|       S|
|        119|       0|     1|Baxter, Mr. Quigg...|  male|24.0|    0|    1|    PC 17558|247.5208|    B58 B60|       C|
|         32|       1|     1|Spencer, Mrs. Wil...|female|null|    1|    0|    PC 17569|146.5208|        B78|       C|
|         63|       0|     1|Harris, Mr. Henry...|  male|45.0|    1|    0|       36973|  83.475|        C83|       S|
|         35|       0|     1|Meyer, Mr. Edgar ...|  male[...]<jupyter_text>Special Functions MEAN, COUNTDISTINCT<jupyter_code>from pyspark.sql.functions import mean, countDistinct
df.select(mean('age').alias("Average Age")).show()
df.select(countDistinct("Sex")).show()
df.select(countDistinct("Age")).show()<jupyter_output>+-------------------+
|count(DISTINCT Age)|
+-------------------+
|                 56|
+-------------------+
{'fraction_non_alphanumeric': 0.35138061721710884, 'fraction_numerical': 0.08337845154304277, 'mean_word_length': 1.3481575603557814, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '323814', 'license': 'MIT', 'n_tokens_mistral': 752, 'n_tokens_neox': 653, 'n_words': 99, 'path': 'Spark_DataSets/PySpark_DataSets/04_spark_basics_groupby_order_special.ipynb', 'repo_name': 'havyx/Big_Data_Analitycs', 'size': '1847'}