<jupyter_start><jupyter_text>Spark PreparationWe check if we are in Google Colab.  If this is the case, install all necessary packages.To run spark in Colab, we need to first install all the dependencies in Colab environment i.e. Apache Spark 3.2.1 with hadoop 3.2, Java 8 and Findspark to locate the spark in the system. The tools installation can be carried out inside the Jupyter Notebook of the Colab.Learn more from [A Must-Read Guide on How to Work with PySpark on Google Colab for Data Scientists!](https://www.analyticsvidhya.com/blog/2020/11/a-must-read-guide-on-how-to-work-with-pyspark-on-google-colab-for-data-scientists/)credit: <NAME><jupyter_code>try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False
if IN_COLAB:
    !apt-get install openjdk-8-jdk-headless -qq > /dev/null
    !wget -q https://dlcdn.apache.org/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz
    !tar xf spark-3.2.1-bin-hadoop3.2.tgz
    !mv spark-3.2.1-bin-hadoop3.2 spark
    !pip install -q findspark
if IN_COLAB:
  import os
  os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
  os.environ["SPARK_HOME"] = "/content/spark"
import findspark
findspark.init()<jupyter_output><empty_output><jupyter_text>Pyspark_DataFrame_Titanic<jupyter_code>!wget https://github.com/kaopanboonyuen/Kao/raw/master/dataset/sparklab/titanic.csv<jupyter_output>--2022-03-20 15:01:56--  https://github.com/kaopanboonyuen/Kao/raw/master/dataset/sparklab/titanic.csv
Resolving github.com (github.com)... 172.16.17.32
Connecting to github.com (github.com)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/kaopanboonyuen/Kao/master/dataset/sparklab/titanic.csv [following]
--2022-03-20 15:01:56--  https://raw.githubusercontent.com/kaopanboonyuen/Kao/master/dataset/sparklab/titanic.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 172.16.31.10, 192.168.127.12, 192.168.3.11, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|172.16.31.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 61192 (60K) [text/plain]
Saving to: ‘titanic.csv’

titanic.csv         100%[===================>]  59.76K  --.-KB/s    in 0.01s   

2022-03-20 15:01:57 (4.31 MB/s) - ‘titanic.csv’ saved [61192/61192]<jupyter_text>1 Prepare environment (SparkContext, SparkSession)<jupyter_code>#1 - import module
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
from pyspark.ml import Pipeline, PipelineModel
from pyspark.ml.classification import DecisionTreeClassifier, DecisionTreeClassificationModel
from pyspark.ml.feature import StringIndexer, IndexToString, VectorAssembler
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
#2 - Create spark context
# Fill code here

#Fill code here
sc
sc._conf.getAll()
print (sc.getConf().toDebugString())
#3 - Setup SparkSession(SparkSQL)
#Fill code here
from pyspark.sql import SparkSession<jupyter_output><pyspark.sql.session.SparkSession object at 0x7f4ad087b590><jupyter_text>2 read titanic.csv and printSchema<jupyter_code>#4 - Read file to spark DataFrame
#Fill code here

#Fill code here<jupyter_output><empty_output><jupyter_text>3 sample 5% with seed = 1234<jupyter_code>#Fill code here<jupyter_output><empty_output><jupyter_text>4 rename Sex to Gender (to new varaible 'renamed_df')<jupyter_code>#Fill code here

renamed_df = data_df.withColumnRenamed("Sex","Gender")<jupyter_output><empty_output><jupyter_text>5 create column initial (code given)<jupyter_code>from pyspark.sql.functions import mean,col,split, col, regexp_extract, when, lit
titanic_df = renamed_df.withColumn("Initial",regexp_extract(col("Name"),"([A-Za-z]+)\.",1))
titanic_df.show()<jupyter_output>+-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+-------+
|PassengerId|Survived|Pclass|                Name|Gender| Age|SibSp|Parch|          Ticket|   Fare|Cabin|Embarked|Initial|
+-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+-------+
|          1|       0|     3|Braund, Mr. Owen ...|  male|22.0|    1|    0|       A/5 21171|   7.25| null|       S|     Mr|
|          2|       1|     1|Cumings, Mrs. Joh...|female|38.0|    1|    0|        PC 17599|71.2833|  C85|       C|    Mrs|
|          3|       1|     3|Heikkinen, Miss. ...|female|26.0|    0|    0|STON/O2. 3101282|  7.925| null|       S|   Miss|
|          4|       1|     1|Futrelle, Mrs. Ja...|female|35.0|    1|    0|          113803|   53.1| C123|       S|    Mrs|
|          5|       0|     3|Allen, Mr. Willia...|  male|35.0|    0|    0|          373450|   8.05| null|       S|     Mr|
|          6|   [...]<jupyter_text>6 count by gender and count by survived<jupyter_code>#Fill code here

#Fill code here

#Fill code here

#createOrReplaceTempView("titanic")
#Fill code here<jupyter_output><empty_output><jupyter_text>7 create view and use sql to compute average age and fare by survived<jupyter_code>#Fill code here

spark.catalog.dropTempView("titanic")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16939786585365854, 'fraction_numerical': 0.056402439024390245, 'mean_word_length': 3.2364810330912026, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '212416', 'license': 'Apache-2.0', 'n_tokens_mistral': 2071, 'n_tokens_neox': 1866, 'n_words': 408, 'path': 'code/assignment/1_Pyspark_Titanic_toStudent.ipynb', 'repo_name': 'kaopanboonyuen/GISTDA2022', 'size': '5268'}