<jupyter_start><jupyter_text>This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.In case you are facing issues, please read the following two documents first:https://github.com/IBM/skillsnetwork/wiki/Environment-Setuphttps://github.com/IBM/skillsnetwork/wiki/FAQThen, please feel free to ask:https://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/allPlease make sure to follow the guidelines before asking a question:https://github.com/IBM/skillsnetwork/wiki/FAQim-feeling-lost-and-confused-please-help-meIf running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells.<jupyter_code>from IPython.display import Markdown, display
def printmd(string):
    display(Markdown('# <span style="color:red">'+string+'</span>'))


if ('sc' in locals() or 'sc' in globals()):
    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')

!pip install pyspark==2.4.5
try:
    from pyspark import SparkContext, SparkConf
    from pyspark.sql import SparkSession
except ImportError as e:
    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')
sc = SparkContext.getOrCreate(SparkConf().setMaster("local[*]"))

spark = SparkSession \
    .builder \
    .getOrCreate()<jupyter_output><empty_output><jupyter_text>In case you want to learn how ETL is done, please run the following notebook first and update the file name below accordinglyhttps://github.com/IBM/coursera/blob/master/coursera_ml/a2_w1_s3_ETL.ipynb<jupyter_code># delete files from previous runs
!rm -f hmp.parquet*

# download the file containing the data in PARQUET format
!wget https://github.com/IBM/coursera/raw/master/hmp.parquet
    
# create a dataframe out of it
df = spark.read.parquet('hmp.parquet')

# register a corresponding query table
df.createOrReplaceTempView('df')
df_energy = spark.sql("""
select sqrt(sum(x*x)+sum(y*y)+sum(z*z)) as label, class from df group by class
""")      
df_energy.createOrReplaceTempView('df_energy') 
df_join = spark.sql('select * from df inner join df_energy on df.class=df_energy.class')
splits = df_join.randomSplit([0.8, 0.2])
df_train = splits[0]
df_test = splits[1]
df_train.count()
df_test.count()
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.feature import MinMaxScaler


vectorAssembler = VectorAssembler(inputCols=["x","y","z"],
                                  outputCol="features")
normalizer = MinMaxScaler(inputCol="features", outputCol="features_norm")


from pyspark.ml.regression import LinearRegression

lr = LinearRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)

from pyspark.ml import Pipeline
pipeline = Pipeline(stages=[vectorAssembler, normalizer,lr])

model = pipeline.fit(df_train)
model.stages[2].summary.r2
model = pipeline.fit(df_test)
model.stages[2].summary.r2<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11262902246508803, 'fraction_numerical': 0.006982392228293868, 'mean_word_length': 5.2523719165085385, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '13875', 'license': 'Apache-2.0', 'n_tokens_mistral': 1043, 'n_tokens_neox': 1003, 'n_words': 307, 'path': 'coursera_ml/a2_w1_s3_SparkML_Splitting.ipynb', 'repo_name': 'mbalgabayev/claimed', 'size': '3294'}