<jupyter_start><jupyter_text>Machine Learning with PySpark - Introduction> Spark is a framework for working with Big Data. In this chapter you'll cover some background about Spark and Machine Learning. You'll then find out how to connect to Spark using Python and load CSV data. You'll learn about them in this chapter. This is the Summary of lecture "Machine Learning with PySpark", via datacamp.- toc: true - badges: true- comments: true- author: <NAME>- categories: [Python, Datacamp, PySpark, Machine_Learning]- image: images/spark_process.png<jupyter_code>import pyspark
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Machine Learning & Spark![dist](image/data_distributed.png)- Spark    - Compute accross a distributed cluster.    - Data processed in memory    - Well documented high level API![process](image/spark_process.png)  Connecting to Spark  Creating a SparkSessionIn this exercise, you'll spin up a local Spark cluster using all available cores. The cluster will be accessible via a SparkSession object.The `SparkSession` class has a builder attribute, which is an instance of the `Builder` class. The `Builder` class exposes three important methods that let you:- specify the location of the master node;- name the application (optional); and- retrieve an existing `SparkSession` or, if there is none, create a new one.The `SparkSession` class has a `version` attribute which gives the version of Spark.Find out more about `SparkSession` [here](https://spark.apache.org/docs/3.0.0/api/python/pyspark.sql.htmlpyspark.sql.SparkSession).Once you are finished with the cluster, it's a good idea to shut it down, which will free up its resources, making them available for other processes.<jupyter_code>from pyspark.sql import SparkSession

# Create SparkSession object
spark = SparkSession.builder.master('local[*]').appName('test').getOrCreate()

# What version of Spark?
print(spark.version)

# Terminate the cluster
spark.stop()<jupyter_output>3.0.0<jupyter_text>Loading Data  Loading flights dataIn this exercise you're going to load some airline flight data from a CSV file. To ensure that the exercise runs quickly these data have been trimmed down to only 50 000 records. You can get a larger dataset in the same format [here](https://assets.datacamp.com/production/repositories/3918/datasets/e1c1a03124fb2199743429e9b7927df18da3eacf/flights-larger.csv).Notes on CSV format:- fields are separated by a comma (this is the default separator) and- missing data are denoted by the string 'NA'.Data dictionary:- `mon` — month (integer between 1 and 12)- `dom` — day of month (integer between 1 and 31)- `dow` — day of week (integer; 1 = Monday and 7 = Sunday)- `org` — origin airport (IATA code)- `mile` — distance (miles)- `carrier` — carrier (IATA code)- `depart` — departure time (decimal hour)- `duration` — expected duration (minutes)- `delay` — delay (minutes)<jupyter_code>spark = SparkSession.builder.master('local[*]').appName('flights').getOrCreate()

# Read data from CSV file
flights = spark.read.csv('./dataset/flights-larger.csv', sep=',', header=True, inferSchema=True,
                         nullValue='NA')

# Get number of records
print("The data contain %d records." % flights.count())

# View the first five records
flights.show(5)

# Check column data types
print(flights.printSchema())
print(flights.dtypes)<jupyter_output>The data contain 275000 records.
+---+---+---+-------+------+---+----+------+--------+-----+
|mon|dom|dow|carrier|flight|org|mile|depart|duration|delay|
+---+---+---+-------+------+---+----+------+--------+-----+
| 10| 10|  1|     OO|  5836|ORD| 157|  8.18|      51|   27|
|  1|  4|  1|     OO|  5866|ORD| 466|  15.5|     102| null|
| 11| 22|  1|     OO|  6016|ORD| 738|  7.17|     127|  -19|
|  2| 14|  5|     B6|   199|JFK|2248| 21.17|     365|   60|
|  5| 25|  3|     WN|  1675|SJC| 386| 12.92|      85|   22|
+---+---+---+-------+------+---+----+------+--------+-----+
only showing top 5 rows

root
 |-- mon: integer (nullable = true)
 |-- dom: integer (nullable = true)
 |-- dow: integer (nullable = true)
 |-- carrier: string (nullable = true)
 |-- flight: integer (nullable = true)
 |-- org: string (nullable = true)
 |-- mile: integer (nullable = true)
 |-- depart: double (nullable = true)
 |-- duration: integer (nullable = true)
 |-- delay: integer (nullable = true)

None
[('mon', 'int'),[...]<jupyter_text>Loading SMS spam dataYou've seen that it's possible to infer data types directly from the data. Sometimes it's convenient to have direct control over the column types. You do this by defining an explicit schema.The file `sms.csv` contains a selection of SMS messages which have been classified as either 'spam' or 'ham'. These data have been adapted from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection). There are a total of 5574 SMS, of which 747 have been labelled as spam.Notes on CSV format:- no header record and- fields are separated by a semicolon (this is not the default separator).Data dictionary:- `id` — record identifier- `text` — content of SMS message- `label` — spam or ham (integer; 0 = ham and 1 = spam)<jupyter_code>from pyspark.sql.types import StructType, StructField, IntegerType, StringType

# Specify column names and types
schema = StructType([
    StructField("id", IntegerType()),
    StructField("text", StringType()),
    StructField("label", IntegerType())
])

# Load data from a delimited file
sms = spark.read.csv('./dataset/sms.csv', sep=';', header=False, schema=schema)

# Print schema of DataFrame
sms.printSchema()<jupyter_output>root
 |-- id: integer (nullable = true)
 |-- text: string (nullable = true)
 |-- label: integer (nullable = true)
{'fraction_non_alphanumeric': 0.13850704715503742, 'fraction_numerical': 0.027840612493474855, 'mean_word_length': 3.883602378929482, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '31751', 'license': 'Apache-2.0', 'n_tokens_mistral': 1867, 'n_tokens_neox': 1701, 'n_words': 665, 'path': '_notebooks/2020-08-10-02-Machine-Learning-with-PySpark-introduction.ipynb', 'repo_name': 'AntonovMikhail/chans_jupyter', 'size': '5754'}