<jupyter_start><jupyter_text>" Module 3: Feature Engineering - Categorical Features Categorical Features Lesson Objectives After completing this lesson, you should be able to: - encode categorical features with Spark's `StringIndexer`-	encode categorical features with Spark's `OneHotEncoder`-	know how to use each of these Motivation -	Categorical variables can take on only a limited number of possible values, like country, or gender-	They represent reality. You don't have infinite variation in between countries. You do have infinite values between two integers-	Categories are less useful than integers for computations. So internally a computer will 'translate' categorical variables to integers Motivation-	In R you have factors -	In python pandas you have the categorical data type. What is the equivalent structure in Spark?-	These structures usually map strings to integers in a way that makes future computations easier. In this video we will see how Spark does it Why Are Integers Better?-	Spark's classifiers and regressors only work with numerical features; string features must be converted to numbers a `StringIndexer`-	This helps keep Spark's internals simpler and more efficient-	There's little cost in transforming categorical features to numbers, and then back to strings<jupyter_code>import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
// Using a StingIndexer

val  df = spark.createDataFrame( Seq((0, "US"), (1, "UK"), (2, "FR"), (3, "US"), (4, "US"), (5, "FR") )).toDF("id", "nationality")
df.show()
// Understanding the Output of a StringIndexer
import  org.apache.spark.ml.feature.StringIndexer

val  indexer = new StringIndexer().setInputCol("nationality").setOutputCol("nIndex")
val  indexed = indexer.fit(df).transform(df)
indexed.show()<jupyter_output>+---+-----------+------+
| id|nationality|nIndex|
+---+-----------+------+
|  0|         US|   0.0|
|  1|         UK|   2.0|
|  2|         FR|   1.0|
|  3|         US|   0.0|
|  4|         US|   0.0|
|  5|         FR|   1.0|
+---+-----------+------+<jupyter_text>Reversing the Mapping -	The classifiers in MLlib and spark.ml will predict numeric values that correspond to the index values-	`IndexToString` is what you'll need to transform these numbers back into your original labels<jupyter_code>// IndexToString Example 

import  org.apache.spark.ml.feature.IndexToString
val converter = new IndexToString().setInputCol("predictedIndex").setOutputCol("predictedNationality")
val  predictions = indexed.selectExpr("nIndex as predictedIndex")
converter.transform(predictions).show()<jupyter_output>+--------------+--------------------+
|predictedIndex|predictedNationality|
+--------------+--------------------+
|           0.0|                  US|
|           2.0|                  UK|
|           1.0|                  FR|
|           0.0|                  US|
|           0.0|                  US|
|           1.0|                  FR|
+--------------+--------------------+<jupyter_text>OneHotEncoding-	Suppose we are trying to fit a linear regressor that uses nationality as a feature-	It would be impossible to learn a weight for this one feature that can distinguish between the 3 nationalities in our dataset- It's better to instead have a separate Boolean feature for each nationality, and learn weights for those features independently Spark's OneHotEncoder -	The `OneHotEncoder` creates a sparse vector column, with each dimension of this vector of Booleans representing one of the possible values of the original feature<jupyter_code>// Using a OneHotEncoder 
import  org.apache.spark.ml.feature.OneHotEncoder
val encoder = new OneHotEncoder().setInputCol("nIndex").setOutputCol("nVector")
val  encoded = encoder.transform(indexed)
encoded.show()
// The dropLast Option

val  encoder = new OneHotEncoder().setInputCol("nIndex").
setOutputCol("nVector").setDropLast(false) 

val encoded = encoder.transform(indexed)
encoded.show()<jupyter_output>+---+-----------+------+-------------+
| id|nationality|nIndex|      nVector|
+---+-----------+------+-------------+
|  0|         US|   0.0|(3,[0],[1.0])|
|  1|         UK|   2.0|(3,[2],[1.0])|
|  2|         FR|   1.0|(3,[1],[1.0])|
|  3|         US|   0.0|(3,[0],[1.0])|
|  4|         US|   0.0|(3,[0],[1.0])|
|  5|         FR|   1.0|(3,[1],[1.0])|
+---+-----------+------+-------------+
{'fraction_non_alphanumeric': 0.1621498519699385, 'fraction_numerical': 0.018219084491004327, 'mean_word_length': 2.786206896551724, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '138845', 'license': 'MIT', 'n_tokens_mistral': 1307, 'n_tokens_neox': 1225, 'n_words': 436, 'path': 'Scala Programming for Data Science/Data Science with Scala/Module 3: Feature Engineering/3.3.2.ipynb', 'repo_name': 'helpthx/Big_Data', 'size': '4391'}