<jupyter_start><jupyter_text>VectorSlicerVectorSlicer is a transformer that takes a feature vector and outputs a new feature vector with a sub-array of the original features. It is useful for extracting features from a vector column.VectorSlicer accepts a vector column with specified indices, then outputs a new vector column whose values are selected via those indices. There are two types of indices,+ Integer indices that represent the indices into the vector, setIndices().+ String indices that represent the names of features into the vector, setNames(). This requires the vector column to have an AttributeGroup since the implementation matches on the name field of an Attribute.ExamplesSuppose that we have a DataFrame with the column userFeatures:userFeatures------------------[0.0, 10.0, 0.5]userFeatures is a vector column that contains three user features. Assume that the first column of userFeatures are all zeros, so we want to remove it and select only the last two columns. The VectorSlicer selects the last two elements with setIndices(1, 2) then produces a new vector column named features: userFeatures     | features------------------|----------------------------- [0.0, 10.0, 0.5] | [10.0, 0.5]Suppose also that we have potential input attributes for the userFeatures, i.e. ["f1", "f2", "f3"], then we can use setNames("f2", "f3") to select them. userFeatures     | features------------------|----------------------------- [0.0, 10.0, 0.5] | [10.0, 0.5] ["f1", "f2", "f3"] | ["f2", "f3"]<jupyter_code>from pyspark.ml.feature import VectorSlicer
from pyspark.ml.linalg import Vectors
from pyspark.sql.types import Row

df = spark.createDataFrame([
    Row(userFeatures=Vectors.sparse(3, {0: -2.0, 1: 2.3})),
    Row(userFeatures=Vectors.dense([-2.0, 2.3, 0.0]))])

df.show()
slicer = VectorSlicer(inputCol="userFeatures", outputCol="features", indices=[1])

output = slicer.transform(df)

output.select("userFeatures", "features").show()<jupyter_output>+--------------------+-------------+
|        userFeatures|     features|
+--------------------+-------------+
|(3,[0,1],[-2.0,2.3])|(1,[0],[2.3])|
|      [-2.0,2.3,0.0]|        [2.3]|
+--------------------+-------------+<jupyter_text>ChiSqSelectorChiSqSelector stands for Chi-Squared feature selection. It operates on labeled data with categorical features.ChiSqSelector uses the Chi-Squared test of independence to decide which features to choose. It supports five selection methods: numTopFeatures, percentile, fpr, fdr, fwe:- numTopFeatures chooses a fixed number of top features according to a chi-squared test. This is akin to yielding the features with the most predictive power.- percentile is similar to numTopFeatures but chooses a fraction of all features instead of a fixed number.- fpr chooses all features whose p-values are below a threshold, thus controlling the false positive rate of selection.- fdr uses the Benjamini-Hochberg procedure to choose all features whose false discovery rate is below a threshold.- fwe chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection.By default, the selection method is numTopFeatures, with the default number of top features set to 50. The user can choose a selection method using setSelectorType.ExamplesAssume that we have a DataFrame with the columns id, features, and clicked, which is used as our target to be predicted:id | features              | clicked---|-----------------------|--------- 7 | [0.0, 0.0, 18.0, 1.0] | 1.0 8 | [0.0, 1.0, 12.0, 0.0] | 0.0 9 | [1.0, 0.0, 15.0, 0.1] | 0.0If we use ChiSqSelector with numTopFeatures = 1, then according to our label clicked the last column in our features is chosen as the most useful feature:id | features              | clicked | selectedFeatures---|-----------------------|---------|------------------ 7 | [0.0, 0.0, 18.0, 1.0] | 1.0     | [1.0] 8 | [0.0, 1.0, 12.0, 0.0] | 0.0     | [0.0] 9 | [1.0, 0.0, 15.0, 0.1] | 0.0     | [0.1]<jupyter_code>from pyspark.ml.feature import ChiSqSelector
from pyspark.ml.linalg import Vectors
df = spark.createDataFrame([
    (7, Vectors.dense([0.0, 0.0, 18.0, 1.0]), 1.0,),
    (8, Vectors.dense([0.0, 1.0, 12.0, 0.0]), 0.0,),
    (9, Vectors.dense([1.0, 0.0, 15.0, 0.1]), 0.0,)], ["id", "features", "clicked"])

df.show()
selector = ChiSqSelector(numTopFeatures=1, featuresCol="features",
                         outputCol="selectedFeatures", labelCol="clicked")

result = selector.fit(df).transform(df)

print("ChiSqSelector output with top %d features selected" % selector.getNumTopFeatures())
result.show()<jupyter_output>ChiSqSelector output with top 1 features selected
+---+------------------+-------+----------------+
| id|          features|clicked|selectedFeatures|
+---+------------------+-------+----------------+
|  7|[0.0,0.0,18.0,1.0]|    1.0|          [18.0]|
|  8|[0.0,1.0,12.0,0.0]|    0.0|          [12.0]|
|  9|[1.0,0.0,15.0,0.1]|    0.0|          [15.0]|
+---+------------------+-------+----------------+
{'fraction_non_alphanumeric': 0.19726948951325682, 'fraction_numerical': 0.047685001978630785, 'mean_word_length': 2.7306273062730626, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '85456', 'license': 'Apache-2.0', 'n_tokens_mistral': 1622, 'n_tokens_neox': 1494, 'n_words': 554, 'path': 'Extracting_transforming_selecting_features/featureSelector.ipynb', 'repo_name': 'iamhimanshu0/Spark', 'size': '5054'}