<jupyter_start><jupyter_text>Overall principal of iterated feature selection1. Perform k-means on each of the features individually for some k. 2. For each cluster measure some clustering performance metric like the Dunn's index or silhouette. 3. Take the feature which gives you the best performance and add it to Sf4. Perform k-means on Sf and each of the remaining features individually5. Take the feature which gives you the best performance and add it to Sf4. If you have reached the desired number of features stop, else go back to 4<jupyter_code>dataset = datasets.load_iris()
X = dataset.data
y = dataset.target
n, m = X.shape[0], X.shape[1]
print(n, m)
X
y
pd.value_counts(pd.Series(y))<jupyter_output><empty_output><jupyter_text>Feature selection - code version 1 Organize Iris dataset into a dataframe in order to better imitate our project data format1. Perform k-means on each of the features individually for some k. 2. For each cluster measure some clustering performance metric like the Dunn's index or silhouette. 3. Take the feature which gives you the best performance and add it to Sf4. Perform k-means on Sf and each of the remaining features individually5. Take the feature which gives you the best performance and add it to Sf4. If you have reached the desired number of features stop, else go back to 4<jupyter_code>dataset = datasets.load_iris()
X = dataset.data
y = dataset.target
dir(dataset)
iris = pd.DataFrame(X)
iris.columns = dataset.feature_names
iris
n, m = iris.shape[0], iris.shape[1]
print("number of data points:", n, ". number of variables:", m)
#Let's first use all the features to perform K-Means clustering
model_test = cluster.KMeans(n_clusters=3)
model_test.fit(iris) 
pred_y=model_test.labels_
print("True class labels", "\n", pd.value_counts(pd.Series(y)))
print("Clustered class labels:", "\n", pd.value_counts(pd.Series(pred_y)))
# let's assume there are 3 clusters
num_of_cluster = 3
num_of_iter = 3
model = cluster.KMeans(n_clusters=num_of_cluster)
score = np.zeros([num_of_iter, m]) # the sum of squared distances of samples to their closest cluster center
exclude_columns = [] # best performed models with selected features will be added to this list after every iteration
include_columns = [i for i in range(np.shape(score)[1]) if i not in exclude_columns] # rest of the features
# Let's assume we're going to select 3 features out of 4 features, therefore we're going to iterate 3 times
for iteration in range(3):
    # The first iteration, we're going to test clustering models on each individual variables
    if iteration == 0:
        print("Now processing iteration %d" %iteration, "\n")   
        for i in range(m):
            model.fit(iris[[i]])
            pred_y = model.labels_
            print("cluster labels based on variable %s:" %iris.columns[i], "\n", pd.value_counts(pd.Series(pred_y)))
            score[iteration][i] = model.inertia_
            print("the sum of squared distances of samples to their closest cluster center based on variable %s" \
                  %iris.columns[i], "is:", score[iteration][i])   
        score = score[:,include_columns]
        selected_feature_index = np.argmin(score[iteration], axis=0) 
        selected_feature_score = np.amin(score[iteration], axis=0) 
        selected_feature = iris[[selected_feature_index]]
        exclude_columns.append(selected_feature_index)
        print("Conclusion: cluster based on variable %s" %iris.columns[selected_feature_index], "gives the best performance", "\n") 
    #for following iteration, we're going to add the rest the feature to the selected feature and perform cluster model
    else:
        print("Now processing iteration %d" %iteration, "\n") 
        for i in range(m):
            if i not in exclude_columns:
                # Generate data with features selected from last iteration plus each individual rest of the features
                data = pd.concat([selected_feature, iris[[i]]], axis=1)
                model.fit(data)
                pred_y = model.labels_
                print("cluster labels based on variables:", data.columns, "\n", pd.value_counts(pd.Series(pred_y)))
                score[iteration][i] = model.inertia_
                print("the sum of squared distances of samples to their closest cluster center based on variables:", \
                 data.columns, "is:", score[iteration][i]) 
        include_columns = [i for i in range(np.shape(score)[1]) if i not in exclude_columns]
        selected_feature_score = np.amin(score[:,include_columns][iteration], axis=0) 
        selected_feature_index = np.where(score[iteration] == selected_feature_score)[0][0]
        selected_feature = pd.concat([selected_feature, iris[[selected_feature_index]]], axis=1)
        exclude_columns.append(selected_feature_index)
        print("Conclusion: cluster based on variable %s" %iris.columns[exclude_columns], "gives the best performance", "\n") 
print("Selected features are %s" %iris.columns[exclude_columns])<jupyter_output>Now processing iteration 0 

cluster labels based on variable sepal length (cm): 
 1    63
0    52
2    35
dtype: int64
the sum of squared distances of samples to their closest cluster center based on variable sepal length (cm) is: 15.7581196581
cluster labels based on variable sepal width (cm): 
 0    81
1    36
2    33
dtype: int64
the sum of squared distances of samples to their closest cluster center based on variable sepal width (cm) is: 5.26434343434
cluster labels based on variable petal length (cm): 
 0    58
1    50
2    42
dtype: int64
the sum of squared distances of samples to their closest cluster center based on variable petal length (cm) is: 24.6580407225
cluster labels based on variable petal width (cm): 
 2    52
1    50
0    48
dtype: int64
the sum of squared distances of samples to their closest cluster center based on variable petal width (cm) is: 4.93217435897
Conclusion: cluster based on variable petal width (cm) gives the best performance 

Now processing iteratio[...]<jupyter_text>That completes three iterations of feature selection and selected three iterations for best performance  Feature selection - code version 2 * Perform k-means on each of the features individually for some k. * For each cluster measure some clustering performance metric like the Dunn's index or silhouette. * Take the feature which gives you the best performance and add it to Sf<jupyter_code>for d in range(1,5):
    exec(f'X{d} = np.zeros(n)')
    for i in range(n):
        exec(f'X{d}[i] = X[i][d-1]')
score = [0]*m
for i in range(1, m+1):
    print("Now processing model %d" %i, ", which uses feature # %d only" %i)
    exec(f'model_{i} = cluster.KMeans(n_clusters=3)')
    exec(f'X{i} = X{i}[:, np.newaxis]')
    exec(f'model_{i}.fit(X{i})')
    exec(f'pred_y = model_{i}.labels_')
    print("Now let's compare the true class label to the class labels obtained by clustering")
    print(pd.value_counts(pd.Series(pred_y)))
    exec(f'score[i-1] = model_{i}.inertia_')
    print("For model %d:" %i,"the sum of squared distances of samples to their closest cluster center is %s" %score[i-1])
selected_feature = score.index(min(score)) + 1
print("The # %d feature" % (score.index(min(score)) + 1), "gives the best performance")<jupyter_output>The # 4 feature gives the best performance
{'fraction_non_alphanumeric': 0.09272156220128362, 'fraction_numerical': 0.01952751604533661, 'mean_word_length': 3.9022757697456494, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '17334', 'license': 'MIT', 'n_tokens_mistral': 2229, 'n_tokens_neox': 2094, 'n_words': 845, 'path': '.ipynb_checkpoints/Iterated feature selection using K-means - Sha-checkpoint.ipynb', 'repo_name': 'mbzhuang/CustomerSegmentationforVAProducts', 'size': '7323'}