<jupyter_start><jupyter_text>INTELLIHUB ML Clustering Tutorial<jupyter_code># If you are running this notebook on GOOGLE COLAB, uncomment below code & run this cell

# !git clone https://github.com/Spotflock/intellihub-sdk-python.git
# %cd intellihub-sdk-python/examples/machine_learning
# !pip install intellihub
# if using source code

import os
os.chdir("../../")<jupyter_output><empty_output><jupyter_text>Importing Packages<jupyter_code>import intellihub
from intellihub import data_preprocessing
from intellihub.dataset_types import Dataset
from intellihub import helper
from getpass import getpass<jupyter_output>New version of intellihub (1.4.1) available, you are still using older (1.4.0) version of the intellihub, Please update using "pip install intellihub==1.4.1"<jupyter_text>Data|Feature | Description||:-|:-||**CustomerId** | Unique ID assigned to the customer||**Gender** | Gender of the customer||**Age** | Age of the customer||**Annual Income (k$)** | Annual Income of the customer||**Spending Score (1-100)**| - Score assigned by the mall based on customer behavior and spending nature|<jupyter_code># read data
df = data_preprocessing.read_data(r'examples/data/csv/mall_customers.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Initializing INTELLIHUB Client Please check intellihub-sdk documentation on setting up intellihub **link**> Parameters Descriptions>> **API key**: `If authentication is enabled`, you need to provide a valid API key>> **base_url**: url where kong/base service provided by INTELLIHUB is deployed.<jupyter_code>api_key = getpass("Valid API key (default=None) : ") or None
base_url = input("Endpoint for INTELLIHUB (default=http://localhost:5000) : ") or "http://localhost:5000"
client = intellihub.IntellihubAiClient(api_key, base_url)<jupyter_output>Valid API key (default=None) : ········
Endpoint for INTELLIHUB (default=http://localhost:5000) : https://preprod-gcp-kong.intellihub.ai<jupyter_text>See Also:  1. [How to generate API key](https://docs.intellihub.ai/intellihub-sdk/docs/build/html/getting_started/generateAPIkey.html)2. [INTELLIHUB deployment](http://docs.intellihub.ai/intellihub-sdk/docs/build/html/getting_started/INTELLIHUB_setup.html)    Uploading Data<jupyter_code>train_data_store_response = client.store(r'examples/data/csv/mall_customers.csv', Dataset.TRAIN_DATA)
print(train_data_store_response)
train_data = train_data_store_response['fileUrl']<jupyter_output>C:\Users\kavya\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py:1013: InsecureRequestWarning:

Unverified HTTPS request is being made to host 'preprod-gcp-kong.intellihub.ai'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings<jupyter_text>Information about supported libraries & Hyperparameters 1. To check list of supported libraries, we can use below code<jupyter_code>intellihub.helper.get_ml_model_info("libraries")<jupyter_output>Supported libraries for Machine Learning: ['scikit', 'weka', 'h2o']<jupyter_text>2. To check list of supported algorithms under a supported library for classification task, For more details on supported algorithm check reference link<jupyter_code>intellihub.helper.get_ml_model_info("algorithms", library='scikit', task='clustering')<jupyter_output>Supported algorithms for CLUSTERING task in SCIKIT library are:

 algorithm : KMeansClustering,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

 algorithm : MiniBatchKMeans,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html

 algorithm : AffinityPropagation,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AffinityPropagation.html

 algorithm : MeanShift,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MeanShift.html

 algorithm : Birch,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.Birch.html

 algorithm : SpectralClustering,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html

 algorithm : AgglomerativeClustering,
 reference link: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html

[...]<jupyter_text>3. To check list of hyperparameter for supported algorithms under a library for classification task, following code can be used<jupyter_code>intellihub.helper.get_ml_model_info("hyperparameters", library='scikit', task='clustering', algorithm='KMeansClustering')<jupyter_output>Supported hyperparameters for KMEANSCLUSTERING algorithm in SCIKIT library are: 
 
Parameter                                              Default Value                 
algorithm                                              auto                          
copy_x                                                 True                          
init                                                   k-means++                     
max_iter                                               300                           
n_clusters                                             8                             
n_init                                                 10                            
tol                                                    0.0001                        
verbose                                                0                             

for more info please refer https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html<jupyter_text>Cluster Recommendation Used to find the optimal number of clusters based on the dataset.<jupyter_code>client.clusterRecommendation(train_data, features=['Age','Annual Income (k$)','Spending Score (1-100)'], algorithm="KMeansClustering")<jupyter_output><empty_output><jupyter_text>4. Set parameters for building the model<jupyter_code>kmeans_parameters = {"n_clusters":5}<jupyter_output><empty_output><jupyter_text>Model Building Next step after uploading the dataset is to train a model using Train Dataset.Please check **link** for supported Libraries/framework for training a model.For this notebook, we will use *  **Library :** Scikit*  **Algorithm :** AgglomerativeClustering<jupyter_code># Define the task - clustering
task = "clustering"

# algorithm and library to use
algorithm = "KMeansClustering"
library = "scikit"

# features to train the model on
features = ['Gender', 'Age', 'Annual Income (k$)', 'Spending Score (1-100)']

# number of clusters
num_of_clusters = 5
cluster_type = "Centroid"

# training
cluster_response = client.cluster(service = "clustering", algorithm = algorithm, dataset = train_data, features = features, lib = library, number_of_clusters = num_of_clusters, cluster_type = cluster_type, model_name = "Clustering_Model",params=kmeans_parameters)
print(cluster_response)<jupyter_output>{'code': 200, 'data': {'jobId': 174601, 'appId': 1608720742960, 'projectId': 0, 'name': 'scikit_clustering_cluster', 'library': 'scikit', 'service': 'Clustering', 'task': 'CLUSTER', 'state': 'RUN', 'startTime': '2021-09-21T09:48:16.689+00:00', 'endTime': None, 'request': {'library': 'scikit', 'jobType': 'CSV', 'queryId': None, 'modelId': None, 'isSingle': False, 'singleRow': None, 'config': {'name': 'Clustering_Model', 'algorithm': 'KMeansClustering', 'datasetUrl': '/YOUR_S3_BUCKET/private/687361782950394294/Train_Data/1632217677828_mall_customers.csv', 'numOfClusters': 5, 'epsilon': 0.1, 'features': ['Gender', 'Age', 'Annual Income (k$)', 'Spending Score (1-100)'], 'params': {'n_clusters': 5}, 'clusterType': 'Centroid'}}, 'isStreamJob': False, 'jobType': 'CSV', 'queryId': None, 'isSingle': False, 'isJobStopped': None, 'automateJobId': None, 'output': None, 'trainDataSet': None}}<jupyter_text>Checking training statusAs training a model might take lot of time depending on size of dataset, we can check current status of model training using below functions<jupyter_code># check job_status
cluster_job_status_response = client.job_status(cluster_response['data']['jobId'])
print(cluster_job_status_response)<jupyter_output>{'jobId': 174601, 'appId': 1608720742960, 'projectId': 0, 'name': 'scikit_clustering_cluster', 'library': 'scikit', 'service': 'Clustering', 'task': 'CLUSTER', 'state': 'FINISH', 'startTime': '2021-09-21T09:48:16.689+00:00', 'endTime': None, 'isStreamJob': False, 'jobType': 'CSV', 'queryId': None, 'isSingle': False, 'isJobStopped': None, 'automateJobId': None, 'output': None, 'trainDataSet': None}<jupyter_text>**Note :** Proceed further only after state is **'FINISH'**      Model Training Job Output<jupyter_code># check job output
cluster_job_output_response = client.job_output(cluster_response['data']['jobId'])
print(cluster_job_output_response['output']['data'].keys())
cluster_job_output_response['output']
# download the clustered dataset for further analysis
response = client.download(cluster_job_output_response['output']['clusterFileUrl'])
# read the data
from io import StringIO
import pandas as pd
pred_data = StringIO(response.text)

df = pd.read_csv(pred_data, sep=",")
# box plots to check distribution of each variable with respect to the clusters formed
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
features = ['Gender', 'Age', 'Annual Income (k$)', 'Spending Score (1-100)']
ROWS, COLS = 2, 2
fig, ax = plt.subplots(ROWS, COLS, figsize=(18,8) )
row, col = 0, 0
for i, feature in enumerate(features):
    if col == COLS - 1:
        row += 1
    col = i % COLS
    
    sns.boxplot(x="cluster",y=feature,data = df,ax=ax[row, col]).set_title(feature)<jupyter_output><empty_output><jupyter_text>We can see that Annual Income is the major contributor for the cluster formation. Clusters 1 and 2 have the highest mean annual income. Along with it, Spending Score also contributes<jupyter_code># scatter plot to check ditribution of 2 variables with respect to the clusters formed
sns.pairplot(df, hue="cluster")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11597374179431072, 'fraction_numerical': 0.020190968768649294, 'mean_word_length': 4.546056260341975, 'pattern_counts': {'":': 1, '<': 49, '<?xml version=': 0, '>': 54, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '365755', 'license': 'MIT', 'n_tokens_mistral': 3209, 'n_tokens_neox': 2987, 'n_words': 765, 'path': 'examples/machine_learning/.ipynb_checkpoints/INTELLIHUB ML Clustering Tutorial-checkpoint.ipynb', 'repo_name': 'Spotflock/intellihub-sdk-python', 'size': '10054'}