<jupyter_start><jupyter_text>> **Note:** In most sessions you will be solving exercises posed in a Jupyter notebook that looks like this one. Because you are cloning a Github repository that only we can push to, you should **NEVER EDIT** any of the files you pull from Github. Instead, what you should do, is either make a new notebook and write your solutions in there, or **make a copy of this notebook and save it somewhere else** on your computer, not inside the `sds` folder that you cloned, so you can write your answers in there. If you edit the notebook you pulled from Github, those edits (possible your solutions to the exercises) may be overwritten and lost the next time you pull from Github. This is important, so don't hesitate to ask if it is unclear.  Exercise Set 16: Exploratory Data Analysis*Afternoon, August 22, 2018*In this exercise set we will be practicing our skills wihtin Exploratory Data Analysis. Furthermore we will get a little deeper into the mechanics of the K-means clustering algorithm.  Exercise Section 16.1: Exploratory data analysis with interactive plotsIn the following exercise you will practice interactive plotting with the Plotly library. ** Preparation:** Setting up plotly:* Install plotly version (2.7.0) using conda: `conda install -c conda-forge plotly==2.7.0`* Intall cufflinks that bins the plotly to the dataframe. `conda install -c conda-forge cufflinks-py`* Create a user on "https://plot.ly/".* Login* Hover over your profile name, and click on settings.* Get the API key and copy it.* Run the following command in the notebook```python First time you run itimport plotly username = 'username'  your.usernameapi_key = 'apikey'  find it under settings  your.apikeyplotly.tools.set_credentials_file(username=username, api_key=api_key)```* Plotly is a sort of social media for Graphs, and automatically save all your figures. If you want to run it in offline mode, run the following in the notebook :```pythonimport plotly.offline as py  import plotly in offline modepy.init_notebook_mode(connected=True)  initialize the offline mode, with access to the internet or not.import plotly.tools as tls tls.embed('https://plot.ly/~cufflinks/8')  embed cufflinks. import cufflinks and make it offlineimport cufflinks as cfcf.go_offline()  initialize cufflinks in offline mode``` > ** Ex. 16.1.1 ** Reproduce the plots made in the Lectures. This means doing a scatter plot where the colors (hue=) are the ratings, and the text comes when hovering over (text=).<jupyter_code>#[Answer goes here]<jupyter_output><empty_output><jupyter_text>Exercise Section 16.2: Implementing the K-means Clustering algorithmIn the following exercise you will implement your own version of the K-means Clustering Algorithm. This will help you practice the basic matrix operations and syntax in python.  > **Ex. 16.2.0:** First we need to load the dataset to practice on. For this task we will use the famous clustering dataset: of properties of 3 iris flower species. This is already build into many packages, including the plotting library seaborn, and can be loaded using the following command: ```df = sns.load_dataset('iris')```Plot the data as a scatter matrix to inspect that it indeed has some rather obvious clusters: search for seaborn and scatter matrix on google and figure out the command. Color the markers (the nodes in the graph) by setting the ```hue='species'```<jupyter_code># [Answer to Ex. 16.2.0]<jupyter_output><empty_output><jupyter_text>If we weren't biologist and we had not already named the three flower species, we might want to find and define the natural groupings using a clustering method. Now you should implement the K-Means Clustering Algorithm.> **Ex. 16.2.1:** First define a matrix X, by extracting the four columns ('sepal_length','sepal_width','petal_length','petal_width') from the dataframe using the .values method.<jupyter_code># [Answer to Ex. 16.2.1]<jupyter_output><empty_output><jupyter_text>Now we are ready to implement the algorithm. > **Ex. 16.2.2:** First we write the initialization, our first *Expectation*. This will initialize our first guess of the cluster centroids. This is done by picking K random points from the data. We do this by sampling a list of K numbers from the index. And then extracting datapoints using this index, same syntax as with a dataframe. ***(hint: use the random.sample function and sample from a range(len(data)))***Check that this works and wrap it in a function named `initialize_clusters`. The function should take the data and a value of K (number of clusters / intial samples) as input parameters. And return the initial cluster centroids.<jupyter_code>#[Answer Ex. 16.2.2]<jupyter_output><empty_output><jupyter_text>Now we will write the *Maximization* step.> **Ex.16.2.3.:** The maximization step is done by assigning each datapoint to the closests cluster centers/centroid. This means:* we need to calculate the distance from each point to each centroid (at first it is just our randomly initialized points). This can be done using the the sklearn.metrics.pairwise_distances() taking the two matrices as input. * Next run an argmin operation on the matrix to obtain the cluster_assignments, using the ```.argmin()``` method built into the matrix object. The argmin gives you the index of smallest value, and not the smallest value itself. Remember to choose the right axis to apply the argmin operation on - i.e. columns or rows to minimize. You do this setting the axis= argument. ```.argmin(axis=0)``` applies it on the columns and  ```.argmin(axis=1)``` applies it on the rows. Finally wrap these operations into a function `maximize` that takes the cluster centers, and the data as input. And return the cluster assignments.<jupyter_code>#[Answer Ex. 16.2.3]<jupyter_output><empty_output><jupyter_text>> **Ex. 16.2.4:** Now we want to update our Expectation of the cluster centroids. We calculate new cluster centroids, by applying the builtin ```.mean``` function on the subset of the data that is assigned to each cluster. First you define a container for the new centroids. Using the function: `np.zeros(shape)`. The `shape` parameter should be a tuple with the dimensions of matrix of cluster centroids i.e. (k, n_columns). >For each cluster you *(this can be done using a for loop from 0 to k number of clusters)*:* filter the data with a boolean vector that is True if the cluster assignment of the datapoint is equal to the cluster. The indexing is done in the same way as you would do with a dataframe. * calculate the mean on the subset of the data. Make sure you are doing it on the right axis. (axis=0) is on the columns, and axis=1 is on the rows.* store the it in a containerEach cluster center should be a vector of 4 values [val,val2,val3,val4] so make sure you take the mean on the right axis. ```.mean(axis=?)```.  Finally wrap these operations into a function `update_expectation` that takes the, `k`, the data `X`, and the `cluster_assignment` as input. And return the new cluster centers.<jupyter_code>#[Answer 16.2.4]<jupyter_output><empty_output><jupyter_text>Lastly we put it all together in the scikitlearn canonical ".fit()" function. This function will use the other functions. The important new things here are setting the number of maximization steps, and checking if the solution has converged, i.e. it is stable with little to no change.Pipeline is the following:* First we initialize our cluster centroids using the initialization function.* Then we run the maximazation function until convergence. Converged is checked by comparing if old_centroids from the previous step is equal to the new centroids.* Once convergence is reached we have our final cluster centroids, and the final cluster assignment.> **Ex. 16.2.5:** You should now implement it by doing the following:* Define a maximum number of iterations`max_iter` to 15.* Use the `initialize_clusters` function to define a variable `centroids`.* make a `for` loop from 0 to max_iter where you:     * copy the current cluster centroids to a new variable: old_centroids. This will be used for checking convergence after the maximization step.    * define the `cluster_assignment`  by running the `maximize` function    * define a new (i.e. overwrite) `centroids` variable by running the `update_expectation` function.    * finally check if old_centroids is equal to new_centroids, using the np.array_equal() function. If they are: break.Make sure that it works and wrap it around a function `fit_transform()` that takes the data `X` as input, and the number of clusters `k` plus the maximum number of iterations `max_iter`. It should return the cluster assignments and the cluster centroids.<jupyter_code>#[Answer exercise 16.2.5]<jupyter_output><empty_output><jupyter_text>> **Ex. 16.2.6:**  Run the algorithm and create a new variable `'cluster'` in your dataframe using the cluster_assignments. Count the overlap between the two variables, by using the `pd.pivot_table()` method. between the species and each cluster. Define the `aggfunc=` to the 'count' method. extra: To avoid a local minima (due to unlucky random initialization) you should run the algorithm more than once. Write a function that fits the algorithm N number of times, and evaluates the best solution by calculating ratio between the average distance between all points within the same cluster and the average distance to points outside ones cluster.<jupyter_code>#[Answer to Ex. 16.2.6]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06948831332912192, 'fraction_numerical': 0.009896820383238577, 'mean_word_length': 4.93316677076827, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '5632', 'license': 'MIT', 'n_tokens_mistral': 2542, 'n_tokens_neox': 2435, 'n_words': 1374, 'path': 'Test_karl/material/session_16/exercise_16.ipynb', 'repo_name': 'karlbindslev/sds_group29', 'size': '9498'}