<jupyter_start><jupyter_text>Machine Learning - Clustering and ClassificationMachine learning is often divided into three broad categories, supervised, unsupervised and reinforcement learning. We'll be skipping reinforcement learning today, so we can focus on supervised and unsupervised algorithms. Supervised LearningSupervised learning is based on the notion of taking a set of data called a training set, consisting of labeled examples, and using it to build a model of how to classify new examples to fit within the existing training set.  K Nearest NeighborsToday we're going to build a training set and use scikit-learn to build a KNN classifier. As you will see, using existing algorithms (rather than building your own) is extremely simple, so there's no reason not to experiment.<jupyter_code>import random<jupyter_output><empty_output><jupyter_text>Instead of generating a set by hand, let's create a randomize set of points on a plot colored differently based on different distributions. We'll use three different gaussian distributions to create the set.<jupyter_code>red   = [(round(random.gauss(27, 10)), round(random.gauss(31, 10))) for x in range(100)]
green = [(round(random.gauss(40, 8)), round(random.gauss(40, 9))) for x in range(100)]
blue  = [(round(random.gauss(50, 10)), round(random.gauss(60, 12))) for x in range(100)]
import matplotlib.pyplot as plt # Let's plot!
plt.rc('axes', color_cycle=['r', 'g', 'b'])
plt.axis([0, 100, 0, 100])
plt.plot([x for x,y  in red], [y for x,y in red], 'ro')
plt.plot([x for x,y in green], [y for x,y in green], 'go')
plt.plot([x for x,y in blue], [y for x,y in blue], 'bo')
plt.show()<jupyter_output><empty_output><jupyter_text>That looks like a pretty good distribution. Now we can see that there are intuitively three regions of the graph where there are points of a particular color dominating. The KNN algorithm predicts an unlabeled point is most likely to be the same color as its K nearest neighbors.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
neighbors = KNeighborsClassifier(n_neighbors=5)
neighbors.fit(red + green + blue, [0]*100 + [1]*100 + [2]*100) # we have to put all points in serially and label each
print("Point\t\tProbability")
for x,y in zip(range(20, 80, 3), range(20, 80, 3)):
    r,g,b = neighbors.predict_proba([x,y])[0]
    print("P: ({},{})\tR: {}%\n\t\tG: {}%\n\t\tB: {}%\n".format(x, y, r*100, g*100, b*100))<jupyter_output>Point		Probability
P: (20,20)	R: 100.0%
		G: 0.0%
		B: 0.0%

P: (23,23)	R: 80.0%
		G: 20.0%
		B: 0.0%

P: (26,26)	R: 100.0%
		G: 0.0%
		B: 0.0%

P: (29,29)	R: 80.0%
		G: 20.0%
		B: 0.0%

P: (32,32)	R: 100.0%
		G: 0.0%
		B: 0.0%

P: (35,35)	R: 20.0%
		G: 80.0%
		B: 0.0%

P: (38,38)	R: 0.0%
		G: 60.0%
		B: 40.0%

P: (41,41)	R: 0.0%
		G: 100.0%
		B: 0.0%

P: (44,44)	R: 40.0%
		G: 60.0%
		B: 0.0%

P: (47,47)	R: 0.0%
		G: 80.0%
		B: 20.0%

P: (50,50)	R: 0.0%
		G: 40.0%
		B: 60.0%

P: (53,53)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (56,56)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (59,59)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (62,62)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (65,65)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (68,68)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (71,71)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (74,74)	R: 0.0%
		G: 0.0%
		B: 100.0%

P: (77,77)	R: 0.0%
		G: 0.0%
		B: 100.0%<jupyter_text>Unupervised LearningUnupervised learning is based on the idea of pattern matching, where an algorithm learns to find patterns in the data and separates data based on which pattern it fits. K-MeansFor the Unsupervises learning example, we're going to cluster some distinct points into groups. Let's start the same way as last time.<jupyter_code>red2   = [(round(random.gauss(17, 5)), round(random.gauss(21, 6))) for x in range(20)]
green2 = [(round(random.gauss(40, 8)), round(random.gauss(40, 6))) for x in range(20)]
blue2  = [(round(random.gauss(60, 7)), round(random.gauss(60, 4))) for x in range(20)]
plt.rc('axes', color_cycle=['r', 'g', 'b'])
plt.axis([0, 100, 0, 100])
plt.plot([x for x,y  in red], [y for x,y in red2], 'ro')
plt.plot([x for x,y in green], [y for x,y in green2], 'go')
plt.plot([x for x,y in blue], [y for x,y in blue2], 'bo')
plt.show()<jupyter_output><empty_output><jupyter_text>It's clear for the most part where the boundaries between these different point groups is when we look at a colored chart. Now let's see if we can get sklearn to figure out the same, just looking at the data with no help.<jupyter_code>from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=3) # We know there are three clusters
kmeans.fit(red2+green2+blue2)
print("Red:\t{}".format(" ".join(str(kmeans.predict(pt)[0]) for pt in red)))
print("Green:\t{}".format(" ".join(str(kmeans.predict(pt)[0]) for pt in green)))
print("Blue:\t{}".format(" ".join(str(kmeans.predict(pt)[0]) for pt in blue)))<jupyter_output>Red:	1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Green:	0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Blue:	2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
{'fraction_non_alphanumeric': 0.15592264302981468, 'fraction_numerical': 0.08400483481063659, 'mean_word_length': 4.005040322580645, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '544820', 'license': 'MIT', 'n_tokens_mistral': 2355, 'n_tokens_neox': 1980, 'n_words': 727, 'path': 'Supervised and Unsupervised ML.ipynb', 'repo_name': 'sdiehl/hacklab-ml', 'size': '4964'}