<jupyter_start><jupyter_text>What is this file, and what does it do? This notebook was done as research, and was intended to explore the information gain provided by utilizing the K-means clustering algorithm as applied directly to the text of cards from Eternal TCG. The intent was to potentially find a large amount of card "types/abilities' that had similar effects to create unique decks surrounding these types/abilities. This could be multi-faction situations, dual-faction, wherever we could find strong clusterings of a certain text type/ability. What do I need to run this notebook successfully and expand on it potentially? You'll also need to have the necessary pre-req. packages installed.<jupyter_code>'''
STEP1: GATHER THE DATA
- Use Selenium/BeautifulSoup to webscrape CARDNAME|CARDTEXT|FACTION(S), then store in csv or xls.
OR
- Find somebody's already made excel with card and text details.
STEP2: REMOVE STOPWORDS AND PERFORM K-MEANS ALG. ON CARDTEXT
- Initially I wanted to group the factions, THEN cluster on the text, but for the first run we will cluster on just the text.
'''


# import numpy as np 
# import pandas as pd 
# import matplotlib.pyplot as plt
# import seaborn as sns
# from sklearn.feature_extraction import text
# from sklearn.feature_extraction.text import TfidfVectorizer
# from sklearn.cluster import KMeans
# from nltk.tokenize import RegexpTokenizer
# from nltk.stem.snowball import SnowballStemmer
# %matplotlib inline
# import re
# import requests
# from bs4 import BeautifulSoup
# # import json
# import warnings
# import mechanicalsoup
# warnings.filterwarnings(action='once')
import pandas as pd
import re
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()  # for plot styling
import numpy as np<jupyter_output><empty_output><jupyter_text>Pull in and clean the data<jupyter_code>all_cards = pd.read_excel('all_cards.xlsx')
all_cards.head()
# Remove all cards where the text is NONE (73/1721 cards)
all_cards = all_cards[all_cards['text'] != 'NONE'].copy()
len(all_cards)
# Remove punctuation and special characters, including periods.

all_cards['text'].str.lower()  #  Lowercase text
all_cards['text'] = all_cards['text'].str.replace('[^\w\s]','')  #  Remove punctuation using regex
all_cards['text'] = all_cards['text'].str.replace('\d+','')  #  Remove numbers using regex
all_cards['text'] = all_cards['text'].str.replace(r'enemy|unit(s?)|turn|damage|deal(s?)|player|play|gets|end|unless|depleted|hand|deck|(?<!dis)card','')

#  Remove stopwords using list comprehension and pre-defined list of stopwords
from nltk.corpus import stopwords
stop = stopwords.words('english')
all_cards['text'].apply(lambda x: [item for item in x if item not in stop])

#  User below if we want to break each sentence up into a list of tokenized words... tfidf takes in text as input,
#  let's leave the card text in tact for now.
# from nltk.tokenize import word_tokenize
# all_cards['text'] = all_cards['text'].apply(word_tokenize)
all_cards.head()<jupyter_output><empty_output><jupyter_text>KMeans Clustering<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score

#  Convert text input into numerical array input.
vectorizer = TfidfVectorizer(stop_words='english') # Probably don't need stop_words, already removed.
#  Create and set our input vector. (1648 items)
X = vectorizer.fit_transform(all_cards['text'])
def cluster_away(X, true_k=2):
    #  Cluster the documents 
    #true_k = 5 # try 5,10,15 (factions, dual faction combos, tri faction combos)

    #  Initialize Kmeans clustering
    model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)
    #  Fit with inputs (tfidf input)
    model = model.fit(X)
    #  Predict the clusters (funny cuz nothing to predict...)
    #labels = model.predict(X)
    #  Get cluster centers
    #C = model.cluster_centers_
    
    #  Success! Print top terms for each kth cluster.
    print("Top terms per cluster:")
    order_centroids = model.cluster_centers_.argsort()[:, ::-1]
    terms = vectorizer.get_feature_names()
    for i in range(true_k):
        print ("Cluster %d:" % i),
        for ind in order_centroids[i, :10]:
            print (' %s' % terms[ind]),
        print
cluster_away(X, 2)<jupyter_output>Top terms per cluster:
Cluster 0:
 flying
 unit
 enemy
 summon
 draw
 play
 damage
 gain
 deck
 deal
Cluster 1:
 units
 turn
 overwhelm
 unit
 attack
 pay
 enemy
 play
 summon
 health<jupyter_text>Based on the above, we may want to remove the words 'enemy, unit(s), turn, damage, deal(s), player, play', and maybe 'gain'. They are not very informative words, and they are most common/what unifies our main centroids<jupyter_code># Now try clustering on 2-4 centroids....
#cluster_away(X, 3) # Remove: hand, card, deck, and MAYBE attack
#cluster_away(X, 4) # Remove: unless, depleted(?)
#cluster_away(X, 5) # Now terms are more ambiguous as to whether or not we should keep them...
cluster_away(X, 10) # Remove: gets, end<jupyter_output>Top terms per cluster:
Cluster 0:
 units
 turn
 overwhelm
 attack
 warcry
 lifesteal
 endurance
 killer
 summon
 aegis
Cluster 1:
 deadly
 summon
 units
 nightfall
 flying
 quickdraw
 infiltrate
 enemy
 warcry
 viper
Cluster 2:
 enemy
 unit
 kill
 summon
 hand
 units
 stun
 silence
 deck
 player
Cluster 3:
 flying
 lifesteal
 summon
 aegis
 unit
 endurance
 turn
 valkyrie
 units
 tribute
Cluster 4:
 draw
 card
 deck
 create
 discard
 cards
 summon
 sigil
 choice
 pay
Cluster 5:
 damage
 deal
 enemy
 player
 unit
 summon
 double
 deals
 turn
 lifesteal
Cluster 6:
 charge
 strangers
 turn
 end
 flying
 play
 attack
 quickdraw
 warcry
 lifesteal
Cluster 7:
 gain
 health
 depleted
 armor
 unless
 summon
 sigil
 scout
 transmute
 draw
Cluster 8:
 unit
 turn
 void
 cursed
 kill
 transform
 attack
 enemy
 gets
 health
Cluster 9:
 play
 weapon
 deck
 turn
 gets
 spell
 spellcraft
 unit
 summon
 units<jupyter_text>Deeper analysis<jupyter_code># Now were going to use our custom regex to reduce the 'common' words in our centroids, hopefully making more distinct cases.
# This will be put in with the regex replacing at the top.

# Will test on k=2 for all cards, but shifting towards analyzing dual-faction combos.

# custom_removal = re.compile(r'enemy|unit(s?)|turn|damage|deal(s?)|player|play|gets|end|unless|depleted|hand|card|deck', re.IGNORECASE)
X = vectorizer.fit_transform(all_cards['text']) # Reset X
cluster_away(2) # Now it's looking cleaner. Maybe remove pay|play.....dis? (lol it's because we remove 'card', therefore discard->dis; fixed)


# It's worth noting that everytime the cluster is made the top terms are different, therefore 2 centroids doesn't inform us well.
# Let's try clustering on different classes (Rakano, Skycrag, etc)<jupyter_output>Top terms per cluster:
Cluster 0:
 attack
 kill
 attachment
 flying
 summon
 health
 quickdraw
 overwhelm
 pay
 draw
Cluster 1:
 flying
 summon
 draw
 gain
 deal
 overwhelm
 play
 pay
 health
 lifesteal<jupyter_text>Praxis (Fire + Time)<jupyter_code># Used excel to see we have 550/1721 cards that are fire + time | multi - Praxis. Note this DOES include 'NONE' text cards.
praxis = all_cards[(all_cards['faction'] == 'Multi  - Praxis')|(all_cards['faction'] == 'Fire')|(all_cards['faction'] == 'Time')].copy()
len(praxis) # 528/550, 22 ARE 'NONE' text, this makes sense! :)
X = vectorizer.fit_transform(praxis['text']) # Change X input into the praxis-only.
cluster_away(X, 2)
#  TODO: Do Clustering k=2-10, on JUST units.<jupyter_output><empty_output><jupyter_text>Combrei (Time + Justice)<jupyter_code># Used excel to see we have 554/1721 cards that are fire + time | multi - Rakano. Note this DOES include 'NONE' text cards.
combrei = all_cards[(all_cards['faction'] == 'Multi  - Combrei')|(all_cards['faction'] == 'Time')|(all_cards['faction'] == 'Justice')].copy()
len(combrei) # 527/551, 24 ARE 'NONE' text, this makes sense! :)
X = vectorizer.fit_transform(combrei['text']) # Change X input into the praxis-only.
cluster_away(X, 2)
#  TODO: Do Clustering k=2-10, on JUST units.<jupyter_output>Top terms per cluster:
Cluster 0:
 flying
 summon
 gain
 draw
 health
 endurance
 attack
 play
 pay
 power
Cluster 1:
 aegis
 overwhelm
 gain
 weapon
 empower
 endurance
 bond
 tribute
 warp
 dinosaur<jupyter_text>Rakano (Fire + Justice)<jupyter_code># Used excel to see we have 554/1721 cards that are fire + time | multi - Rakano. Note this DOES include 'NONE' text cards.
rakano = all_cards[(all_cards['faction'] == 'Multi  - Rakano')|(all_cards['faction'] == 'Fire')|(all_cards['faction'] == 'Justice')].copy()
len(rakano) # 525/554, 29 ARE 'NONE' text, this makes sense! :)
X = vectorizer.fit_transform(rakano['text']) # Change X input into the praxis-only.
cluster_away(X, 2) # Each centroid is around the other centroid; a lot of terms overlap.
#  TODO: Do Clustering k=2-10, on JUST units.
# Rakano k-centroid = 3
cluster_away(X, 3)
print('___________________NEXT K AMOUNT_____________________')
# Rakano k-centroid = 5
cluster_away(X, 5)

# Rakano k-centroid = 8
cluster_away(X, 8)

# Rakano k-centroid = 10
cluster_away(X, 10)<jupyter_output>Top terms per cluster:
Cluster 0:
 attack
 summon
 weapon
 draw
 charge
 overwhelm
 play
 pay
 gain
 endurance
Cluster 1:
 deal
 summon
 overwhelm
 entomb
 equal
 weapon
 night
 destiny
 scout
 void
Cluster 2:
 flying
 warcry
 summon
 valkyrie
 lifesteal
 gain
 armor
 aegis
 endurance
 attack
___________________NEXT K AMOUNT_____________________
Top terms per cluster:
Cluster 0:
 attack
 warcry
 charge
 overwhelm
 endurance
 double
 quickdraw
 lifesteal
 deal
 kill
Cluster 1:
 summon
 deal
 silence
 flying
 choose
 cost
 attack
 draw
 revenge
 stun
Cluster 2:
 flying
 gain
 armor
 warcry
 valkyrie
 aegis
 depleted
 lifesteal
 endurance
 summon
Cluster 3:
 weapon
 draw
 play
 create
 entomb
 void
 tribute
 power
 grenadin
 spell
Cluster 4:
 pay
 ultimate
 exhaust
 grenadin
 attack
 sacrifice
 draw
 valkyrie
 weapon
 flying
Top terms per cluster:
Cluster 0:
 scout
 spell
 kill
 void
 cost
 equal
 weapon
 strength
 skills
 battle
Cluster 1:
 flying
 lifesteal
 summon
 valkyrie
 warcry
 ar[...]<jupyter_text>______________________  TEST REALM IS BELOW EW DONT TOUCH ME  ____________________________________________<jupyter_code>#Plotting
fig = plt.figure(1, figsize=(7,7))
ax = Axes3D(fig, rect=[0, 0, 0.95, 1], elev=48, azim=134)
ax.scatter(X[:, 3], X[:, 0], X[:, 2],
          c=labels.astype(np.float), edgecolor="k", s=50)
ax.set_xlabel("Petal width")
ax.set_ylabel("Sepal length")
ax.set_zlabel("Petal length")
plt.title("K Means", fontsize=14)
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

plt.rcParams['figure.figsize'] = (16, 9)

fig = plt.figure()
ax = Axes3D(fig)
#ax.scatter(X[:, 0], X[:, 1], X[:, 2])
ax.scatter(C[:, 0], C[:, 1], C[:, 2], marker='*', c='#050505', s=1000)
# Graph it!
y_kmeans = model.predict(X)
plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50, cmap='viridis')

centers = model.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c='black', s=200, alpha=0.5);
# Cluster on the faction combinations & graph
# Cluster all of them generally and graph for k=1-10<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09764250527797326, 'fraction_numerical': 0.0166256157635468, 'mean_word_length': 4.625432953983177, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '202778', 'license': 'MIT', 'n_tokens_mistral': 3909, 'n_tokens_neox': 3675, 'n_words': 1298, 'path': 'Old/Research/Clustering-KMeans.ipynb', 'repo_name': 'kubicki14/BurtTheCardKing', 'size': '11368'}