<jupyter_start><jupyter_text>Example of application of the sdlabb package for metadata extaction TF-IDF method Created 13-11-20Author: <NAME>   WalkthroughThis walkthrough will only deal with the `NCOF`method available in the package, an example of the `TF-IDF` based method is available in the demos folder. This walkthrough is available as a notebook in the demos folder. Install the alpacka package through pip, or download the package through github. `pip install alpacka`[Link to github repo](https://github.com/BernhardMoller/alpacka)  Set upTo be able to use the alpacka package a data set for the analysis is needed. For this demo we will use the "Twitter US Airline Sentiment" data set. Availabe at https://www.kaggle.com/crowdflower/twitter-airline-sentiment<jupyter_code>import pandas as pd
data = pd.read_csv('Tweets.csv')


from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))

data.head()<jupyter_output><empty_output><jupyter_text>From the code we can see that the data has been loaded but the "airline_sentiment" are not integer labels, we need to fix this.**IMPORTANT:** The alpacka package can only handle integer labels in the range of [0 n]. If your data set has labels in any other range you will need to translate them to the [0 n] range.<jupyter_code>print(data.airline_sentiment[:10])
score = pd.factorize(data.airline_sentiment)
print(score[:10])
score= score[0]
print(score[:10])<jupyter_output>I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than  most.<jupyter_text>We can now see that the data is transformed from stings to integers in the range of [0 2]. With the mapping neutral: 0 , positive: 1, negative: 2.  Importing and initiating alpackaNow we are ready to import and initiate alpacka.<jupyter_code>from alpacka.Pipeline import Pipeline

p = Pipeline(num_words=10000, class_perspective = 2)<jupyter_output><empty_output><jupyter_text>There are some setting that we can make in the NCOF method that we need to specify before we start. One of which is how many unique tokens do we want to to take into consideration in the analysis, variable `num_words`. The defult setting for this variable is `None` meaning that all unique tokens will be used in the analysis. For "large" data sets this choice is quite ambitious given that the number of tokens that appear only once or twice in the a corpus. For this walkthrough we will limit the analysis to the $10,000$ most common words in the corpus.Another setting that we need to specify is for what class we want the results presented for, variable `class_perspective`. As the NCOF method presents results regarding if a token is overrepresented in a class compared to the rest of the corpus, which class to investigate nees to be specified, the defult value of `class_perspective` is `1`. For this demo we have chosen the perspective of class 2 (negatives). Now the `TF-IDF` pipeline is initiated through the wrapper `Pipeline` and the induvidual analysis methods can be accessed be calling: `p.ncof.some_functions()` Now we are ready to start the analysis of our data. Note that the  `.some_functions()` function is a placeholder and do exist.  Calculate TF-IDFNow we are ready to calculate the NCOF score for the review data and its scores. This is done by calling the `.calc_tfidf(data,labels)` function.  For this example the input in the `data` field is the texts, and the `labels` is the review scores.<jupyter_code>tfidf_score, dictionary = p.tfidf.calc_tfidf(data.text, score)<jupyter_output><empty_output><jupyter_text>Stop words can also automatically be removed from the NCOF calculating by passing a set of stop words in the `.calc_tfidf` function. The reason you might want to remove stop words before NCOF calculations is due tothe stop words altering frequency distrubution of "value adding" words by by increasing the frequencylimit of when a word is considered as an NCOF outliers. Top top is to test with and without the stop word removal and you will see the difference. For this demo we keep the score where stop words has been removed.<jupyter_code>tfidf_score, dictionary = p.ncof.calc_ncof(data.text, score,stop_words=stop_words)<jupyter_output><empty_output><jupyter_text>We now have an array, `tfidf_score`, that contains the TF-IDF results for our data. This array will have the size `[2,num_words]` with and positive values. the values from the first row are the TF-iDF score for all investigated tokens from the classes that we have deemed as "not important" i.e. allclasses that are not the `class_perspective`. The second rown contains all TF-IDF score for the class that we have put as the `class_prespective`.<jupyter_code>print(ncof_score[0][:10])<jupyter_output><empty_output><jupyter_text>In addition to an array with the scores the `.calc_tfidf()` function returns a dictionary that maps the indexes in the  `tfidf_score ` array to its text representations, and can be accessed by calling:<jupyter_code>w = list(dictionary.items())[:10]
for item in w:
    print(item)<jupyter_output>TF-IDF score added under 'self.score'  use self.get_--- to access the result<jupyter_text>Sorting resultsTo sort the array into inliers and outliers for the positive and negative values the function `.split_score()`needs to be called. The inliers can be accessed through:**Important:** The outliers associated with the class set as `class_perspective` will be contained in the second element in the outliers list. Such that they can be called from `outliers[1]`. The same for all outliers calculated from the other classes by calling `outliers[0]`.<jupyter_code>inliers, outliers = p.tfidf.split_score(tfidf_score)<jupyter_output>[[0.28835468 0.11496277 0.31456163 ... 0.16087201 0.09147622 0.26491669]
 [0.31219396 0.18952645 0.10242599 ... 0.33223068 0.33409239 0.35338639]]<jupyter_text>Since the outliers from the two groups can have the same word indexes idenified as outliers the symmteric set difference is taken from both groups such that the only outlier words that remain are those that are identified in only the "positive" group or the "negative" group. This is done by using the function `.unique_outliers_per_class()`<jupyter_code>outliers_pos , outliers_neg = p.tfidf.unique_outliers_per_class(outliers=outliers)<jupyter_output><empty_output><jupyter_text>This call will return the indexes of the words in the dictionary that are considered as outliers in the TF-IDF results.The results are sorted within the `outliers_pos `and `outliers_neg ` as the following:For class $1$tfidf_pos[0] = $\mu+\sigma\leq result <\mu+2\sigma$tfidf_pos[1] = $\mu+2\sigma\leq result <\mu+3\sigma$tfidf_pos[2] = $\mu+3\sigma\leq result$    For class $0$tfidf_neg[0] = $\mu+\sigma\leq result <\mu+2\sigma$   tfidf_neg[1] = $\mu+2\sigma\leq result <\mu+3\sigma$tfidf_neg[2] = $\mu+3\sigma\leq result$  Plotting resultsThese results can be plotted by calling the function `.plot_merged_histogram()` this will give some visual information regarding how the distrubutions of the TF-IDF scores are in the two groups.<jupyter_code>p.tfidf.plot_merged_histogram(tfidf_score,x_label = 'tf-idf score', y_label= 'counts', legend= ['class 0', 'class 1'])<jupyter_output><empty_output><jupyter_text>Plotting resultsThese results can be plotted by calling the function `.scatter()` which will give visual information regarding what tokens are over or under represented in the investigated class.<jupyter_code>p.tfidf.scatter()<jupyter_output><empty_output><jupyter_text>The results can also be plotted as an histogram. Note that the input in this example is the entirery of the TF-IDF score.   Converting results from indexes to textSince it is quite difficult to interpret the socre for each the indexes directly, it is suggested that the indexes are transformed back to their text representations. This can be done by calling the `.tfidf.ind_2_txt(data)` function, the function input should be either indexes of the positive or negative outlers.<jupyter_code>words_pos = p.tfidf.ind_2_txt(tfidf_pos)

words_neg = p.tfidf.ind_2_txt(tfidf_neg)<jupyter_output><empty_output><jupyter_text>If the text results want to be cleaned from stop words for clarification. The function `.remove_stop_words(data,stop_words)` can be called. This functon compares the content of the input `data` to that of the input `stop-words` and removes any matches between them from the `data`. For this walkthrough we will use the stop words available from the NLTK package.We have already removed the stop words from the data set when we calculated the NCOF score, so we will not need to preform this step. That is why this part is commented out.<jupyter_code># words_pos = p.tfidf.remove_stop_words(words_pos,stop_words)
    
# words_neg = p.tfidf.remove_stop_words(words_neg,stop_words)<jupyter_output><empty_output><jupyter_text>Print results to terminal. We have now gone through all the steps required to produce, plot, and clean the reults from the NCOF analysis method. The last part is to either save the results to a file or to print them to the terminal. Since format to save the results to is a user preference no function for this is provided in the alpacka package, however the results can be printed to the terminal by calling the following function.The input variable `sort` can be set to either `True` or `False` and decides if the results should be printed as alphabetically sorted or not.<jupyter_code>print(f"printing outliers from the investigated class")
p.tfidf.get_result(tfidf_score,words_pos, sort = True)
print(f" ")
print(f"printing outliers from the remaining classes")
p.tfidf.get_result(tfidf_score,words_neg, sort = True)<jupyter_output>printing outliers from the investigated class
Printing 1-sigma outliers, alphabetically sorted
####################
125
1lbs
24
3x
41
4lb
53
62
73
81
a1
absolute
absolutly
accept
access
accompanied
achieved
acrid
ad
afghanistan
african
alarm
alfredo
aloe
aloha
alone
amazons
ambrosia
annoying
another
anticipation
antler
antlers
anyhow
appear
appetizers
applesauce
appreciates
approved
apso
argentina
arrival
artificially
asap
assortment
attachment
aunt
austin
australia
automatic
awaiting
award
babies
bad
bands
bang
barry
baskets
bbq
begging
beneful
bengal
berry
betty
bichon
bike
biking
binge
bionaturae
biscuit
bitter
blair
blending
blessed
blood
blooming
blues
boil
bone
bonito
bonus
booster
bothersome
bouillon
bowls
bowser
boys
brainer
branches
brazil
brazilian
breakfast
breakfasts
breaks
breathe
breathing
breville
bridal
brighter
brush
buffet
bugs
bulky
bunnies
busted
butterscotch
buttons
buzz
caf
calif
calming
calms
cancelling
candies
canines
captured
carb
carbquik
cardamom
cared
carrag[...]
{'fraction_non_alphanumeric': 0.0636279926335175, 'fraction_numerical': 0.017495395948434623, 'mean_word_length': 5.484179104477612, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 54, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '216992', 'license': 'MIT', 'n_tokens_mistral': 3422, 'n_tokens_neox': 3200, 'n_words': 1420, 'path': 'demos/alpacka_TFIDF_demo.ipynb', 'repo_name': 'BernhardMoller/alpacka', 'size': '10868'}