<jupyter_start><jupyter_text>Questions- [X] What words combinations get the most Engagement?- [X] What words combinations get less engagement?- [X] What words combinations get more engagement in the start of headline?- [X] What words combinations get more engagement in the end of headline? Next steps- [ ] Use Plotly To get started, let's read in the necessary libraries we will need to wrangle our data: pandas and numpy. If we decided to build some basic plots, matplotlib might prove useful as well. We also need unicodedat, re and nltk normalize text data.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import unicodedata
import re


from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

stop_words = set(stopwords.words('english'))

plt.rcParams["figure.figsize"] = (10, 10)

pd.set_option('display.max_colwidth', None)

df = pd.read_csv('medium-data-science-articles-2021.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Preparing data Find url duplicateds values and drop itAs same Url means same articles, let's remove it.<jupyter_code>print(f' We have {df.url.duplicated().sum()} duplicated values in url colum')
df[df.url.duplicated(keep=False)]
# I analysed the values and conclude that the correct duplicated values 
#is aways the first! So let's keep it.

df = df.drop_duplicates(subset=['url'], keep='first')
print(f' We have {df.url.duplicated().sum()} duplicated values in url column')
df.shape<jupyter_output>We have 0 duplicated values in url column<jupyter_text>Modeling Data  Data understandingLet's take a look on df titles<jupyter_code>df.title[:10]<jupyter_output><empty_output><jupyter_text>Cleaning the dataAs you can see, it has some confusing character, lots of ponctuation, and HTML infos, so we need to normalize the data.<jupyter_code>def normalize_title(title):
    title = unicodedata.normalize("NFKD", title) # normalize data
    title = re.sub('<[^>]+>', '', title) # remove anything beteween <> (html noise)
    title = title.lower() # put title in lower case
    title = re.sub("\d", 'X', title) # replace numbers by X
    
    title = re.sub(r'[^\w\s]','',title) # Remove ponctuation
    title = re.sub(' +', ' ', title) #Remove double spaces
    
    #Remove stop words
    #title = word_tokenize(title) #remove stop words
    #title = [word for word in title if not word in stop_words]
    #title = (" ").join(title)
    
    return title 
    
df.title = df.title.apply(normalize_title)
df.title[:10]<jupyter_output><empty_output><jupyter_text>Much Better!!As you can see, i commented the stop word treatment, i did it because it was overshadowing my analisys.  Get data insights Extracting word combinations data from headline<jupyter_code>#Build a python dict of dicts, that contains raw infos about words combinations

dict_word_combinations = {}

for title, claps, responses in zip(df.title, df.claps, df.responses):
    splited_title = title.split(' ')
    
    if len(splited_title) > 2:
        for i in range((len(splited_title) - 2)):
            word_combination = f'{splited_title[i]} {splited_title[i+1]} {splited_title[i+2]}' # Get sets of 3 words
            if word_combination in dict_word_combinations.keys(): 
                dict_word_combinations[word_combination]['frequency'] += 1
                dict_word_combinations[word_combination]['total_claps'] += claps
                dict_word_combinations[word_combination]['total_responses'] += responses
            else: #If it's new
                dict_word_combinations[word_combination] = {'word_combination': word_combination,
                                                            'frequency' : 1, 
                                                            'total_claps': claps,
                                                            'total_responses': responses,
                                                            'first_words': 0,
                                                            'first_words_total_claps' : 0,
                                                            'last_words': 0,
                                                            'last_words_total_claps' : 0,
                                                           }
                                                            
            if i == 0: # If it the headline start
                dict_word_combinations[word_combination]['first_words'] += 1
                dict_word_combinations[word_combination]['first_words_total_claps'] += claps
            if i == len(splited_title) - 3:  # If it the headline finish
                dict_word_combinations[word_combination]['last_words'] += 1
                dict_word_combinations[word_combination]['last_words_total_claps'] += claps<jupyter_output><empty_output><jupyter_text>To make our analisys easy, lets transform the dict in a panda DataFrame<jupyter_code>df_word_combination = pd.DataFrame([dict_word_combinations[x] for x in dict_word_combinations.keys()])
df_word_combination<jupyter_output><empty_output><jupyter_text>What is the most popular set of words?<jupyter_code>df_word_combination.sort_values('frequency',ascending=False)[:10].plot.barh(x='word_combination', y='frequency').invert_yaxis()
plt.grid(True, axis='x')
plt.title('Word combination popularity.')
plt.xlabel('Apparitions')
plt.ylabel('Word Combniation')
plt.show()<jupyter_output><empty_output><jupyter_text>WordCloud for words combination popularity<jupyter_code>from wordcloud import WordCloud
weights = {x: dict_word_combinations[x]['frequency'] for x in dict_word_combinations}

wc = WordCloud(background_color='white', contour_width=1000)
wc.generate_from_frequencies(weights)
plt.imshow(wc, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>What cobination of words in headlines atract more claps?<jupyter_code>#list_word_combination
#Extract mean values for claps and trashold for freqency to remove outliers.

X = 50
df_word_combination['claps_mean'] = df_word_combination.apply(lambda row: 
                                                              row.total_claps / row.frequency 
                                                              if row.frequency >= X 
                                                              else None , axis=1)

Y = 25
df_word_combination['last_words_claps_mean'] = df_word_combination.apply(lambda row: 
                                                              row.last_words_total_claps / row.last_words 
                                                              if row.last_words >= Y
                                                              else None , axis=1)

Z = 25
df_word_combination['first_words_claps_mean'] = df_word_combination.apply(lambda row: 
                                                              row.first_words_total_claps / row.first_words 
                                                              if row.first_words >= Z
                                                              else None , axis=1)

df_word_combination.head()
print(f' There are {df_word_combination.claps_mean.notna().sum()} set of words that appear more than {X} times')
print(f' There are {df_word_combination.last_words_claps_mean.notna().sum()} set of words as last words that appear more than {Y} times')
print(f' There are {df_word_combination.first_words_claps_mean.notna().sum()} set of words as first words that appear more than {Z} times')<jupyter_output>There are 126 set of words that appear more than 50 times
 There are 53 set of words as last words that appear more than 25 times
 There are 71 set of words as first words that appear more than 25 times<jupyter_text>Analysing Best words combinations<jupyter_code>df_word_combination.sort_values('claps_mean',ascending=False)[:10].plot.barh(x='word_combination', y='claps_mean').invert_yaxis()
plt.grid(True, axis='x')
plt.title('Most attractive words combinations')
plt.xlabel('Claps Means')
plt.ylabel('Word combinations')
plt.show()
df_word_combination.sort_values('last_words_claps_mean',ascending=False)[:10].plot.barh(x='word_combination', y='last_words_claps_mean').invert_yaxis()
plt.grid(True, axis='x')
plt.title('Most attractive last words combinations')
plt.xlabel('Clap Means')
plt.ylabel('Word Combniation')
plt.show()
df_word_combination.sort_values('first_words_claps_mean',ascending=False)[:10].plot.barh(x='word_combination', y='first_words_claps_mean').invert_yaxis()
plt.grid(True, axis='x')
plt.title('Most attractive first words combinations')
plt.xlabel('Claps Means')
plt.ylabel('Word combinations')
plt.show()<jupyter_output><empty_output><jupyter_text>Analysing Worses words combinations<jupyter_code>df_word_combination.sort_values('claps_mean',ascending=True)[:40].plot.barh(x='word_combination', y='claps_mean')

plt.title('Less atractive word combinations')
plt.xlabel('Claps Means')
plt.ylabel('Set of words')
plt.show()
df_word_combination.sort_values('last_words_claps_mean',ascending=True)[:20].plot.barh(x='word_combination', y='last_words_claps_mean')
plt.title('Less attractive last words combinations')
plt.xlabel('Claps Means')
plt.ylabel('Word combinations')
plt.show()
df_word_combination.sort_values('first_words_claps_mean',ascending=True)[:20].plot.barh(x='word_combination', y='first_words_claps_mean')
plt.title('Less attractive first words combinations')
plt.xlabel('Claps Means')
plt.ylabel('Word combinations')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10119672475330674, 'fraction_numerical': 0.007138358177619148, 'mean_word_length': 3.1171132238547967, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '633535', 'license': 'CNRI-Python', 'n_tokens_mistral': 2874, 'n_tokens_neox': 2714, 'n_words': 691, 'path': '2021HeadLineWordCombinations.ipynb', 'repo_name': 'viniciusLambert/analyse-data-science-articles', 'size': '9526'}