<jupyter_start><jupyter_text>Preprocess text dataSince the data we're now working with is at its rawest form, we need to preprocess it before extracting information from it.in this step, we will:* Convert all text to lower cases* Delete all tabulation,spaces, and new lines* Delete all numericals* Delete nltk's defined stop words * Lemmatize text<jupyter_code>from fold_to_ascii import fold as ascii_fold
import re

## Lower case, digits, force ascii
test['description'] = test['description'].apply(lambda x: ascii_fold(x.lower(), 'REMOVE_ME').replace('REMOVE_ME', ''))

## only alpha numeric
test['description'] = test['description'].apply(lambda x: re.sub(r'[^a-z]', ' ', x))

## remove double space
test['description'] = test['description'].apply(lambda x: " ".join(x.strip() for x in x.split()))

#remove stop words
stop = stopwords.words('english') + stopwords.words('portuguese')
test['description'] = test['description'].apply(lambda x: " ".join(x for x in x.split() if x not in stop))

## lemmatization
test['description'] = test['description'].apply(lambda x: " ".join([Word(word).lemmatize() for word in x.split()]))

print("Preprocessed data: \n")
print(test.head())<jupyter_output>Preprocessed data: 

            similarity_all_titles  \
0             front end developer   
1                designer grafico   
2                designer grafico   
3  comercial business acquisition   
4                gestor marketing   

                                         description  
0  seek flexible people open explore different pr...  
1  projecto editorial queremos recrutar designer ...  
2  procuramos designer grafico criacao recorte im...  
3  estudio design full service expansao procuramo...  
4  admite departamento marketing gestor marketing...<jupyter_text>Visualize dataIn this step, **we will aggregate our data by job titles** in order to visualy detect the most frequent words for each job.<jupyter_code>## jda stands for job description aggregated
jda = test.groupby(['similarity_all_titles']).sum().reset_index()
print("Aggregated job descriptions: \n")
print(jda)
## Visualize data
jobs_list = jda.similarity_all_titles.unique().tolist()
for job in jobs_list[:5]:

    # Start with one review:
    text = jda[jda.similarity_all_titles == job].iloc[0].description
    # Create and generate a word cloud image:
    wordcloud = WordCloud().generate(text)
    print("\n***",job,"***\n")
    # Display the generated image:
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis("off")
    plt.show()
## Delete more stop words
other_stop_words = ['junior', 'senior','experience','experiência','etc','empresa', 'trabalho', 'pessoa', 'profissionais', 
                    'procuramos']

test['description'] = test['description'].apply(lambda x: " ".join(x for x in x.split() if x not in other_stop_words))<jupyter_output><empty_output><jupyter_text>ModelingWe are now going to translate this skill-extraction problem into a classification one first.And then extract the most important features from each class.The most important features, in this case, represent the words that most likely will belong to a class ( in our case job title)<jupyter_code>## Converting text to features 
vectorizer = TfidfVectorizer(min_df=10, max_df=0.9, ngram_range=(1,2))
#Tokenize and build vocabulary
X = vectorizer.fit_transform(test.description[:10000])
y = test.similarity_all_titles[:10000]

# split data into 80% training and 20% test
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5,random_state=109) 

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.5, random_state=1)
print("train data shape: ",X_train.shape)
print("validation data shape: ",X_val.shape)
print("test data shape: ",X_test.shape)
vectorizer.get_feature_names()[124:142]
# Fit model
clf = MultinomialNB()
clf.fit(X_train, y_train)
## Predict
y_predicted = clf.predict(X_val)
# y_train.hist()
# y_test.hist()<jupyter_output><empty_output><jupyter_text>Model Evaluation<jupyter_code>#evaluate the predictions
print("Accuracy score is: ",accuracy_score(y_val, y_predicted))
print("Classes: (to help read Confusion Matrix)\n", clf.classes_)
print("Confusion Matrix: ")

print(confusion_matrix(y_val, y_predicted))
print("Classification Report: ")
print(classification_report(y_val, y_predicted))<jupyter_output>Accuracy score is:  0.1008
Classes: (to help read Confusion Matrix)
 ['account agencia video digital' 'account desk officer' 'account manager'
 ... 'yupi promotores' 'yupi promotores comerciais' 'yupi talkaholics']
Confusion Matrix: 
[[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
Classification Report: 
                                                                                            precision    recall  f1-score   support

                                            academia canto vocal emotion procura comercial       0.00      0.00      0.00         1
                                                                         account comercial       0.00      0.00      0.00         1
                                                                            account junior       0.00      0.00      0.00         1
                                                                           account manager   [...]
{'fraction_non_alphanumeric': 0.11239669421487604, 'fraction_numerical': 0.022222222222222223, 'mean_word_length': 3.070254110612855, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '91559', 'license': 'MIT', 'n_tokens_mistral': 1705, 'n_tokens_neox': 1604, 'n_words': 496, 'path': 'Code/ESCO project/topic_modeling_test.ipynb', 'repo_name': 'gilnribeiro/Work-Project', 'size': '5445'}