<jupyter_start><jupyter_text>Whose questions you can answer and which questions you might be interested in? OutlineThe content of this kernel will cover two parts.- Part 1: Finding the users who always ask the similar questions with the specific user.- Part 2: Finding the users who always provide similar answers with the specific user.Both parts will be finished with a two-step process: NLP and KNN model fitting. While the first part will be analyzed with the text of questions while the second part will use the text of answers to solve and analyze.<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import sklearn
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.cross_validation import train_test_split
from wordcloud import WordCloud,STOPWORDS

Questions=pd.read_csv('./Questions.csv',encoding = 'iso-8859-1')
Answers=pd.read_csv('./Answers.csv',encoding = 'iso-8859-1')
User_id_inQ= Questions['OwnerUserId'].unique()
User_id_inA= Answers['OwnerUserId'].unique()
All_id=set(User_id_inQ).intersection(User_id_inA)
print('So we have '+str(len(All_id))+ \
      ' users that post both questions and answers on StackOverFlow')
users=pd.DataFrame({'idUser':list(All_id)})
users.head()
users['Quantity']=users['idUser'].apply(lambda x: \
                    len(Questions[Questions['OwnerUserId']==x]['Body']) \
                    +len(Answers[Answers['OwnerUserId']==x]['Body']))
users.head()
users_final=users.sort(['Quantity'],ascending=0).reset_index(drop=True)
users_final.head()
users_final=users_final.iloc[0:10000,]
users_final.shape
All_id=list(users_final['idUser'])<jupyter_output><empty_output><jupyter_text>Firstly, create a function that can clean the body of questions and answers. Only the main body of questions will be used.<jupyter_code># remove the code part from questions
body = Questions['Body'].str.replace(r'<code>[^<]+</code>',' ')
# build up the question part from questions
Questions['QuestionBody'] = body.str.replace(r"<[^>]+>|\n|\r", " ")
Questions.head()
# remove the code part from questions
body = Answers['Body'].str.replace(r'<code>[^<]+</code>',' ')
# build up the question part from questions
Answers['QuestionBody'] = body.str.replace(r"<[^>]+>|\n|\r", " ")
Answers.head()
Q_data=Questions[['OwnerUserId','QuestionBody']]
A_data=Answers[['OwnerUserId','QuestionBody']]
Question=Q_data[Q_data['OwnerUserId'].isin(All_id)]
Answer=A_data[A_data['OwnerUserId'].isin(All_id)]
Question.head()
Answer.head()
Answer.shape
Answer['QuestionBody'][6]
type(Question.QuestionBody)
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf=TfidfVectorizer()
Q_features=tfidf.fit_transform(Question.QuestionBody)
A_features=tfidf.fit_transform(Answer.QuestionBody)
type(Q_features)
Q_features
A_features<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13459516298633017, 'fraction_numerical': 0.0070101647388713636, 'mean_word_length': 6.692722371967655, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '277301', 'license': 'MIT', 'n_tokens_mistral': 934, 'n_tokens_neox': 893, 'n_words': 232, 'path': 'other notebook/Id.ipynb', 'repo_name': 'sjqgithub/rquestions', 'size': '2853'}