<jupyter_start><jupyter_text>This dataset will be very large, so I will be using a single companies customer service data.<jupyter_code>pd.options.display.float_format = '{:,.0f}'.format
answers = full_data.loc[full_data['author_id'] == "sainsburys"]
full_data.shape
full_data.info()
answers.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 19466 entries, 773 to 2810390
Data columns (total 7 columns):
 #   Column                   Non-Null Count  Dtype  
---  ------                   --------------  -----  
 0   tweet_id                 19466 non-null  int64  
 1   author_id                19466 non-null  object 
 2   inbound                  19466 non-null  bool   
 3   created_at               19466 non-null  object 
 4   text                     19466 non-null  object 
 5   response_tweet_id        8172 non-null   object 
 6   in_response_to_tweet_id  19417 non-null  float64
dtypes: bool(1), float64(1), int64(1), object(4)
memory usage: 1.1+ MB<jupyter_text>We reduced the dataset from 2811774 entries to 19466 entries, but we still need to include the sutomers tweets<jupyter_code>answers.head()<jupyter_output><empty_output><jupyter_text>We will get the customer tweets by using in_response_to_tweet to find what tweet the customer service of sainsburys was answering<jupyter_code>answers.columns[answers.isnull().any()]
questionTweetId = []
answers["in_response_to_tweet_id"] = answers["in_response_to_tweet_id"].fillna(0.0).astype(int)

questionTweetId = answers["in_response_to_tweet_id"].to_list()

#Below was code used for various testing/troubleshooting


#tweetId = questionTweetId[3819]
#tweetId = 758160
#print(tweetId)

#index = full_data.index
#condition = full_data["tweet_id"] == tweetId
#indices = index[condition]
#indices_list = indices.tolist()

#condition1 = full_data["in_response_to_tweet_id"] == tweetId
#indices1 = index[condition1]
#indices_list1 = indices1.tolist()

#print(indices_list)

#print(indices_list1)
#if indices_list:
   # print(full_data.at[indices_list[0], "text"])
#if indices_list1:
    #print(full_data.at[indices_list1[0], "text"])


#full_data.iloc[[677615]]<jupyter_output><empty_output><jupyter_text>This gets all the tweets from sainsburys and what they were responding to<jupyter_code>QandA = []
i = len(questionTweetId)
for j in range(i):
    tweetId = questionTweetId[j]
    if tweetId != 0:
        index = full_data.index
        condition = full_data["tweet_id"] == tweetId
        indices = index[condition]
        indices_list = indices.tolist()
        condition1 = full_data["in_response_to_tweet_id"] == tweetId
        indices1 = index[condition1]
        indices_list1 = indices1.tolist()
        
        #Below was used for testing
        #print("TweetId:{}".format(tweetId))
        #print(indices_list[0])
        
        #The use of these if statements take advantage of the fact that empty sequences are false. This was implemented because of a missing tweetId, 758160
        if indices_list:
            Q = full_data.at[indices_list[0], "text"]   
        if indices_list1:
            A = full_data.at[indices_list1[0], "text"]
        QandA.append([Q, A])<jupyter_output><empty_output><jupyter_text>Created a new dataset called data_clean that consists of 'Questions' and 'Answers'<jupyter_code>data_clean = pd.DataFrame(QandA, columns =['Question', 'Answer'])
data_clean
data_clean.shape
data_clean = data_clean.sample(n=2000)
data_clean
data_clean["Original_Question"] = data_clean["Question"].astype(str)<jupyter_output><empty_output><jupyter_text>Cleaning/pre-proccessing of the data'Answer' has all @'s and Hashtags removed<jupyter_code>data_clean["Original_Answer"] = data_clean["Answer"].astype(str)
data_clean["Answer"] = data_clean["Answer"].astype(str)
data_clean["Answer"] = data_clean.apply(lambda row: re.sub("\B#\w+", "", row['Answer']), axis=1)
data_clean["Answer"] = data_clean.apply(lambda row: re.sub("\B@\w+", "", row['Answer']), axis=1)<jupyter_output><empty_output><jupyter_text>'cleanText' function created.* Removes all @'s and hashtags* Runs a spellchecker* Removes stop words* Removes puncutation* Lemmatizes words<jupyter_code>lemmatizer = WordNetLemmatizer()
wordnet_map = {"N":wordnet.NOUN, "V":wordnet.VERB, "J":wordnet.ADJ, "R":wordnet.ADV}
stopwords = set(stopwords.words('english'))
spell = SpellChecker()
def cleanText(text):
    text = str(text).lower()
    text = re.sub("\B#\w+", "", text)
    text = re.sub("\B@\w+", "", text)
    corrected_text = []
    misspelled_words = spell.unknown(text.split())
    for word in text.split():
        if word in misspelled_words:
            corrected_text.append(spell.correction(word))
        else:
            corrected_text.append(word)
        text = " ".join(corrected_text)
        text = " ".join([word for word in str(text).split() if word not in stopwords])
    text.translate(str.maketrans('', '', string.punctuation))
    pos_tagged_text = nltk.pos_tag(text.split())
    return " ".join([lemmatizer.lemmatize(word, wordnet_map.get(pos[0], wordnet.NOUN)) for word, pos in pos_tagged_text])<jupyter_output><empty_output><jupyter_text>applies 'cleanText' to 'Question'<jupyter_code>data_clean["Question"] = data_clean["Question"].apply(cleanText)
data_clean<jupyter_output><empty_output><jupyter_text>function 'userAsk' created* uses tfidf to get coseine similarity to all values in 'question'<jupyter_code>def userAsk(text):
    tfidf=TfidfVectorizer()
    x_tfidf=tfidf.fit_transform(data_clean["Question"]).toarray()

    df_tfidf = pd.DataFrame(x_tfidf,columns=tfidf.get_feature_names())

    Question_tfidf = tfidf.transform([text]).toarray()

    cos=1-pairwise_distances(df_tfidf,Question_tfidf,metric='cosine')#Find cosine similarity

    index_value = cos.argmax()
    
    #Resets the index of data_clean as otherwise it starts at like 6000 which means it won't work with index-value
    data_clean.reset_index(inplace = True, drop = True)

    return data_clean["Answer"].loc[index_value]<jupyter_output><empty_output><jupyter_text>gets the new question and runs it through 'cleanText' & 'userAsk'<jupyter_code>UserQuestion = input("Enter your Question: ")
UserQuestion = cleanText(UserQuestion)
UserAnswer = userAsk(UserQuestion)
print(UserAnswer)<jupyter_output>If you DM us on the below link your order number, email and postcode I'll arrange an evoucher to be emailed out to you. Thanks, Sam. https://t.co/MdMtnTlX6H
{'fraction_non_alphanumeric': 0.12350287758593871, 'fraction_numerical': 0.022398506766215586, 'mean_word_length': 4.4491525423728815, 'pattern_counts': {'":': 4, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '171258', 'license': 'MIT', 'n_tokens_mistral': 2302, 'n_tokens_neox': 2158, 'n_words': 469, 'path': 'twitter-customer-service.ipynb', 'repo_name': 'DavidMorgan1999/silver-pancake', 'size': '6429'}