<jupyter_start><jupyter_text>Data cleaning pipeline Demo Import preprocessed sample dataset<jupyter_code>import pandas as pd

df_prepro = pd.read_csv('../data/sample/corona_sample.csv', index_col = 0)<jupyter_output><empty_output><jupyter_text>Clean the data with the following steps (in sequence):- Remove username- Remove emoticon- Remove url- Remove html- Remove stopwords- Perform lemmatisation- Remove unknown words (including acronyms)- Remove tweets that are shorter than minimum length- Remove duplicated rows (based on the tweet text)- Convert the created_date to date_time format<jupyter_code>import re

def remove_username(entry):
    pattern = r'@.+?\s'
    output = re.sub(pattern, '', entry).strip()
    return output

df_prepro['tweet'] = df_prepro.tweet.apply(remove_username)
def remove_emoticon(inputString):
    return inputString.encode('ascii', 'ignore').decode('ascii')

df_prepro['tweet'] = df_prepro.tweet.apply(remove_emoticon)
def remove_url(entry):
    pattern = r'http\S+'
    output = re.sub(pattern, '', entry).strip()
    return output

df_prepro['tweet'] = df_prepro.tweet.apply(remove_url)
def remove_html(entry):
    pattern = r'<.+?>'
    output = re.sub(pattern, '', entry).strip()
    return output

df_prepro['tweet'] = df_prepro.tweet.apply(remove_html)
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
nltk.download('stopwords')

stop_word = set(stopwords.words('english'))

def remove_stop_word(entry):    
    output = [w for w in word_tokenize(entry) if w.lower() not in stop_word]
    output = ' '.join(output)
    return output

df_prepro['tweet'] = df_prepro.tweet.apply(remove_stop_word)
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer

nltk.download('wordnet')
nltk.download('averaged_perceptron_tagger') 

lemmatizer = WordNetLemmatizer()

def lemmat(entry):
    tokens = [word for word in word_tokenize(entry.lower()) if (word.isalpha() or word.isnumeric())]
    tags = nltk.pos_tag_sents([tokens])
    output = []
    for i,tk in enumerate(tokens):
        tag = tags[0][i][1][0]
        try:
            word = lemmatizer.lemmatize(tk, pos=tag.lower())
        except KeyError:
            word = tk
        output.append(word)
    output = ' '.join(output)
    return output

df_prepro['tweet'] = df_prepro.tweet.apply(lemmat)
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import words
nltk.download('words')

english_word = set(words.words())

def remove_unknown_word(entry):    
    output = [w for w in word_tokenize(entry) if w.lower() in english_word]
    output = ' '.join(output)
    return output

df_prepro['tweet'] = df_prepro.tweet.apply(remove_unknown_word)
long_tweet = df_prepro['tweet'].str.len() > 33
df_prepro = df_prepro[long_tweet]
df_prepro = df_prepro.drop_duplicates(subset='tweet', keep='first')
df_prepro.loc[:,'created_date'] = pd.to_datetime(df_prepro['created_date'])<jupyter_output><empty_output><jupyter_text>Clean sample dataset have 7277 rows<jupyter_code>df_prepro.info()
df_prepro.head(100)<jupyter_output><empty_output><jupyter_text>Validation of the data cleaning pipeline script'cd ..' is to make relative import work for jupyter notebook<jupyter_code>cd ..<jupyter_output>C:\Users\simon\OneDrive\Desktop\team5\team5<jupyter_text>Load preprocessed sample dataset and perform cleaning with data cleaning script<jupyter_code>import pandas as pd
from src.datapipeline.clean_data import clean_pipeline

df_prepro = pd.read_csv('data/sample/corona_sample.csv', index_col = 0)

df_clean = clean_pipeline(df_prepro, min_length=33)<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\simon\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package words to
[nltk_data]     C:\Users\simon\AppData\Roaming\nltk_data...
[nltk_data]   Package words is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\simon\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\simon\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     C:\Users\simon\AppData\Roaming\nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!<jupyter_text>The dataset cleaned by the script also have 7277 rows<jupyter_code>df_clean.info()
df_clean.head(100)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1154263397700152, 'fraction_numerical': 0.0054241701019743975, 'mean_word_length': 5.212938005390836, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '431736', 'license': 'MIT', 'n_tokens_mistral': 1669, 'n_tokens_neox': 1627, 'n_words': 348, 'path': 'notebooks/data_cleaning.ipynb', 'repo_name': 'dashtagger/sentiment_analysis_covid', 'size': '4609'}