<jupyter_start><jupyter_text>Project 3  Introduction - This data was collected from twitter using twint tool, collected based on specific keywords about STC service in Saudi Arabia from 2015-2021. This data contains over 10K tweet cleaned and pre-processed.  Datatset Link[STC Reviews](https://www.kaggle.com/samaher133/tweets-for-stc-service)  Imorting<jupyter_code>################################################################################
# INSTALL CONDA ON GOOGLE COLAB
################################################################################
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')

################################################################################

! pip install twint 
! conda install -c-forge nest-asyncio -y
! pip install idna-ssl
import pandas as pd<jupyter_output><empty_output><jupyter_text>upgrading the twint library<jupyter_code>! pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint
! pip3 install --upgrade aiohttp_socks
import twint
import nest_asyncio
nest_asyncio.apply()<jupyter_output><empty_output><jupyter_text>- The first dataset based on the **keyword** (ماي اس تي سي) and also based on spesicfic date 2015 until jan of 2021<jupyter_code>c = twint.Config()
c.Search ='ماي اس تي سي'

#num of tweets
# c.Limit = 4000

#languge 
c.Lang='ar'


# Date 
c.Since = "2015-01-01 00:30:15"
c.Until = '2021-01-26 18:03:00'



# creating in a dataframe
c.Pandas=True
mystc_1 =  twint.storage.panda.Tweets_df
twint.run.Search(c)

mystc_1.shape<jupyter_output><empty_output><jupyter_text>- save data on csv file on google drive<jupyter_code>mystc_1.to_csv('mystc_1.csv')
! cp mystc_1.csv 'drive/My Drive/'
import pandas as pd
df1 = pd.read_csv("/content/drive/My Drive/mystc_1.csv",lineterminator='\n')<jupyter_output><empty_output><jupyter_text>- choose specific features<jupyter_code>df1=df1[['id','created_at',	'date' ,'tweet' ,'user_id' ,'name','retweet'	,'nlikes',	'nreplies',	'nretweets']]<jupyter_output><empty_output><jupyter_text>- The third dataset based on the **keyword** ( خدمة اس تي سي) and also based on spesicfic date 2020 until jan of 2021<jupyter_code>c = twint.Config()
c.Search = 'خدمة اس تي سي'

#num of tweets
# c.Limit = 4000

#languge 
c.Lang='ar'


# Date 
c.Since = "2015-01-01 00:30:15"
c.Until = '2021-01-26 18:03:00'



# creating in a dataframe
c.Pandas=True
stc_service_ver3 =  twint.storage.panda.Tweets_df
twint.run.Search(c)
stc_service_ver3.shape<jupyter_output><empty_output><jupyter_text>- save data on csv file on google drive<jupyter_code>stc_service_ver3.to_csv('stc_service_tweet_ver3.csv')
! cp stc_service_tweet_ver3.csv 'drive/My Drive/'
import pandas as pd
df3 = pd.read_csv("/content/drive/My Drive/stc_service_tweet_ver3.csv",lineterminator='\n')<jupyter_output><empty_output><jupyter_text>- choose specific features<jupyter_code>df3 = df3[['id','created_at',	'date' ,'tweet' ,'user_id' ,'name','retweet'	,'nlikes',	'nreplies',	'nretweets'	]]
df1.shape , df3.shape<jupyter_output><empty_output><jupyter_text>Concat the tow datasets of reviews<jupyter_code>import pandas as pd
stc = [df1, df3]
stc_info = pd.concat(stc)
stc_info.shape
stc_info.to_csv('stc_info_result11.csv')
! cp stc_info_result11.csv 'drive/My Drive/'
import pandas as pd
stc_all_tweets = pd.read_csv("/content/drive/My Drive/stc_info_result11.csv",lineterminator='\n' )
stc_all_tweets.drop('Unnamed: 0' , axis=1 , inplace=True)
stc_all_tweets.head()<jupyter_output><empty_output><jupyter_text>Check null values<jupyter_code>stc_all_tweets.isna().sum()
stc_all_tweets.dropna(inplace=True)<jupyter_output><empty_output><jupyter_text>Cleaning Data<jupyter_code>import re
# Removing mention 
stc_all_tweets['tweet']= stc_all_tweets['tweet'].apply(lambda x: re.sub("@\S+", " " , x)) 
# Removing hashtags 
stc_all_tweets['tweet']= stc_all_tweets['tweet'].apply(lambda x: re.sub("#\S+", " ", x)) 
# Removing special charachter 
stc_all_tweets['tweet']= stc_all_tweets['tweet'].apply(lambda x: re.sub(r'[؟|؟؟|?|$|.|!]',r'',x)) 
#Remove urls 
stc_all_tweets['tweet']= stc_all_tweets['tweet'].apply(lambda x:re.sub("https*\S+", " ", x)) 
#Remove over spaces 
stc_all_tweets['tweet']= stc_all_tweets['tweet'].apply(lambda x:re.sub('\s{2,}', " ", x)) 
# Remove 
stc_all_tweets['tweet']= stc_all_tweets['tweet'].apply(lambda x:re.sub(r'\w*\d+\w*', '', x)) 

stc_all_tweets<jupyter_output><empty_output><jupyter_text>Preprocessing Data<jupyter_code>import nltk 
from nltk import word_tokenize
from nltk.corpus import  stopwords
nltk.download('punkt')
nltk.download('stopwords')
stc_all_tweets['tweet2'] = stc_all_tweets['tweet'].apply(nltk.word_tokenize)
sw= stopwords.words('arabic')
stc_all_tweets['tweet3'] = stc_all_tweets['tweet2'].apply(lambda x:[i for i in x if i not in sw])
stc_all_tweets['tweet4']= stc_all_tweets['tweet3'].apply(lambda x: ' '.join([i for i in x]))
stc_all_tweets.drop(['tweet2' , 'tweet3'] , axis=1 , inplace=True )
stc_all_tweets.drop('tweet' ,axis=1 , inplace=True )
stc_all_tweets.head() # tweet4 after removing stopwords<jupyter_output><empty_output><jupyter_text>Removing all accounts that dose not related to our goal<jupyter_code>clean=['صحيفة','صحيفه' ,'اخبار', 'أخبار', 'قناة', 'قناه' , 'عاجل' ,'جريده', 'جريدة' ,'هام' ,'زيادة','متابعين']<jupyter_output><empty_output><jupyter_text>Searching for the accounts in our dataframe<jupyter_code>val=[]
for i  in  stc_all_tweets.name:
  print(i.index)
  for j in clean:
     if j in i: 
       o=getIndexes(stc_all_tweets,i)
       val.append(o)<jupyter_output><empty_output><jupyter_text>Function to get the index of the value<jupyter_code>def getIndexes(dfObj, value):
    ''' Get index positions of value in dataframe i.e. dfObj.'''
    listOfPos = list()
    # Get bool dataframe with True at positions where the given value exists
    result = dfObj.isin([value])
    # Get list of columns that contains the value
    seriesObj = result.any()
    columnNames = list(seriesObj[seriesObj == True].index)
    # Iterate over list of columns and fetch the rows indexes where value exists
    for col in columnNames:
        rows = list(result[col][result[col] == True].index)
        for row in rows:
            listOfPos.append((row, col))
    # Return a list of tuples indicating the positions of value in the dataframe
    return listOfPos<jupyter_output><empty_output><jupyter_text>Shape after removing the accounts<jupyter_code>#(10831, 10)<jupyter_output><empty_output><jupyter_text>Removing rows<jupyter_code>for i in range(len(val)):
  value=val[i][0][0]
  try:
     stc_all_tweets.drop( value , inplace=True ) 
  except:
    print('not deleted' , value)<jupyter_output><empty_output><jupyter_text>Shape before removing the accounts<jupyter_code>stc_all_tweets.shape
stc_all_tweets
stc_all_tweets.to_csv('stc_tweets_after_final_cleaning_ver5.csv')
! cp stc_tweets_after_final_cleaning_ver5.csv 'drive/My Drive/'
stc_tweet= pd.read_csv("/content/drive/My Drive/stc_tweets_after_final_cleaning_ver5.csv",lineterminator='\n' )
stc_tweet.drop('Unnamed: 0' , axis=1 , inplace=True)
stc_tweet.head()
stc_tweet.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16768832584116605, 'fraction_numerical': 0.024928483857785042, 'mean_word_length': 6.334665334665335, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '615949', 'license': 'Apache-2.0', 'n_tokens_mistral': 3079, 'n_tokens_neox': 2918, 'n_words': 540, 'path': 'Project3 - STC reviwes.ipynb', 'repo_name': 'samaher133/Projects-', 'size': '7341'}