<jupyter_start><jupyter_text>Importing Reddit's API, PRAW to scrap subreddits, and Pandas<jupyter_code>import praw
import pandas as pd<jupyter_output><empty_output><jupyter_text>Initialising the Reddit object with user's credentials<jupyter_code>reddit = praw.Reddit(client_id = "#", 
                     client_secret = "#", 
                     user_agent = "#")<jupyter_output><empty_output><jupyter_text>Choosing the subreddit of r/india<jupyter_code>subred = reddit.subreddit('india')<jupyter_output><empty_output><jupyter_text>Considering "hot", "top", "new" and "controversial" filters to scrap subreddits<jupyter_code>hot = subred.hot(limit=3000)
top = subred.top(limit=3000)
controversial = subred.controversial(limit=3000)
new = subred.new(limit=3000)
type(hot)<jupyter_output><empty_output><jupyter_text>Initilizing a dictionary "data" to store the scrapped data<jupyter_code>data = {"id":[], "url":[],"title":[],  "body":[], "flair":[]}<jupyter_output><empty_output><jupyter_text>Getting data from corresponding filtered subreddits and appending in "data"<jupyter_code>for i in hot:
    
    data['id'].append(i.id)
    data['url'].append(i.url)
    data['title'].append(i.title)
    data['body'].append(i.selftext)    
    data['flair'].append(i.link_flair_text)    
for i in top:
    
    data['id'].append(i.id)
    data['url'].append(i.url)
    data['title'].append(i.title)
    data['body'].append(i.selftext)    
    data['flair'].append(i.link_flair_text) 
for i in controversial:
    
    data['id'].append(i.id)
    data['url'].append(i.url)
    data['title'].append(i.title)
    data['body'].append(i.selftext)    
    data['flair'].append(i.link_flair_text) 
for i in new:
    
    data['id'].append(i.id)
    data['url'].append(i.url)
    data['title'].append(i.title)
    data['body'].append(i.selftext)    
    data['flair'].append(i.link_flair_text)<jupyter_output><empty_output><jupyter_text>Converting the dictionary to a Pandas DataFrame<jupyter_code>import pandas as pd
df = pd.DataFrame(data)
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Since one subreddit can appear in more than one filter, the DF is sorted and duplicate elements are dropped.<jupyter_code>df.sort_values("id", inplace = True)
df.drop_duplicates(subset ="id", inplace = True)
df.shape<jupyter_output><empty_output><jupyter_text>Analyzing the unique flairs and their frequencies from the collected data<jupyter_code>df['flair'].value_counts()<jupyter_output><empty_output><jupyter_text>Choosing the top 10 flairs from the scrapped list as the categories for the succeeding prediction task.<jupyter_code>top_flairs = ["Politics", "Non-Political", "Coronavirus", "AskIndia", "Policy/Economy", "[R]eddiquette", 
              "Photography", "Business/Finance", "Sports", "Science/Technology","Food"]

df_top = df.loc[df['flair'].isin(top_flairs)]
df_top.shape
df_top.head()<jupyter_output><empty_output><jupyter_text>As the data is skewed towards certain flairs and not others, we are scrapping again from the r/india subreddit for the above mentioned flairs with a limit of 100 for each flairs<jupyter_code>reddit = praw.Reddit(client_id = "#", 
                     client_secret = "#", 
                     user_agent = "#")

subred = reddit.subreddit('india')
sample_data = {"id":[], "url":[], "title":[], "body":[], "flair":[]}

top_flairs = ["Politics", "Non-Political", "Coronavirus", "AskIndia", "Policy/Economy", "[R]eddiquette", 
              "Photography", "Business/Finance", "Sports", "Science/Technology"]

for flair in top_flairs:
  
  top_f = subred.search(flair, limit=1000)
  
  for i in top_f:
    
    sample_data["id"].append(i.id)
    sample_data["url"].append(i.url)
    sample_data["title"].append(i.title)
    sample_data["body"].append(i.selftext)
    sample_data["flair"].append(flair)
    
sample = pd.DataFrame(sample_data)

sample.head()
sample['flair'].value_counts()<jupyter_output><empty_output><jupyter_text>It is seen that "[R]eddiquette"  flair has very less samples. Even if we use the previously scrapped data and concatenate it, it wont be in the same scale as others, so we will drop that category.<jupyter_code>top_9 = ["Politics", "Non-Political", "Coronavirus", "AskIndia", "Policy/Economy", 
              "Photography", "Business/Finance", "Sports", "Science/Technology"]

final = sample.loc[sample['flair'].isin(top_9)]
final['flair'].value_counts()<jupyter_output><empty_output><jupyter_text>Removing duplicate entries<jupyter_code>final.sort_values("id", inplace = True)
final.drop_duplicates(subset ="id", inplace = True)
final['flair'].value_counts()<jupyter_output><empty_output><jupyter_text>Converting the DataFrame into a CSV file<jupyter_code>final.to_csv('Data/reddit-top-flairs.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14868804664723032, 'fraction_numerical': 0.005622657226155768, 'mean_word_length': 5.378486055776892, 'pattern_counts': {'":': 10, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '26938', 'license': 'MIT', 'n_tokens_mistral': 1755, 'n_tokens_neox': 1672, 'n_words': 316, 'path': 'Models/Part I - Reddit Data Collection.ipynb', 'repo_name': 'sgouda789/Reddit-Flair-Detection', 'size': '4802'}