<jupyter_start><jupyter_text>The global warming issue and Narratives around it Part 1: Pulling data from web API using pushshiftIn this notebook, I imported the posts from the following subreddits:     1- [Globalwarming](https://www.reddit.com/r/GlobalWarming/) >: With 6.2k subscribers (Created on May 28, 2008).     2- [Conspiracytheory](https://www.reddit.com/r/ConspiracyTheory/) : with 3.7k subscribers (Created on Dec 11, 2009).     - **Problem statement:** The reddit users in **[Globalwarming]** subreddit were in general concerned, while the subreddit users in **[Conspiracytheory]** reddit were more interested in raising conspiracy theories. This contrasting viewpoint may be a good binary class target for our NLP analysis, to develop a trained model.  - The imported posts were converted to DataFrame and later saved into "../datasets" folder for further processing Importing the required libraries:- Built a general function to read reddit APIs: "get_reddit_posts.py"    ("Pushshift API" serves the purpose in pulling the reddit posts)<jupyter_code>#imports
import requests
import pandas as pd
import time

#Importing the built function, prior to that added the assets path to the system path
#Inspiration: https://stackoverflow.com/questions/4383571/importing-files-from-different-folder

import sys
# inserting the parent directory into current path
sys.path.insert(1, '../assets')

from get_reddit_posts import get_reddit_posts

import pickle<jupyter_output><empty_output><jupyter_text>---  Part 1.1. Reading in the Global warming API Calling in the created function and reading global warming data:<jupyter_code>#Defining API pull initial parameteres:

par = {"subreddit": "GlobalWarming", #The subreddit title
       "post_num": 4000, # Numer of posts to pull from
        "time_1": int(time.mktime(time.strptime('1 July, 2020', '%d %B, %Y'))), # The latest pull time
       "API_limit": 100, # API pull number limits for reddit per time
       "API_wait": 1 #API wait time berfore the next pull
      }



climate_change = get_reddit_posts(par["subreddit"], par["post_num"], par["time_1"], par[ "API_limit"], par["API_wait"])
climate_change.head(5)
climate_change.shape<jupyter_output><empty_output><jupyter_text>Saving the dataframe into CSV<jupyter_code>file_path = "../datasets/" + par["subreddit"] + "_raw" + ".csv"
climate_change.to_csv(file_path)<jupyter_output><empty_output><jupyter_text>---  Part 1.2. Reading in the ConspiracyTheory API Calling in the created function and reading conspiracyTheory data:<jupyter_code>#Defining API pull initial parameteres:

par = {"subreddit": "ConspiracyTheory", #The subreddit title
       "post_num": 900, # Numer of posts to pull from (limited reddit)
        "time_1": int(time.mktime(time.strptime('1 July, 2020', '%d %B, %Y'))), # The latest pull time
       "API_limit": 100, # API pull number limits for reddit per time
       "API_wait": 1 #API wait time berfore the next pull
      }



cons_theory = get_reddit_posts(par["subreddit"], par["post_num"], par["time_1"], par[ "API_limit"], par["API_wait"])
cons_theory.head(5)
cons_theory.shape<jupyter_output><empty_output><jupyter_text>Saving the dataframe into csv<jupyter_code>file_path = "../datasets/" + par["subreddit"] + "_raw" + ".csv"
cons_theory.to_csv(file_path)<jupyter_output><empty_output><jupyter_text>Now, the data is ready for the next steps<jupyter_code>print("Hellow world!")<jupyter_output>Hellow world!
{'fraction_non_alphanumeric': 0.11027278003482298, 'fraction_numerical': 0.017991874637260593, 'mean_word_length': 5.2672727272727276, 'pattern_counts': {'":': 10, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '528994', 'license': 'MIT', 'n_tokens_mistral': 1120, 'n_tokens_neox': 1062, 'n_words': 351, 'path': 'Codes/P01_API_requests.ipynb', 'repo_name': 'saiedmighani/Global_warming_NLP_analysis', 'size': '3446'}