<jupyter_start><jupyter_text>Read the data from subreddit posts.At the following, a function is called which gathers data from sub-redit sites. The following sub-Reddit details are used:* subreddit: which is **sub-Reddit name*** num: **number of sub-Reddit** which is read by the function* time: which is used as a **start reading time** Install some libraries as it would be needed.<jupyter_code># !pip install BeautifulSoup4<jupyter_output><empty_output><jupyter_text>Install necessary libraries.<jupyter_code>import sys # always define the sys at the begining
import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import time
from datetime import datetime<jupyter_output><empty_output><jupyter_text>The read_redit function is provided in the Asset folder. The function is imported by inserting the directory into the current path at the following.<jupyter_code># inserting the parent directory into current path
sys.path.insert(1, '../Assets')<jupyter_output><empty_output><jupyter_text>Import read_redit function from the Asset folder now.<jupyter_code>import read_redit<jupyter_output><empty_output><jupyter_text>Determine the objective reading details as follow.<jupyter_code>subreddit = 'space'
num = 6000
time= int(time.mktime(time.strptime('19 January, 2020', '%d %B, %Y')))
size_limi = 100<jupyter_output><empty_output><jupyter_text>Read data using the sub_redit and insert it into the df.<jupyter_code>df_space = read_redit.get_posts(subreddit, num, time, size_limi)<jupyter_output>500 posts have been gathered, oldest post is from 2020-01-12 12:31:04
1000 posts have been gathered, oldest post is from 2020-01-07 09:17:20
1500 posts have been gathered, oldest post is from 2020-01-01 16:21:25
2000 posts have been gathered, oldest post is from 2019-12-26 19:12:59
2500 posts have been gathered, oldest post is from 2019-12-20 19:53:15
3000 posts have been gathered, oldest post is from 2019-12-14 23:04:19
3500 posts have been gathered, oldest post is from 2019-12-07 16:32:44
4000 posts have been gathered, oldest post is from 2019-12-01 02:58:14
4500 posts have been gathered, oldest post is from 2019-11-24 15:52:10
5000 posts have been gathered, oldest post is from 2019-11-19 08:21:14
5500 posts have been gathered, oldest post is from 2019-11-14 10:11:56
6000 posts have been gathered, oldest post is from 2019-11-09 16:13:20<jupyter_text>Check out the data frame first five rows, the shape, and the info as follows.<jupyter_code>df_space.head()
df_space.shape
df_space.info()
file_path = "../DataSet/"
file_name = "df_space.csv"
df_space.to_csv(file_path+file_name)<jupyter_output><empty_output><jupyter_text>Read the next class data from subredit.<jupyter_code>import time
from datetime import datetime
subreddit = 'nasa'
num = 6000
time= int(time.mktime(time.strptime('19 January, 2020', '%d %B, %Y')))
size_limi = 100
df_nasa = read_redit.get_posts(subreddit, num, time, size_limi)
df_nasa.head()
df_nasa.shape
df_nasa.info()
file_path = "../DataSet/"
file_name = "df_nasa.csv"
df_nasa.to_csv(file_path+file_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09076227390180878, 'fraction_numerical': 0.07881136950904392, 'mean_word_length': 6.185614849187935, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '445198', 'license': 'MIT', 'n_tokens_mistral': 1174, 'n_tokens_neox': 996, 'n_words': 350, 'path': 'Codes/P01_Gathering_Data.ipynb', 'repo_name': 'Atashnezhad/Natural_language_processing_Project', 'size': '3096'}