<jupyter_start><jupyter_text>Importing required libraries<jupyter_code>import os
import glob
import pandas as pd<jupyter_output><empty_output><jupyter_text>Here, glob method from the module glob returns a list of all file paths that match the regex pattern *.txt  This stores all file paths from the folder pos into a list. And then a loop is used to open each text file from the path and the text in that file is appended to the list text. Finally a dataframe of positive reviews is created<jupyter_code>os.chdir("/.../Chapter 11/CS - IMDB Classification")
os.getcwd()

path="/.../Chapter 11/CS - IMDB Classification/txt_sentoken/pos/*.txt"

files = glob.glob(path)
text_pos = []

for p in files:
    file_read = open(p, "r")
    to_append_pos = file_read.read()
    text_pos.append(to_append_pos)
    file_read.close()

df_pos = pd.DataFrame({'text':text_pos,'label':'positive'})
df_pos.head()<jupyter_output><empty_output><jupyter_text>Similarly reading negative reviews, a data frame is created<jupyter_code>path="/.../Chapter 11/CS - IMDB Classification/txt_sentoken/neg/*.txt"

files = glob.glob(path)
text_neg = []

for n in files:
    file_read = open(n, "r")
    to_append_neg = file_read.read()
    text_neg.append(to_append_neg)
    file_read.close()


df_neg = pd.DataFrame({'text':text_neg,'label':'negative'})
df_neg.head()<jupyter_output><empty_output><jupyter_text>Concatenating df_pos and df_neg<jupyter_code>df_moviereviews=pd.concat([df_pos, df_neg])
print(df_moviereviews.head())
print(df_moviereviews.tail())<jupyter_output>label                                               text
0  positive  assume nothing . \nthe phrase is perhaps one o...
1  positive  plot : <NAME> is a male model . \nhe ...
2  positive  i actually am a fan of the original 1961 or so...
3  positive  a movie that's been as highly built up as the ...
4  positive   " good will hunting " is two movies in one : ...
        label                                               text
995  negative  synopsis : when a meteorite crashlands in the ...
996  negative  it's now the anniversary of the slayings of ju...
997  negative  coinciding with the emerging popularity of mov...
998  negative  and now the high-flying hong kong style of fil...
999  negative  battlefield long , boring and just plain stupi...<jupyter_text>Since we concatenated df_pos and df_neg as it is, the first half of them are labelled positive and the bottom half is negative.So in order to create randomness, we first shuffle the rows of the new dataframe and then renumber the row index<jupyter_code>from sklearn.utils import shuffle

df_moviereviews=shuffle(df_moviereviews)
df_moviereviews.head(10)
df_moviereviews.index = range(df_moviereviews.shape[0])
df_moviereviews.head(10)<jupyter_output><empty_output><jupyter_text>Writing this data frame to a csv file for future uses<jupyter_code>df_moviereviews.shape
df_moviereviews.to_csv("/.../Chapter 11/CS - IMDB Classification/txt_sentoken/Data_IMDB.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09363545484838387, 'fraction_numerical': 0.012329223592135955, 'mean_word_length': 4.0200668896321075, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '357062', 'license': 'MIT', 'n_tokens_mistral': 993, 'n_tokens_neox': 948, 'n_words': 304, 'path': 'Chapter11/CS - IMDB Classification/[To submit]IMDB_Read_Data.ipynb', 'repo_name': 'PacktPublishing/Ensemble-Machine-Learning-Cookbook', 'size': '3010'}