<jupyter_start><jupyter_text>Text Classification Notebook applied on MoneyControl Dataset Name: <NAME> Roll No.: PC-45 Serial No.: 1032171054 Demo for sentiment analysis for seminar topic Do refer the dataset scraping notebook to get a better feel of the data.<jupyter_code># Importing all the necessary libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
#loading our dataset of news articles freshly scrapped from moneycontrol website
df1 = pd.read_csv('MoneyControl-First-1-10-Pages.csv')
df2 = pd.read_csv('MoneyControl-First-11-20-Pages.csv')
df = pd.concat([df1,df2])
# Removing unnecessary columns
df = df.iloc[:,1:]
df.head()
# Searching for empty cells
df.isnull().sum()
# We do not require authors as of yet so we can ignore those missing values
# The following model will be applied on titles only
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 457 entries, 0 to 218
Data columns (total 8 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   time          457 non-null    object
 1   Label         457 non-null    int64 
 2   author        425 non-null    object
 3   title         457 non-null    object
 4   summary       457 non-null    object
 5   description   449 non-null    object
 6   article_text  457 non-null    object
 7   url           457 non-null    object
dtypes: int64(1), object(7)
memory usage: 32.1+ KB<jupyter_text>Dataset news articles have been labeled are as follows:    -2 Label Score -> Very Negative News    -1 Label Score -> Negative News     0 Label Score -> Neutral News     1 Label Score -> Positive News     2 Label Score -> Very Positive News     Since so many classes will not be correctly classified with the help of simpler machine learning models we will map the scores as follows...    -1 Label Score -> Negative News     1 Label Score -> Positive News<jupyter_code>maps = {2: 1, 1: 1, 0: 0, -1: -1, -2: -1}
x = [1, 2, -1, -1, -2, 0]
df.Label = [maps[i] for i in df.Label]
df01 = df[df.Label != 0]
X = df01['title']
y = df01['Label']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 0)
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
text_clf = Pipeline([('tfidf', TfidfVectorizer()), ('svc', LinearSVC())])
text_clf.fit(X_train, y_train)
predictions = text_clf.predict(X_test)
from sklearn import metrics
metrics.confusion_matrix(predictions, y_test)
metrics.accuracy_score(predictions, y_test)<jupyter_output><empty_output><jupyter_text>Trying to explain poor accuracy of negative news articles    Looking at the histogram below it is clear that the number of positive news articles far exceeds the negative news articles, hence this maybe one of the possible explantions that there does not exist enough training data for the negative news articles.    Solution for the same: Scrape from more websites or simply add in more negative news articles.<jupyter_code>plt.hist(df01.Label)<jupyter_output><empty_output><jupyter_text>Examples of the model in action    We provide the model with unseen headlines and it shall classify the same as positive and negative, do recall that...    -1 Label Score Means news article is negative     1 Label Score Means news article is positive<jupyter_code>text_clf.predict(["Pharma sector stocks gain 20% amid Covid-19 pandemic"])
text_clf.predict(["SENSEX down by 1450 points"])
text_clf.predict(["Kotak Equities suggests buy on IBM stock."])
text_clf.predict(["Tata Motors falls 8% auto markets in decline. "])
df.to_csv("MoneyControl-First-1-20")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08891855807743658, 'fraction_numerical': 0.033110814419225636, 'mean_word_length': 3.9224704336399476, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '94756', 'license': 'MIT', 'n_tokens_mistral': 1206, 'n_tokens_neox': 1106, 'n_words': 445, 'path': 'Text Classification Money Control.ipynb', 'repo_name': 'mahajanhrishikesh/Financial-Document-Sentiment-Analysis', 'size': '3764'}