<jupyter_start><jupyter_text>Scrape Now that the sections of the HTML code are known, start by scraping the first page of the thread.<jupyter_code>import requests
import urllib.request
from bs4 import BeautifulSoup
import pandas as pd

url = "https://forum.lowyat.net/topic/4828500"

response = requests.get(url)
response<jupyter_output><empty_output><jupyter_text>Scrape the comments next. Comments are in **`Lod incoming `** (i.e. div class called postcolor post_text). Parse Use `BeautifulSoup`’s `html.parser` to look for all `div` classes named `postcolor post_text`. Get the text within the HTML data using `get_text(). comments_list` should resemble the results as follows.<jupyter_code>soup = BeautifulSoup(response.text, "html.parser")
comments = soup.findAll("div",{'class':"postcolor post_text"})

comments_temp = list(comments)

comments_list = []

for i in range(len(comments)):
    comments_list.append(comments[i].get_text())

comments_list<jupyter_output><empty_output><jupyter_text>Now go for the usernames. Usernames are in **`Lyu`** (i.e. span class called normalname). Modify the `soup.findAll` code to look for all `span` classes named `normalname`.<jupyter_code>soup2 = BeautifulSoup(response.text, "html.parser")
names = soup2.findAll("span", {"class": "normalname"})
names_temp = list(names)
names_list = []

for i in range(len(names)):
    names_list.append(names[i].get_text())

names_list<jupyter_output><empty_output><jupyter_text>Now that the usernames and comments have been obtained, merge both lists into a Python dataframe.<jupyter_code>compiled_list = list(zip(names_list, comments_list))
dataframe = pd.DataFrame(compiled_list, columns = ["Person", "Comment"])
dataframe<jupyter_output><empty_output><jupyter_text>Your table should look like the snapshot above. Now that we have the scraping algorithm, the `url` needs to be modified, since this thread has multiple pages. For page 1, the URL is https://forum.lowyat.net/topic/4828500For page 2, the URL is https://forum.lowyat.net/topic/4828500/+20For page 3, the URL is https://forum.lowyat.net/topic/4828500/+40There’s a pattern in the URL. Notice the end of the URL increases by 20 for subsequent pages. This must be some sort of comment count, since every page contains 20 comments. The URL modification can be achieved as follows.<jupyter_code>import requests
import urllib.request
from bs4 import BeautifulSoup
import pandas as pd

pages = ["/+0", "/+20", "/+40", "/+60"]
comments_list = []
names_list = []

for i in range(0, len(pages)):
  url = "https://forum.lowyat.net/topic/4828500"+pages[i]
  response = requests.get(url)
  soup = BeautifulSoup(response.text, "html.parser")
  comments = soup.findAll("div",{'class':"postcolor post_text"})
  comments_temp = list(comments)

  for i in range(len(comments_temp)):
    comments_list.append(comments_temp[i].get_text())

  names = soup.findAll("span", {"class": "normalname"})
  names_temp = list(names)

  for i in range(len(names_temp)):
    names_list.append(names_temp[i].get_text())

compiled_list = list(zip(names_list, comments_list))

dataframe = pd.DataFrame(compiled_list, columns = ["Person", "Comment"])

dataframe<jupyter_output><empty_output><jupyter_text>Notice again, the code built previously served as a “foundation” for the code above. This feature is common across many data projects. Whenever you are unsure whether you can achieve a certain task, break it down by trying to get a small portion of the data first. Once this is completed, scale up your code to get all the data. Notice this dataframe has 63 rows. It indicates you now have all the comments from this particular thread.  Sentiment Analysis Now for the exciting part. The comments have been obtained - they need to be classified (into Positive, Negative, or Neutral) and scored (i.e. not all negative comments are the same as some will be more serious than others, and likewise for positive comments). There are multiple libraries available for sentiment analysis of English comments: 1.   [NLTK’s sentiment](https://www.nltk.org/howto/)2.   [Google Cloud Natural Language API’s Sentiment Analysis](https://cloud.google.com/natural-language/docs/sentiment-tutorial)3.   [scikit-learn’s classifiers](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html)and so on.For this simple demonstration, I’ll be using the [afinn](https://github.com/fnielsen/afinn) classifier to obtain some business insights. Once this has been achieved, do feel free to test out other Python libraries for improved results. In your IDE, it’s likely afinn is not installed yet. In Google Colab, run the following code to install and initialize afinn.<jupyter_code>!pip install afinn
from afinn import Afinn
af = Afinn()<jupyter_output>Collecting afinn
[?25l  Downloading https://files.pythonhosted.org/packages/86/e5/ffbb7ee3cca21ac6d310ac01944fb163c20030b45bda25421d725d8a859a/afinn-0.1.tar.gz (52kB)

[K     |██████▎                         | 10kB 15.8MB/s eta 0:00:01
[K     |████████████▌                   | 20kB 1.7MB/s eta 0:00:01
[K     |██████████████████▊             | 30kB 2.2MB/s eta 0:00:01
[K     |█████████████████████████       | 40kB 2.5MB/s eta 0:00:01
[K     |███████████████████████████████▏| 51kB 2.0MB/s eta 0:00:01
[K     |████████████████████████████████| 61kB 1.8MB/s 
[?25hBuilding wheels for collected packages: afinn
  Building wheel for afinn (setup.py) ... [?25l[?25hdone
  Created wheel for afinn: filename=afinn-0.1-cp36-none-any.whl size=53452 sha256=94bcda66e9633b4d6ff09f1d05bd5d5f83b9c1a996a46b87d317b0e748a81cdc
  Stored in directory: /root/.cache/pip/wheels/b5/1c/de/428301f3333ca509dcf20ff358690eb23a1388fbcbbde008b2
Successfully built afinn
Installing collected packages: afinn
Succes[...]<jupyter_text>The next step is to create an empty array to store the sentiment results. Do create a sentiment category too. It’s to classify as `positive` comments which have scores above 0, `negative` for scores below 0, and `neutral` for scores which are 0.<jupyter_code>sentiment_scores = []

for i in range(len(dataframe)):
  sentiment_scores.append(af.score(dataframe.iloc[i, 1]))

sentiment_category = ['positive' if score > 0 
                           else 'negative' if score < 0 
                               else 'neutral' 
                                   for score in sentiment_scores]

sentiment_scores<jupyter_output><empty_output><jupyter_text>The next step is to merge `sentiment_scores` and `sentiment_category` as new columns to `dataframe`. The steps to do that are as follows:1.   Turn `sentiment_scores` into a dataframe2.   Turn `sentiment_category` into a dataframe3.   Merge `sentiment_scores` and `sentiment_category` to dataframe4.   Display the merged dataframe and export the file as csv<jupyter_code>sentiment_df = pd.DataFrame(sentiment_scores, columns = {'Sentiment_Scores'})

category_df = pd.DataFrame(sentiment_category, columns = {"Sentiment Category"})

df = pd.concat([dataframe, sentiment_df, category_df], axis = 1)

df['Sentiment_Scores'] = df.Sentiment_Scores.astype('float')

df
df.to_csv("complaints.csv")
from google.colab import files
files.download("complaints.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11868162304389973, 'fraction_numerical': 0.035452153441351615, 'mean_word_length': 4.895510204081632, 'pattern_counts': {'":': 2, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '408710', 'license': 'MIT', 'n_tokens_mistral': 2582, 'n_tokens_neox': 2290, 'n_words': 707, 'path': 'Chapter_5_Sentiment_analysis_using_Lowyat_NET_comments.ipynb', 'repo_name': 'atlas-github/fi_analytics', 'size': '7221'}