<jupyter_start><jupyter_text>***READING THE WEB PAGE INTO PYTHON***<jupyter_code>import requests
# Fetch the webpage from the URL & store it in a response object (r)
r = requests.get ('https://www.nytimes.com/interactive/2017/06/23/opinion/trumps-lies.html')
# Print the 1st 500 characters of the URL
print (r.text [0:500])<jupyter_output><!DOCTYPE html>
<!--[if (gt IE 9)|!(IE)]> <!--><html lang="en" class="no-js page-interactive section-opinion page-theme-standard tone-opinion page-interactive-default limit-small layout-xlarge app-interactive" itemid="https://www.nytimes.com/interactive/2017/06/23/opinion/trumps-lies.html" itemtype="http://schema.org/NewsArticle" itemscope xmlns:og="http://opengraphprotocol.org/schema/"><!--<![endif]-->
<!--[if IE 9]> <html lang="en" class="no-js ie9 lt-ie10 page-interactive section-opinion page<jupyter_text>***PARSE THE HTML USING BEAUTIFUL SOUP LIBRARY***<jupyter_code>from bs4 import BeautifulSoup
# Parse the HTML stored in the object text into a special object called soup
soup = BeautifulSoup (r.text, 'html.parser')<jupyter_output><empty_output><jupyter_text>***COLLECTING ALL THE RECORDS***<jupyter_code># FORMAT OF EACH RECORD
# This is the pattern that allows us to build our dataset
'''<span class = "short-decs"><strong> DATE </strong> LIE <span class = "short-truth">
<a href = "URL"> EXPLANATION </a></span></span>'''
# Find all the records using Beautiful Soup
# This code searches the soup object for all span tags with the attribute class = "short-decs"
# A special Beautiful Soup object is returned called results set containing the search results
results = soup.find_all('span', attrs={'class':'short-desc'})
# results acts like a Python list
len (results)
print("Title of the website is : ")
for title in soup.find_all('title'):
    print(title.get_text())
# Slice the object like a list, in order to examine the first three results.
results [0:3]
# Last result
results [-1]<jupyter_output><empty_output><jupyter_text>***EXTRACTING THE DATE***<jupyter_code># First record
first_result = results [0]
first_result<jupyter_output><empty_output><jupyter_text>*In order to locate the date, we can use its find() method to find a single tag that matches a specific pattern, in contrast to the find_all() method we used above to find all tags that match a pattern:*<jupyter_code>'''This code searches first_result for the first instance of a <strong> tag, 
and again returns a Beautiful Soup "Tag" object (not a string).'''
first_result.find ('strong')
'''Since we want to extract the text between the opening and closing tags, 
we can access its text attribute, which does in fact return a regular Python string:'''
first_result.find ('strong').text 
'''xa0 is called an "escape sequence" that represents the &nbsp; 
character we saw earlier in the HTML source.'''
# Slice it off from the end of the string
first_result.find ('strong').text [0:-1]
'''Finally, we're going to add the year, 
since we don't want our dataset to include ambiguous dates:'''
first_result.find ('strong').text [0:-1] + ', 2017'<jupyter_output><empty_output><jupyter_text>***EXTRACTING THE LIE***<jupyter_code>first_result<jupyter_output><empty_output><jupyter_text>*Our goal is to extract the two sentences about Iraq. Unfortunately, there isn't a pair of opening and closing tags that starts immediately before the lie and ends immediately after the lie. Therefore, we're going to have to use a different technique :*<jupyter_code>first_result.contents
# Slice the list to extract the 2nd element
first_result.contents [1]  # 1 since Python follows zero-indexing
'''Finally, we'll slice off the curly quotation marks as well as the extra space at the end:'''
first_result.contents [1] [1:-2]<jupyter_output><empty_output><jupyter_text>***EXTRACTING THE EXPLANATION***<jupyter_code># The first option is to slice the contents attribute
first_result.contents [2]
# The second option is to search for the surrounding tag, like we did when extracting the date:
first_result.find ('a')
'''Either way, we can access the text attribute and then slice off 
the opening and closing parentheses:'''
first_result.find ('a').text [1:-1]<jupyter_output><empty_output><jupyter_text>***EXTRACTING THE URL***<jupyter_code>'''Extract the URL of the article that substantiates the writer's claim that the President was lying.

Let's examine the <a> tag within first_result:'''
first_result.find ('a')<jupyter_output><empty_output><jupyter_text>*In this case, the text we want to extract is located within the tag itself. Specifically, we want to access the value of the href attribute within the  tag.Beautiful Soup treats tag attributes and their values like key-value pairs in a dictionary: you put the attribute name in brackets (like a dictionary key), and you get back the attribute's value:*<jupyter_code>first_result.find ('a') ['href']<jupyter_output><empty_output><jupyter_text>**You can apply these two methods to either the initial soup object or a Tag object (such as first_result):find(): searches for the first matching tag, and returns a Tag objectfind_all(): searches for all matching tags, and returns a ResultSet object (which you can treat like a list of Tags)You can extract information from a Tag object (such as first_result) using these two attributes:text: extracts the text of a Tag, and returns a stringcontents: extracts the children of a Tag, and returns a list of Tags and strings** ***BUILDING THE DATASET***<jupyter_code># Store the output in the form of empty tuples
records = []
# Create a loop to repeat the same process of extracting the 4 components on all results
# Loop through the results
for result in results : 
  date = result.find('strong').text[0:-1] + ', 2017'
  lie = result.contents[1][1:-2]
  explanation = result.find('a').text[1:-1]
  url = result.find('a')['href'] 
# Append the results in the records list
  records.append ((date,lie,explanation,url))
# Since there were 180 results, we should have 180 records
len (records)
# 1st 3 records
records [0:3]<jupyter_output><empty_output><jupyter_text>***APPLYING A TABULAR DATA STRUCTURE*** *The last major step in this process is to apply a tabular data structure to our existing structure (which is a list of tuples). We're going to do this using the pandas library. The primary data structure in pandas is the "DataFrame", which is suitable for tabular data with columns of different types.*<jupyter_code># We can convert our list of tuples into a DataFrame by passing it to the 
# DataFrame constructor and specifying the desired column names.
import pandas as pd
df = pd.DataFrame (records, columns = ['date','lie','explanation','url'])
# Top of the DataFrame
df.head ()
# Bottom of the DataFrame
df.tail ()
# Convert the date column to pandas' special "datetime" format
df['date'] = pd.to_datetime(df['date'])
df<jupyter_output><empty_output><jupyter_text>***EXPORTING THE DATASET TO A CSV FILE*** *We'll use pandas to export the DataFrame to a CSV (comma-separated value) file, which is the simplest and most common way to store tabular data in a text file*<jupyter_code>df.to_csv ('trump_lies.csv', index = False, encoding = 'utf-8')
# Read the CSV file back into pandas
df = pd.read_csv('trump_lies.csv', parse_dates=['date'], encoding='utf-8')
# Summary (16 Lines Of Code)

# import requests
# r = requests.get('https://www.nytimes.com/interactive/2017/06/23/opinion/trumps-lies.html')

# from bs4 import BeautifulSoup
# soup = BeautifulSoup(r.text, 'html.parser')
# results = soup.find_all('span', attrs={'class':'short-desc'})

# records = []
# for result in results:
#     date = result.find('strong').text[0:-1] + ', 2017'
#     lie = result.contents[1][1:-2]
#     explanation = result.find('a').text[1:-1]
#     url = result.find('a')['href']
#     records.append((date, lie, explanation, url))

# import pandas as pd
# df = pd.DataFrame(records, columns=['date', 'lie', 'explanation', 'url'])
# df['date'] = pd.to_datetime(df['date'])
# df.to_csv('trump_lies.csv', index=False, encoding='utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11674937965260546, 'fraction_numerical': 0.01228287841191067, 'mean_word_length': 5.5165723524656425, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 68, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '150035', 'license': 'MIT', 'n_tokens_mistral': 2528, 'n_tokens_neox': 2400, 'n_words': 942, 'path': "WEB_SCRAPING_(Trump's_Lies).ipynb", 'repo_name': 'Swayms-stack/Super-coder', 'size': '8060'}