<jupyter_start><jupyter_text>Master Data Science for Business - Data Science Consulting - Session 2  Notebook 1:  Introduction to Web Scraping with ScrapyThis notebook aims at explaining the basics of scraping a website using the Python package Scrapy.   Useful ressources: -The officiel tutorial: https://docs.scrapy.org/en/latest/intro/tutorial.html -Tutorial to use Scrapy within a Jupyter Notebook: https://www.jitsejan.com/using-scrapy-in-jupyter-notebook.html  **To Do**: Run all the cells, and prepare a short explanation of what the Scrapy code is doing in the 3rd part, "Our first spider". Don't spend time looking at the other parts today, since these are mainly scripts to use Scrapy inside a Jupyter Notebook.   1. Importing packages<jupyter_code>import scrapy
from scrapy.crawler import CrawlerProcess
import json
import logging
import pandas as pd<jupyter_output><empty_output><jupyter_text>2. Creating the JSon pipeline<jupyter_code>#JSon pipeline, you can rename the "trustpilot.jl" to the name of your choice
class JsonWriterPipeline(object):

    def open_spider(self, spider):
        self.file = open('trustpilot.jl', 'w')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item<jupyter_output><empty_output><jupyter_text>3. Our first spider Scrapy uses spiders that crawl through the web to get what we want as data. A spider is a piece of code allowing to define the websites we want to scrape, as well as defining the elements you want such as titles, author, content, images...<jupyter_code>#We create a class with the name of our Spider
class TrustSpider(scrapy.Spider):
    name = "trust"
    #Adding URLs to scrap 
    start_urls = [
        'https://fr.trustpilot.com/review/www.centerparcs.fr/fr-fr',
    ]
    #Custom settings to modify settings usually found in the settings.py file 
    custom_settings = {
        'LOG_LEVEL': logging.WARNING,
        'ITEM_PIPELINES': {'__main__.JsonWriterPipeline': 1}, # Used for pipeline 1
        'FEED_FORMAT':'json',                                 # Used for pipeline 2
        'FEED_URI': 'trustpilot.json'                        # Used for pipeline 2
    }
    #The following function parses the web pages to get the elements 
    #This is the main function we want you to understand
    def parse(self, response):
        stars = 'article.review section.review__content div.review-content div.review-content__header div.review-content-header'
        date = 'article.review section.review__content div.review-content div.review-content__header div.review-content-header__dates script::text'
        for review in response.css('article.review'):
            nb_stars = review.css(stars).extract_first()[77] if review.css(stars).extract_first() is not None else -1
            pub_date = review.css(date).extract_first()[19:38] if review.css(stars).extract_first() is not None else -1
            yield {
                'title': review.css('a.link.link--large.link--dark::text').extract_first(),
                'content': review.css('p.review-content__text::text').extract_first(),
                'author': review.css('div.consumer-information__name::text').extract_first(),
                'date': pub_date,
                'stars': nb_stars,
            }<jupyter_output><empty_output><jupyter_text>4. CrawlingThe following code is used to launch the crawling of the spider.  **Warning**: You can execute the process only once and for one spider per notebook. If you want to relaunch the process, you have to "Restart and run all", otherwise you will get an error.<jupyter_code>process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(TrustSpider)
process.start()<jupyter_output>2019-01-21 19:04:22 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)
2019-01-21 19:04:22 [scrapy.utils.log] INFO: Versions: lxml 4.3.0.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.19.0, Twisted 17.5.0, Python 3.6.8 |Anaconda, Inc.| (default, Dec 29 2018, 19:04:46) - [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 18.0.0 (OpenSSL 1.1.1a  20 Nov 2018), cryptography 2.4.2, Platform Darwin-15.6.0-x86_64-i386-64bit
2019-01-21 19:04:22 [scrapy.crawler] INFO: Overridden settings: {'FEED_FORMAT': 'json', 'FEED_URI': 'trustpilot.json', 'LOG_LEVEL': 30, 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'}<jupyter_text>5. Loading data into Pandas' dataframe<jupyter_code>dfjson = pd.read_json('trustpilot.json')
#Preview of the dataframe
dfjson.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10922226944326392, 'fraction_numerical': 0.032936676583085425, 'mean_word_length': 3.6511857707509883, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '424778', 'license': 'MIT', 'n_tokens_mistral': 1625, 'n_tokens_neox': 1463, 'n_words': 457, 'path': 'Day2/St_Notebook 1 - First spider.ipynb', 'repo_name': 'ALaks96/CenterParcs_NLP_SentimentAnalysis_Webscraping', 'size': '4706'}