<jupyter_start><jupyter_text>Scrapy je besplatan i *open-source* Python okvir za *web-crawling*. Pruža alate koji su potrebni za učinkovito *povlačenje* podataka s web stranica, njihovu obradu te pohranu u željenom formatu i strukturi.> Zadatak: *web-scrapanje* Youtube web stranice*   Link: https://www.youtube.com/  > Instalacija okvira Scrapy<jupyter_code>pip install scrapy
%cd /content/drive/MyDrive/yt_scrapy<jupyter_output>/content/drive/MyDrive/yt_scrapy<jupyter_text>> *Importanje* scrapy biblioteke i ostalih potrebnih modula<jupyter_code>import scrapy 
from scrapy.crawler import CrawlerProcess
import json<jupyter_output><empty_output><jupyter_text>> Kreiranje scrapy Spidera za *crawling* stranice i izvlačenje podataka                  *   YoutubeSpider je klasa, nasljeđuje se od scrapy.Spidera                  *   parse() je zadana callback metoda                  *   yield<jupyter_code>class YoutubeSpider(scrapy.Spider):
    
    name = "YoutubeSpider"
    
    start_urls = []
    #start_urls = ['https://www.youtube.com/watch?v=Tc0tLGWIqxA']

    input_file = open('yt_links.txt', 'r')

    for link in input_file:
      start_urls.append(link)

    def parse(self, response):
        """parse data from urls"""
        
        #print(response.text)

        yield {
            'title': response.xpath('//meta[@name="title"]/@content').get(),
            'datePublished': response.xpath('//meta[@itemprop="datePublished"]/@content').get(),
            'genre': response.xpath('//meta[@itemprop="genre"]/@content').get(),
            'publisher' : response.xpath('//link[@itemprop="name"]/@content').get(),
            'viewsCount' : response.xpath('//meta[@itemprop="interactionCount"]/@content').get()
        }<jupyter_output><empty_output><jupyter_text>> HTML Youtube stranice  > Pokretanje scrapy Spidera iz skripte        *   Klasa CrawlerProcess modula Crawler        *   Instancira se settings objekt<jupyter_code>process = CrawlerProcess(
    settings=
    {
        'FEEDS': 
     {
        'yt_data.csv': 
        {
            'format': 'csv',
            'overwrite': True
        }
    }
})<jupyter_output>2021-11-21 15:40:24 [scrapy.utils.log] INFO: Scrapy 2.5.1 started (bot: scrapybot)
2021-11-21 15:40:24 [scrapy.utils.log] INFO: Versions: lxml 4.2.6.0, libxml2 2.9.8, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.7.12 (default, Sep 10 2021, 00:21:48) - [GCC 7.5.0], pyOpenSSL 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021), cryptography 35.0.0, Platform Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
2021-11-21 15:40:24 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor<jupyter_text>> Pokretanje YoutubeSpidera<jupyter_code>process.crawl(YoutubeSpider)
process.start()
#print(response.text)<jupyter_output>2021-11-21 14:38:06 [scrapy.crawler] INFO: Overridden settings:
{}
2021-11-21 14:38:06 [scrapy.extensions.telnet] INFO: Telnet Password: <PASSWORD>
2021-11-21 14:38:06 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2021-11-21 14:38:06 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermid[...]<jupyter_text>> Skripta s JSON objektom koji sadrži podatake o videozapisu u responsu<jupyter_code>obj = json.loads(response.css('script::text').extract()[30][20:-1])

#print(obj)
# spremanje podataka o videozapisu u varijable

title = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][0]['videoPrimaryInfoRenderer']['title']['runs'][0]['text']
          
viewCount = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][0]['videoPrimaryInfoRenderer']['viewCount']['videoViewCountRenderer']['viewCount']['simpleText'][:-6]

likesCount = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][0]['videoPrimaryInfoRenderer']['videoActions']['menuRenderer']['topLevelButtons'][0]['toggleButtonRenderer']['defaultText']['accessibility']['accessibilityData']['label'][:-6]

dislikesCount = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][0]['videoPrimaryInfoRenderer']['videoActions']['menuRenderer']['topLevelButtons'][1]['toggleButtonRenderer']['defaultText']['accessibility']['accessibilityData']['label'][:-8]

song = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][1]['videoSecondaryInfoRenderer']['metadataRowContainer']['metadataRowContainerRenderer']['rows'][3]['metadataRowRenderer']['contents'][0]['simpleText']

artist = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][1]['videoSecondaryInfoRenderer']['metadataRowContainer']['metadataRowContainerRenderer']['rows'][4]['metadataRowRenderer']['contents'][0]['simpleText']

album = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][1]['videoSecondaryInfoRenderer']['metadataRowContainer']['metadataRowContainerRenderer']['rows'][5]['metadataRowRenderer']['contents'][0]['simpleText']

# yield novog rječnika
yield {
    'title': title,
    'song' : song,
    'artist': artist,
    'album' : album,
    'viewsCount': viewCount,
    'likesCount': likesCount,
    'dislikesCount': dislikesCount
    }<jupyter_output><empty_output><jupyter_text>> ❗ Problem: objekt ne sadrži istoimenovane ključeve (ili uopće ne sadrži neki od ključeva) za svaki videozapis ❗<jupyter_code># primjer kada nisu istoimenovani ključevi

#prvi videozapis
artist = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][1]['videoSecondaryInfoRenderer']['metadataRowContainer']['metadataRowContainerRenderer']['rows'][4]['metadataRowRenderer']['contents'][0]['text']

#drugi videozapis
artist = obj["contents"]["twoColumnWatchNextResults"]['results']['results']['contents'][1]['videoSecondaryInfoRenderer']['metadataRowContainer']['metadataRowContainerRenderer']['rows'][4]['metadataRowRenderer']['contents'][0]['simpleText']<jupyter_output><empty_output><jupyter_text>> Dva videozapisa, npr. jedan se odnosi na video spot pjesme, a drugi na spremanje recepta              *   U tom slučaju ne mogu  se dohvatiti podatci o npr. artist ili song za drugi videozapise jer videozapisi nisu istog tipa >  Importanje podataka yt_data.csv datoteke pomoću pandasa<jupyter_code>import pandas as pd
yt_data = pd.read_csv('yt_data.csv')
yt_data.columns = ['Title', 'Date published', 'Genre', 'Publisher', 'Views count']<jupyter_output><empty_output><jupyter_text>> Prikaz dataFrame okvira s podacima<jupyter_code>yt_data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15611520801447057, 'fraction_numerical': 0.028384583275358286, 'mean_word_length': 6.379876796714579, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '176642', 'license': 'Apache-2.0', 'n_tokens_mistral': 2480, 'n_tokens_neox': 2361, 'n_words': 376, 'path': 'yt_scraper.ipynb', 'repo_name': 'mmravak/Youtube_Scrapy', 'size': '7193'}