<jupyter_start><jupyter_text>Playnomore 사이트 데이터 수집- `http://playnomore.co.kr/`- scrapy에서 fake_useragent 사용- scrapy를 실행할때 아규먼트를 설정해서 실행- pipelines에서 데이터 베이스로 데이터를 저장<jupyter_code>import scrapy
import requests
from scrapy.http import TextResponse
import pandas as pd<jupyter_output><empty_output><jupyter_text>1. 프로젝트 생성<jupyter_code>!rm -rf playnomore # 있으면 제거
!scrapy startproject playnomore<jupyter_output>New Scrapy project 'playnomore', using template directory '/home/ubuntu/.pyenv/versions/3.6.9/envs/python3/lib/python3.6/site-packages/scrapy/templates/project', created in:
    /home/ubuntu/python3/notebook/fastcampus_1학기_수업내용/06_scrapy/playnomore

You can start your first spider with:
    cd playnomore
    scrapy genspider example example.com<jupyter_text>2. items.py- title, price, img, link<jupyter_code># items.py를 읽어온다.
# %load playnomore/playnomore/items.py

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class PlaynomoreItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    pass<jupyter_output><empty_output><jupyter_text>- `%` : 한줄을 읽어서 실행- `%%` : 셀 단위를 읽어서 실행<jupyter_code>%%writefile playnomore/playnomore/items.py
import scrapy


class PlaynomoreItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    img = scrapy.Field()
    link = scrapy.Field()<jupyter_output>Overwriting playnomore/playnomore/items.py<jupyter_text>3. xpath 확인- 링크- 링크 -> 상세페이지(제목, 이미지URL, 가격)- fake_useragent install    - `pip install fake_useragent`<jupyter_code>url = "http://playnomore.co.kr/category/bag/24/"
req = requests.get(url)
req
# <Response [403]> : useagent를 체크해서 막고있기 때문이다.
!pip install fake_useragent
from fake_useragent import UserAgent
url = "http://playnomore.co.kr/category/bag/24/"
headers = {"User-Agent" : UserAgent().chrome}
req = requests.get(url, headers=headers)
req # <Response [200]> : 잘 된것을 확인
response = TextResponse(req.url, body=req.text, encoding="utf-8")
response<jupyter_output><empty_output><jupyter_text>상세페이지 링크<jupyter_code>links = response.xpath('//*[@id="contents"]/div[2]/div/ul/li/div/a/@href').extract()

# response.urljoin(links[0]) # 도메인이 없어서 도메인을 추가 : urljoin(links[0])

links = list(map(response.urljoin, links)) # 한페이지의 데이터<jupyter_output><empty_output><jupyter_text>상세페이지 : 제목, 가격, 이미지URL 가져오기<jupyter_code>url = links[0]
headers = {"User-Agent" : UserAgent().chrome}
req = requests.get(url, headers=headers)
response = TextResponse(req.url, body=req.text, encoding="utf-8")
response<jupyter_output><empty_output><jupyter_text>타이들을 두개로 나누어서 가져와야된다.- 하위 엘리먼트의 텍스트는 가져오지 않기 때문이다.<jupyter_code>title1 = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[1]/font/text()').extract()
title2 = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[1]/text()').extract()

title = "".join(title1) + "".join(title2)
title<jupyter_output><empty_output><jupyter_text>가격 가져오기<jupyter_code>price = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[2]/text()').extract()[0]
price<jupyter_output><empty_output><jupyter_text>이미지 URL 가져오기<jupyter_code>img = "http:" + response.xpath('//*[@id="contents"]/div[1]/div[1]/div[1]/div[1]/img/@src').extract()[0]
img<jupyter_output><empty_output><jupyter_text>4. spider.py- scrapy-fake-useragent install    - `pip install scrapy-fake-useragent`<jupyter_code>!pip install scrapy-fake-useragent
!pip list | grep fake
%%writefile playnomore/playnomore/spiders/spider.py
import scrapy
from playnomore.items import PlaynomoreItem


class PlaynomoreSpider(scrapy.Spider):
    # 이 이름을 사용해야 한다 규칙이다.
    name = "Playnomore"
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
            'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
        }
    }
    
    def start_requests(self):
        url = "http://playnomore.co.kr/category/bag/24/"
        yield scrapy.Request(url, callback=self.parse)
        
    def parse(self, response):
        links = response.xpath('//*[@id="contents"]/div[2]/div/ul/li/div/a/@href').extract()
        links = list(map(response.urljoin, links))
        for link in links:
            yield scrapy.Request(link, callback=self.page_parse)
                         
    # 상세페이지 데이터를 객체로 만든다.
    def page_parse(self, response):
        item = PlaynomoreItem()
        title1 = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[1]/font/text()').extract()
        title2 = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[1]/text()').extract()
        item["title"] = "".join(title1) + "".join(title2)
        item["price"] = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[2]/text()').extract()[0]
        item["img"] = "http:" + response.xpath('//*[@id="contents"]/div[1]/div[1]/div[1]/div[1]/img/@src').extract()[0]
        item["link"] = response.url
        yield item
%%writefile run.sh
cd playnomore
scrapy crawl Playnomore -o playnomore.csv
!chmod +x run.sh
!./run.sh<jupyter_output>2019-11-28 05:24:27 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: playnomore)
2019-11-28 05:24:27 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.6.9 (default, Oct 24 2019, 05:23:48) - [GCC 7.4.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-4.15.0-1052-aws-x86_64-with-debian-buster-sid
2019-11-28 05:24:27 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'playnomore', 'FEED_FORMAT': 'csv', 'FEED_URI': 'playnomore.csv', 'NEWSPIDER_MODULE': 'playnomore.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['playnomore.spiders']}
2019-11-28 05:24:27 [scrapy.extensions.telnet] INFO: Telnet Password: <PASSWORD>468e1
2019-11-28 05:24:27 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'sc[...]<jupyter_text>pandas로 확인<jupyter_code>df = pd.read_csv("playnomore/playnomore.csv")
df.tail(2)<jupyter_output><empty_output><jupyter_text>5. argument 설정<jupyter_code>%%writefile playnomore/playnomore/spiders/spider.py
import scrapy
from playnomore.items import PlaynomoreItem


class PlaynomoreSpider(scrapy.Spider):
    # 이 이름을 사용해야 한다 규칙이다.
    name = "Playnomore"
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
            'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
        }
    }
    
    def __init__(self, category1="bag", category2=24, **kwargs):
        self.start_url = "http://playnomore.co.kr/category/{}/{}/".format(category1, category2)
        super().__init__(**kwargs) # 기존에 있던 생성자 함수를 상속, 위코드만 추가
    
    def start_requests(self):
        url = self.start_url
        yield scrapy.Request(url, callback=self.parse)
        
    def parse(self, response):
        links = response.xpath('//*[@id="contents"]/div[2]/div/ul/li/div/a/@href').extract()
        links = list(map(response.urljoin, links))
        for link in links:
            yield scrapy.Request(link, callback=self.page_parse)
                         
    # 상세페이지 데이터를 객체로 만든다.
    def page_parse(self, response):
        item = PlaynomoreItem()
        title1 = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[1]/font/text()').extract()
        title2 = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[1]/text()').extract()
        item["title"] = "".join(title1) + "".join(title2)
        item["price"] = response.xpath('//*[@id="contents"]/div[1]/div[1]/div[2]/div[2]/text()').extract()[0]
        item["img"] = "http:" + response.xpath('//*[@id="contents"]/div[1]/div[1]/div[1]/div[1]/img/@src').extract()[0]
        item["link"] = response.url
        yield item<jupyter_output>Overwriting playnomore/playnomore/spiders/spider.py<jupyter_text>신발정보 수집- shoes, 25<jupyter_code>%%writefile run.sh
cd playnomore
scrapy crawl Playnomore -o playnomore2.csv -a category1=shoes -a category2=25
!./run.sh
df = pd.read_csv("playnomore/playnomore2.csv")
df.tail(2)<jupyter_output><empty_output><jupyter_text>6. Mongodb에 저장- pymongo 사용- pymongo를 pipelines.py에 적용- pymongo install    - `pip install pymongo==2.8.1`<jupyter_code>import pymongo
client = pymongo.MongoClient('mongodb://172.16.17.32:27017/')
client
db = client.playnomore # 데이터베이스
collection = db.shoes # 테이블
collection
# 한글이 잘 들어가는지 확인
data = {"title":"신발"}
collection.insert(data)<jupyter_output><empty_output><jupyter_text>Mongodb 모듈 파일 생성- import 해서 생성을 하기 위해서<jupyter_code>%%writefile playnomore/playnomore/mongodb.py
import pymongo

client = pymongo.MongoClient('mongodb://172.16.17.32:27017/')
db = client.playnomore 
collection = db.shoes 
!cat playnomore/playnomore/pipelines.py
%%writefile playnomore/playnomore/pipelines.py
from .mongodb import collection

class PlaynomorePipeline(object):
    
    def process_item(self, item, spider):
        
        data = { "title": item["title"], 
                 "price": item["price"], 
                 "img": item["img"],
                 "link": item["link"],
                }
        collection.insert(data)
        return item
!echo "ITEM_PIPELINES = {" >> playnomore/playnomore/settings.py
!echo "    'playnomore.pipelines.PlaynomorePipeline': 300," >> playnomore/playnomore/settings.py
!echo "}" >> playnomore/playnomore/settings.py
!tail -n 5 playnomore/playnomore/settings.py
!cat run.sh
!./run.sh<jupyter_output>2019-11-28 05:55:28 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: playnomore)
2019-11-28 05:55:28 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.6.9 (default, Oct 24 2019, 05:23:48) - [GCC 7.4.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-4.15.0-1052-aws-x86_64-with-debian-buster-sid
2019-11-28 05:55:28 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'playnomore', 'FEED_FORMAT': 'csv', 'FEED_URI': 'playnomore2.csv', 'NEWSPIDER_MODULE': 'playnomore.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['playnomore.spiders']}
2019-11-28 05:55:28 [scrapy.extensions.telnet] INFO: Telnet Password: <PASSWORD>
2019-11-28 05:55:28 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 's[...]
{'fraction_non_alphanumeric': 0.16493591542707456, 'fraction_numerical': 0.04191224623444663, 'mean_word_length': 4.263417035942886, 'pattern_counts': {'":': 5, '<': 64, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '567642', 'license': 'MIT', 'n_tokens_mistral': 4546, 'n_tokens_neox': 4472, 'n_words': 675, 'path': 'Scrapy/04_scrapy_playnomore.ipynb', 'repo_name': 'kimchangkyu/TIL', 'size': '10696'}