<jupyter_start><jupyter_text>Web Crawling- scrapy- BeautifulSoup- selenium...<jupyter_code># importamos la librería
#!pip install scrapy<jupyter_output>Collecting scrapy
  Downloading Scrapy-2.4.1-py2.py3-none-any.whl (239 kB)
[K     |████████████████████████████████| 239 kB 8.5 MB/s eta 0:00:01
[?25hRequirement already satisfied: cryptography>=2.0 in /opt/conda/lib/python3.8/site-packages (from scrapy) (3.2.1)
Requirement already satisfied: lxml>=3.5.0 in /opt/conda/lib/python3.8/site-packages (from scrapy) (4.6.2)
Requirement already satisfied: pyOpenSSL>=16.2.0 in /opt/conda/lib/python3.8/site-packages (from scrapy) (20.0.0)
Requirement already satisfied: six>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from cryptography>=2.0->scrapy) (1.15.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /opt/conda/lib/python3.8/site-packages (from cryptography>=2.0->scrapy) (1.14.4)
Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.0->scrapy) (2.20)
Collecting cssselect>=0.9.1
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting itemadapter>=[...]<jupyter_text>```    title="Diseño y Creación Digitales"    href="/es/grados/diseño-creacion-digital/presentacion"    class="card-absolute-link">    &nbsp;``` ***```   title="Multimedia"    href="/es/grados/multimedia/presentacion"    class="card-absolute-link">    &nbsp; ``` Existe una forma de extraer estos contenidos a través de **XPATH** . Entonces si quiero sacar todos los elementos de la `class` necesitamos a través de esta sintaxis:```//a[@class="card-absolute-link"]/@title```<jupyter_code># importamos scrapy
import scrapy
from scrapy.crawler import CrawlerProcess
# creamos primero el bicho "araña"
class uoc_spider(scrapy.Spider):
    # Asignamos un nombre al bicho
    name = "uoc_spider"
    
    # Indicamos la url que queremos analizar
    start_urls = [
        "https://estudios.uoc.edu/es/grados"
    ]
    
    # Definimos el analizador
    def parse(self, response):
        # Extraemos el título del grado
        for grado in response.xpath('//a[@class="card-absolute-link"]'):
            yield {
                'title': grado.extract()
            }<jupyter_output><empty_output><jupyter_text>La idea es ir saltando de USER_AGENT con diferentes elementos en Lista `[AGENT]`<jupyter_code># una vez definida la araña con la class ahora lanzamos
if __name__ == "__main__":

    # Creamos un crawler.
    process = CrawlerProcess({
        'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
        'DOWNLOAD_HANDLERS': {'s3': None},
        'LOG_ENABLED': True
    })

    # Inicializamos el crawler con nuestra araña.
    process.crawl(uoc_spider)
    
    # Lanzamos la araña.
    process.start()<jupyter_output>2020-12-18 19:55:18 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: scrapybot)
2020-12-18 19:55:18 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.6 | packaged by conda-forge | (default, Nov 27 2020, 19:31:52) - [GCC 9.3.0], pyOpenSSL 20.0.0 (OpenSSL 1.1.1h  22 Sep 2020), cryptography 3.2.1, Platform Linux-4.19.76-linuxkit-x86_64-with-glibc2.10
2020-12-18 19:55:18 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2020-12-18 19:55:18 [scrapy.crawler] INFO: Overridden settings:
{'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'}
2020-12-18 19:55:18 [scrapy.extensions.telnet] INFO: Telnet Password: <PASSWORD>
2020-12-18 19:55:18 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
20[...]
{'fraction_non_alphanumeric': 0.15473380540257015, 'fraction_numerical': 0.06949908208759507, 'mean_word_length': 4.044973544973545, 'pattern_counts': {'":': 1, '<': 12, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '230169', 'license': 'MIT', 'n_tokens_mistral': 1616, 'n_tokens_neox': 1453, 'n_words': 313, 'path': '02-python-201/labs/APIs/05_Web_Crawling.ipynb', 'repo_name': 'MariaPCampos/NEOLAND-DS2020-datalabs', 'size': '3819'}