<jupyter_start><jupyter_text>Blog scraper==This scraper runs in two parts:1. Iterate over some number of blog pages and collect URLs to individual posts.1. Iterate over the collected posts URLs and collect their content.In order to run the scraper for some other blog you will need to provide:1. the blog URL in the `curr_page_url`;1. Post XPATH URL selector (should be an link tag) in the `post_url_selector`;1. Blog post content selector (could be multiple divs. p-tags, etc) in the `content_selector`;1. Path to the ouput file. **NOTE**: The output directory should exist.<jupyter_code>import requests
from lxml import html

def get_it(url, selector, value_selector):
    page = requests.get(url)
    tree = html.fromstring(page.content)
    return [[x for x in post_body.xpath(value_selector) if x.strip() != ''] for post_body in tree.xpath(selector)]

flatten = lambda l: [item for sublist in l for item in sublist]

max_page_num = 6
blog_post_urls = []
for page_num in range(max_page_num):
    cur_page_url = 'https://yurukov.net/blog/page/' + str(page_num + 1)
    post_url_selector = '//h1[contains(@class, "entry-title")]/a'
    blog_post_urls += get_it(cur_page_url, post_url_selector, './@href')


print('Number of posts: ' + str(len(blog_post_urls)))

for num, url in enumerate(flatten(blog_post_urls)):
    content_selector = '//div[@class="entry-content"]/p'
    content = get_it(url, content_selector, './text()')
    output_file = './blogs/yurukov/' + str(num) + ".txt"
    with open(output_file, "w") as text_file:
        print(''.join(flatten(content)), file=text_file)<jupyter_output>Number of posts: 30
{'fraction_non_alphanumeric': 0.12138016019716574, 'fraction_numerical': 0.006777572396796057, 'mean_word_length': 4.678321678321678, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '322180', 'license': 'MIT', 'n_tokens_mistral': 545, 'n_tokens_neox': 537, 'n_words': 170, 'path': 'blog_scraper.ipynb', 'repo_name': 'SlothWorks-Hackathon/stylometrics', 'size': '1623'}