<commit_before>---
name: Scraper rewrite
desc: Improve the current scraper to download data
requirements:
 - Familiarity with Python
 - Knowledge of Python regex
 - Familiar with web protocols
difficulty: medium
issues:
 - https://github.com/sunpy/sunpy/issues/4888
mentors:
 - nabobalis
 - samaloney
 - hayesla
initiatives:
 - GSOC
project_size:
 - medium
tags:
 - python
 - web
collaborating_projects:
 - sunpy
---

This project will provide the ability to parse URL information in a programmatic way.

#### Description

sunpy provides a way for users to access data using Fido and its many internal clients.
Many of these simpler clients do URL scraping to return metadata and data files to a user.

This scraping is done by the [scraper class](https://docs.sunpy.org/en/stable/api/sunpy.util.scraper.Scraper.html?highlight=scraper).<commit_msg>Fix sunpy scraper's api link<commit_after>---
name: Scraper rewrite
desc: Improve the current scraper to download data
requirements:
 - Familiarity with Python
 - Knowledge of Python regex
 - Familiar with web protocols
difficulty: medium
issues:
 - https://github.com/sunpy/sunpy/issues/4888
mentors:
 - nabobalis
 - samaloney
 - hayesla
initiatives:
 - GSOC
project_size:
 - medium
tags:
 - python
 - web
collaborating_projects:
 - sunpy
---

This project will provide the ability to parse URL information in a programmatic way.

#### Description

sunpy provides a way for users to access data using Fido and its many internal clients.
Many of these simpler clients do URL scraping to return metadata and data files to a user.

This scraping is done by the [scraper class](https://docs.sunpy.org/en/stable/api/sunpy.net.scraper.Scraper.html?highlight=scraper).
{'fraction_non_alphanumeric': 0.07777777777777778, 'fraction_numerical': 0.004678362573099415, 'mean_word_length': 4.047197640117994, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4016327', 'n_tokens_mistral': 525, 'n_tokens_neox': 503, 'n_words': 197}