<jupyter_start><jupyter_text>Scraping Top Repositories for Topics on GitHubTODO  (Intro): - Introduction about web scraping- Introduction about GitHub and the problem statement- Mention the tools you're using (Python, requests, Beautiful Soup, Pandas) Here are the steps we'll follow:- We're going to scrape https://github.com/topics- We'll get a list of topics. For each topic, we'll get topic title, topic page URL and topic description- For each topic, we'll get the top 25 repositories in the topic from the topic page- For each repository, we'll grab the repo name, username, stars and repo URL- For each topic we'll create a CSV file in the following format:```Repo Name,Username,Stars,Repo URLthree.js,mrdoob,69700,https://github.com/mrdoob/three.jslibgdx,libgdx,18300,https://github.com/libgdx/libgdx```  Scrape the list of topics from GithubExplain how you'll do it.- use requests to downlaod the page- user BS4 to parse and extract information- convert to a Pandas dataframeLet's write a function to download the page.<jupyter_code>import requests
from bs4 import BeautifulSoup
import os
import pandas as pd
def get_topics_page():
    
    topics_url = 'https://github.com/topics'
    response = requests.get(topics_url)
    if response.status_code != 200:
        raise Exception('Failed to load page {}'.format(topics_url))
    doc = BeautifulSoup(response.text, 'html.parser')
    return doc
doc = get_topics_page()<jupyter_output><empty_output><jupyter_text>Let's create some helper functions to parse information from the page.To get topic titles, we can pick `p` tags with the `class` ...![](https://i.imgur.com/Yp7jt81.png)<jupyter_code>def get_topic_titles(doc):
    selection_class = 'f3 lh-condensed mb-0 mt-1 Link--primary'
    topic_title_tags = doc.find_all('p', {'class': selection_class})
    topic_titles = []
    for tag in topic_title_tags:
        topic_titles.append(tag.text)
    return topic_titles<jupyter_output><empty_output><jupyter_text>`get_topic_titles` can be used to get the list of titles<jupyter_code>titles = get_topic_titles(doc)
titles[:5]<jupyter_output><empty_output><jupyter_text>Similarly we have defined functions for descriptions and URLs.<jupyter_code>def get_topic_descs(doc):
    desc_selector = 'f5 color-text-secondary mb-0 mt-1'
    topic_desc_tags = doc.find_all('p', {'class': desc_selector})
    topic_descs = []
    for tag in topic_desc_tags:
        topic_descs.append(tag.text.strip())
    return topic_descs


descs = get_topic_descs(doc)
descs[:5]
def get_topic_urls(doc):
    topic_link_tags = doc.find_all('a', {'class': 'd-flex no-underline'})
    topic_urls = []
    base_url = 'https://github.com'
    for tag in topic_link_tags:
        topic_urls.append(base_url + tag['href'])
    return topic_urls

url = get_topic_urls(doc)
url[:5]<jupyter_output><empty_output><jupyter_text>Let's put this all together into a single function<jupyter_code>def scrape_topics():
    topics_url = 'https://github.com/topics'
    response = requests.get(topics_url)
    if response.status_code != 200:
        raise Exception('Failed to load page {}'.format(topic_url))
    doc = BeautifulSoup(response.text, 'html.parser')
    topics_dict = {
        'title': get_topic_titles(doc),
        'description': get_topic_descs(doc),
        'url': get_topic_urls(doc)
    }
    return pd.DataFrame(topics_dict)
topics_df = scrape_topics()
topics_df[:5]
topics_df.to_csv('topics.csv', index=None)<jupyter_output><empty_output><jupyter_text>Get the top repositories from a topic page<jupyter_code>def parse_count_star(str):
    str = str.strip()
    if str[-1] == 'k':
        return int(float(str[:-1]) * 1000)
    return int(str)
def get_topic_page(topic_url):
    
    # Download the page
    response = requests.get(topic_url)
    # Check successful response
    if response.status_code != 200:
        raise Exception('Failled to load page {}', format(topic_url))
    # Parse using Beautiful soup    
    topic_doc = BeautifulSoup(response.text, 'html.parser')
    
    return topic_doc
def get_repo_info(h3_tag, star_tag):
    
    # returns all the required info about a repository
    a_tags = h3_tag.find_all('a')
    base_url = 'https://github.com'
    username = a_tags[0].text.strip()
    repo_name = a_tags[1].text.strip()
    repo_url = base_url + a_tags[1]['href']
    stars = parse_count_star(star_tag.text.strip())
    
    return username, repo_name, stars, repo_url
def get_topic_repos(topic_doc):
    
    # Get the h3 tags containing repo title, repo URL and username
    h3_class_selector = 'f3 color-text-secondary text-normal lh-condensed'
    repo_tags = topic_doc.find_all('h3', {'class': h3_class_selector})
    # Get star tags
    star_tags = topic_doc.find_all('a', {'class' : 'social-count float-none'})
    
    topics_repos_dict = {
        'username' : [],
        'repo_name' :[],
        'stars' : [],
        'repo_url' : []
    }
    
    # Get repo info
    for i in range(len(repo_tags)) :
        repo_info = get_repo_info(repo_tags[i],star_tags[i])
        topics_repos_dict['username'].append(repo_info[0])
        topics_repos_dict['repo_name'].append(repo_info[1])
        topics_repos_dict['stars'].append(repo_info[2])
        topics_repos_dict['repo_url'].append(repo_info[3])
        
    return pd.DataFrame(topics_repos_dict)
def scrape_topic(topic_url, path) :
    if os.path.exists(path):
        print('The file {} already exists. Skipping...' .format(path))
        return
    topic_df = get_topic_repos(get_topic_page(topic_url))
    topic_df.to_csv(path, index = None)<jupyter_output><empty_output><jupyter_text>Putting it all together- We have a funciton to get the list of topics- We have a function to create a CSV file for scraped repos from a topics page- Let's create a function to put them together<jupyter_code>def scrape_topics_repos():
    print('Scraping list of topics')
    topics_df = scrape_topics()
    
    os.makedirs('data', exist_ok = True)
    for index, row in topics_df.iterrows():
        print('Scraping top repositries for {}' .format(row['title']))
        scrape_topic(row['url'], 'data/{}.csv' .format(row['title']))
scrape_topics_repos()<jupyter_output>Scraping list of topics
Scraping top repositries for 3D
Scraping top repositries for Ajax
Scraping top repositries for Algorithm
Scraping top repositries for Amp
Scraping top repositries for Android
Scraping top repositries for Angular
Scraping top repositries for Ansible
Scraping top repositries for API
Scraping top repositries for Arduino
Scraping top repositries for ASP.NET
Scraping top repositries for Atom
Scraping top repositries for Awesome Lists
Scraping top repositries for Amazon Web Services
Scraping top repositries for Azure
Scraping top repositries for Babel
Scraping top repositries for Bash
Scraping top repositries for Bitcoin
Scraping top repositries for Bootstrap
Scraping top repositries for Bot
Scraping top repositries for C
Scraping top repositries for Chrome
Scraping top repositries for Chrome extension
Scraping top repositries for Command line interface
Scraping top repositries for Clojure
Scraping top repositries for Code quality
Scraping top repositries for Code rev[...]
{'fraction_non_alphanumeric': 0.09924937447873228, 'fraction_numerical': 0.0079232693911593, 'mean_word_length': 4.286554004408523, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '246027', 'license': 'MIT', 'n_tokens_mistral': 2429, 'n_tokens_neox': 2327, 'n_words': 656, 'path': 'code.ipynb', 'repo_name': 'srv-shrma/Web-Scraping-from-Github', 'size': '7194'}