<jupyter_start><jupyter_text>This script scrapes Python-related Common Vulnerabilities and Exposures from the National Vulnerabilities Dataset Importing the neccesary libraries.<jupyter_code>import requests
import bs4
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>Pulling the dedicated page for Python vulnerabilities from http://cve.mitre.orgNote: keyword = python<jupyter_code>URL = "http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=python"<jupyter_output><empty_output><jupyter_text>Requesting the URL above<jupyter_code>page = requests.get(URL)<jupyter_output><empty_output><jupyter_text>Specify a desired format for the “page” using the html parser in BeautifulSoup. This allows Python to read the various components of the page, rather than treating it as a long string.<jupyter_code>formated_page = BeautifulSoup(page.text, "html.parser")<jupyter_output><empty_output><jupyter_text>Printing the formatted page in a structured tree format (using the prettify method) for readability<jupyter_code>print(formated_page.prettify())<jupyter_output><?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
 <head>
  <meta content="IE=Edge" http-equiv="X-UA-Compatible"/>
  <meta content="text/html; charset=utf-8" http-equiv="content-type"/>
  <meta content="CVE, Common Vulnerabilities and Exposures, CVE List, CVE Records, CVE ID, CVE IDs, CVE Identifier, CVE Identifiers, CVE ID number, CVE ID numbers, CVE number, CVE numbers, CVE Entry, CVE Entries, CVE name, CVEs, CVE-, Reserved but Public, RBP, CVE Numbering Authority, CVE Naming Authority, CNA, CNAs, Root CNA, Top-Level Root CNA, TLR-CNA, CNA of Last Resort, CNA-LR, Secretariat, Authorized Data Publisher, ADP, CVE Adoption, National Vulnerability Database, NVD, Common Vulnerability Scoring System, CVSS, standard, standards, vulnerability, vulnerabilities, vulnerability management, vulnerability names, vulnerability naming, vuln[...]<jupyter_text>A function to extract the CVE ids<jupyter_code>def extract_CVE_ids(formated_page):
    CVE_ids =[]
    for table in formated_page.find_all(name="div", attrs={"id":"TableWithRules"}):
        for row in table.find_all(name="tr"):
            for column in row.find_all(name="td", attrs={"valign":"top", "nowrap":"nowrap"}):
                for ids in column.find_all(name="a"):
                    CVE_ids.append(ids.string)
    return CVE_ids<jupyter_output><empty_output><jupyter_text>Calling the extract_CVE_ids function<jupyter_code>CVE_ids_list = extract_CVE_ids(formated_page)
CVE_ids_list<jupyter_output><empty_output><jupyter_text>A function to filter record in the extract_CVE_description function. The function is passed as an argument in one of its .find_all<jupyter_code>def has_valign_but_no_wrap(tag):
    return tag.has_attr('valign') and not tag.has_attr('nowrap') and not tag.has_attr('style')<jupyter_output><empty_output><jupyter_text>Implementation of extract_CVE_description function Importing the regular expression module to treat the vulnerability description string<jupyter_code>import re
def extract_CVE_description(formated_page):
    CVE_descr =[]
    for table in formated_page.find_all(name="div", attrs={"id":"TableWithRules"}):
        for row in table.find_all(name="tr"):
            for descr in row.find_all(has_valign_but_no_wrap):
                delimited_descr = re.split("\n+", descr.string)
                CVE_descr.append(delimited_descr[0])
    return CVE_descr<jupyter_output><empty_output><jupyter_text>Calling the extract_CVE_description function<jupyter_code>CVE_description_list = extract_CVE_description(formated_page)
CVE_description_list<jupyter_output><empty_output><jupyter_text>Building frame for the extracted data Import required library<jupyter_code>import pandas as pd
data_frame = pd.DataFrame(list(zip(CVE_ids_list,CVE_description_list)),columns=["CVE","vulnerability_description"])
data_frame.head()<jupyter_output><empty_output><jupyter_text>Complete the code below (r"") with path to the local storage<jupyter_code>data_frame.to_csv(r"", encoding='utf-8')<jupyter_output><empty_output><jupyter_text>This is when the CVE and their respective descriptions are expected to be stored in a database.<jupyter_code>data_dictionary = data_frame.to_dict('records')
data_dictionary<jupyter_output><empty_output><jupyter_text>Import required modules<jupyter_code>import pymysql
connection = pymysql.connect(host='127.0.0.1', user='root', passwd="", db='mysql')
cursor = connection.cursor()
cursor.execute('USE vuldb')
def store_to_db(cve, descr):
    cursor.execute('INSERT INTO PYTHON_CVEVUL_TBL (cve_name, cve_description) VALUES''(%s, %s)', (cve, descr))
    cursor.connection.commit()
def from_dict_to_db (data_dictionary):
    for items in data_dictionary:
        if 'CVE' in items and 'vulnerability_description' in items:
            #calling the store_to_db function
            store_to_db(items['CVE'], items['vulnerability_description'])
from_dict_to_db (data_dictionary)<jupyter_output><empty_output><jupyter_text>Close the cursor and connection<jupyter_code>cursor.close()
connection.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1079136690647482, 'fraction_numerical': 0.004543733434305188, 'mean_word_length': 6.645441389290883, 'pattern_counts': {'":': 4, '<': 67, '<?xml version=': 1, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 3}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '227185', 'license': 'MIT', 'n_tokens_mistral': 1794, 'n_tokens_neox': 1737, 'n_words': 395, 'path': 'data-scraping-curating/.ipynb_checkpoints/Python CVEs from the National Vulnerabilities Database-checkpoint.ipynb', 'repo_name': 'Semiu/python-codesecurity', 'size': '5282'}