<jupyter_start><jupyter_text>CVE Prophet Notebook This notebook will pull all [JSON Data](https://nvd.nist.gov/vuln/data-feedsJSON_FEED) from the NVD and performs some basic time series forecasting using [Prophet](https://facebook.github.io/prophet/) from Facebook.    Getting Started  Retrieve The Data This cell pulls all JSON files from the NVD that we will be working with.<jupyter_code>%%capture
!pip3 install prophet
!mkdir -p jsondata
%cd jsondata
!rm *.json 
!rm *.zip 
!wget https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-{2000..2021}.json.zip 
!unzip -o "*.zip"<jupyter_output><empty_output><jupyter_text>(Note: The Output of this cell is hidden with `%%capture`)  Import Python Libraries<jupyter_code>import calplot
import glob
import json
import logging
import sys
import numpy as np
import os
import pandas as pd
import re
import uuid
from datetime import datetime
from prophet import Prophet
from prophet.plot import add_changepoints_to_plot
from scipy import stats
logging.disable(sys.maxsize)<jupyter_output><empty_output><jupyter_text>Build Base DataFrame This code builds a Panda dataframe from the JSON files we downloaded, removing all CVE's marked rejected.<jupyter_code>row_accumulator = []
for filename in glob.glob('*.json'):
    with open(filename, 'r', encoding='utf-8') as f:
        nvd_data = json.load(f)
        for entry in nvd_data['CVE_Items']:
            cve = entry['cve']['CVE_data_meta']['ID']
            try:
                published_date = entry['publishedDate']
            except KeyError:
                published_date = 'Missing_Data_JG'
            try:
                attack_vector = entry['impact']['baseMetricV3']['cvssV3']['attackVector']
            except KeyError:
                attack_vector = 'Missing_Data_JG'
            try:
                attack_complexity = entry['impact']['baseMetricV3']['cvssV3']['attackComplexity']
            except KeyError:
                attack_complexity = 'Missing_Data_JG'
            try:
                privileges_required = entry['impact']['baseMetricV3']['cvssV3']['privilegesRequired']
            except KeyError:
                privileges_required = 'Missing_Data_JG'
            try:
                user_interaction = entry['impact']['baseMetricV3']['cvssV3']['userInteraction']
            except KeyError:
                user_interaction = 'Missing_Data_JG'
            try:
                scope = entry['impact']['baseMetricV3']['cvssV3']['scope']
            except KeyError:
                scope = 'Missing_Data_JG'
            try:
                confidentiality_impact = entry['impact']['baseMetricV3']['cvssV3']['confidentialityImpact']
            except KeyError:
                confidentiality_impact = 'Missing_Data_JG'
            try:
                integrity_impact = entry['impact']['baseMetricV3']['cvssV3']['integrityImpact']
            except KeyError:
                integrity_impact = 'Missing_Data_JG'
            try:
                availability_impact = entry['impact']['baseMetricV3']['cvssV3']['availabilityImpact']
            except KeyError:
                availability_impact = 'Missing_Data_JG'
            try:
                base_score = entry['impact']['baseMetricV3']['cvssV3']['baseScore']
            except KeyError:
                base_score = 'Missing_Data_JG'
            try:
                base_severity = entry['impact']['baseMetricV3']['cvssV3']['baseSeverity']
            except KeyError:
                base_severity = 'Missing_Data_JG'
            try:
                exploitability_score = entry['impact']['baseMetricV3']['exploitabilityScore']
            except KeyError:
                exploitability_score = 'Missing_Data_JG'
            try:
                impact_score = entry['impact']['baseMetricV3']['impactScore']
            except KeyError:
                impact_score = 'Missing_Data_JG'
            try:
                cwe = entry['cve']['problemtype']['problemtype_data'][0]['description'][0]['value']
            except IndexError:
                cwe = 'Missing_Data_JG'
            try:
                description = entry['cve']['description']['description_data'][0]['value']
            except IndexError:
                description = ''
            new_row = { 
                'CVE': cve, 
                'Published': published_date,
                'AttackVector': attack_vector,
                'AttackComplexity': attack_complexity,
                'PrivilegesRequired': privileges_required,
                'UserInteraction': user_interaction,
                'Scope': scope,
                'ConfidentialityImpact': confidentiality_impact,
                'IntegrityImpact': integrity_impact,
                'AvailabilityImpact': availability_impact,
                'BaseScore': base_score,
                'BaseSeverity': base_severity,
                'ExploitabilityScore': exploitability_score,
                'ImpactScore': impact_score,
                'CWE': cwe,
                'Description': description
            }
            if not description.startswith('** REJECT **'): # disputed, rejected and other non issues start with '**'
                row_accumulator.append(new_row)
    nvd = pd.DataFrame(row_accumulator)

nvd['Published'] = pd.to_datetime(nvd['Published']).apply(lambda x: x.date())
print ('CVEs from NVD:', nvd['CVE'].count())
nvdcount = nvd['Published'].count()
nvdunique = nvd['Published'].nunique()
per_day = nvdcount/nvdunique
per_day = round(per_day, 0)
print('CVEs Published Per Publishing Day:', per_day)<jupyter_output>CVEs from NVD: 157110
CVEs Published Per Publishing Day: 25.0<jupyter_text>(Note: The code for building this dataframe is based off the great work in this [repo](https://github.com/joocer/mitre-tools/blob/74a08bca9e79f2c18cc4e8671d1d7da5b43a2dd4/source/data/02%20process_rawdata.py))  Build Base DataFrame For Prophet<jupyter_code>cve_day = pd.DataFrame(nvd.Published.value_counts().reset_index().values, columns=["ds", "y"])
cve_day = cve_day.sort_index(axis = 0, ascending=True)
cve_day = cve_day[cve_day['y'] <= 400]  
cve_day.head(10)<jupyter_output><empty_output><jupyter_text>There were ~5 days in the dataset that have between 400-1100 CVEs published. For this excercise I have removed them to correct the look of the graphs.   Build The Future Dataframe<jupyter_code>m = Prophet()
m.fit(cve_day)
future = m.make_future_dataframe(periods=3650)<jupyter_output><empty_output><jupyter_text>Plot the Forecast This shows the plot for all previous data and the next 10 years.<jupyter_code>forecast = m.predict(future)
fig = m.plot(forecast, xlabel='Date', ylabel='CVEs')
fig.savefig('./forecast.jpg', dpi=300, bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Plot the Forecast With Change Points This shows the plot for all previous data and the next 10 years with changepoints marked where the data had abrupt changes in its trajectories.<jupyter_code>fig = m.plot(forecast, xlabel='Date', ylabel='CVEs')
a = add_changepoints_to_plot(fig.gca(), m, forecast)
fig.savefig('./forecast_changepoints.jpg', dpi=300, bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>CVE Predictions For The Rest Of The Year<jupyter_code>cve_prophet_thisyear = forecast[(forecast['ds'] > '2021-07-18') & (forecast['ds'] < '2021-12-31')]
cve_prophet_thisyear[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].round(0).sort_values(by=['yhat'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>CVE Predictions For The Next 5 Years<jupyter_code>cve_prophet_next5 = forecast[(forecast['ds'] > '2021-07-18') & (forecast['ds'] < '2026-12-31')]
cve_prophet_next5[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].round(0).sort_values(by=['yhat'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>CVE Predictions For The Next 10 Years<jupyter_code>cve_prophet_next10 = forecast[(forecast['ds'] > '2021-07-18') & (forecast['ds'] < '2031-12-31')]
cve_prophet_next10[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].round(0).sort_values(by=['yhat'], ascending=False).head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12208081057704188, 'fraction_numerical': 0.020882243914494007, 'mean_word_length': 3.273495248152059, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 43, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '197777', 'license': 'MIT', 'n_tokens_mistral': 2655, 'n_tokens_neox': 2460, 'n_words': 509, 'path': 'CVE-Prophet.ipynb', 'repo_name': 'jgamblin/cveprophet', 'size': '8093'}