<jupyter_start><jupyter_text>Detection Malicious URL Using ML Models A URL consists of the top-level domain, hostname, paths, and port of the webaddress, as in the following diagram: ![url](https://user-images.githubusercontent.com/70811337/150021474-4ba64b34-1d09-427f-9cf6-25508571d18d.PNG) <h2 style="width: 100%;    margin: 0;    padding: 0;    text-align: center;">---- Table of contents ----1. [Importing Libraries](1)2. [Load Dataset](2)3. [Meta information of Dataframe](3)4. [Checking for NaN values](4)5. [Feature Extraction](5)6. [Train & Test Split](6)7. [Training models](7)        ✔️ Decision Tree Classifier      ✔️ Random Forest Classifier      ✔️ AdaBoost Classifier      ✔️ KNeighbors Classifier      ✔️ SGD Classifier      ✔️ Extra Trees Classifier      ✔️ Gaussian NB  8. [Final Report](8)<jupyter_code>pip install tld<jupyter_output>Collecting tld
  Downloading tld-0.12.6-py37-none-any.whl (412 kB)
     |████████████████████████████████| 412 kB 4.3 MB/s            
[?25hInstalling collected packages: tld
Successfully installed tld-0.12.6
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m
Note: you may need to restart the kernel to use updated packages.<jupyter_text>📥 Importing Libraries<jupyter_code>import re
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from colorama import Fore
from urllib.parse import urlparse
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, ExtraTreesClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import SGDClassifier
from sklearn.naive_bayes import GaussianNB
from tld import get_tld, is_tld<jupyter_output><empty_output><jupyter_text>🗃️ Load Dataset<jupyter_code>data = pd.read_csv('../input/malicious-urls-dataset/malicious_phish.csv')
data.head()<jupyter_output><empty_output><jupyter_text>📝 Meta information of Dataframe<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 651191 entries, 0 to 651190
Data columns (total 2 columns):
 #   Column  Non-Null Count   Dtype 
---  ------  --------------   ----- 
 0   url     651191 non-null  object
 1   type    651191 non-null  object
dtypes: object(2)
memory usage: 9.9+ MB<jupyter_text>🔎 Checking for NaN values<jupyter_code>data.isnull().sum()
count = data.type.value_counts()
count
sns.barplot(x=count.index, y=count)
plt.xlabel('Types')
plt.ylabel('Count');<jupyter_output><empty_output><jupyter_text>first have to omit the (www.) from the URL which is in fact a sub domain in itself.<jupyter_code>data['url'] = data['url'].replace('www.', '', regex=True)
data
data.head()
rem = {"Category": {"benign": 0, "defacement": 1, "phishing":2, "malware":3}}
data['Category'] = data['type']
data = data.replace(rem)<jupyter_output><empty_output><jupyter_text>🦾 Feature Extraction<jupyter_code>data['url_len'] = data['url'].apply(lambda x: len(str(x)))
def process_tld(url):
    try:
        res = get_tld(url, as_object = True, fail_silently=False,fix_protocol=True)
        pri_domain= res.parsed_url.netloc
    except :
        pri_domain= None
    return pri_domain
data['domain'] = data['url'].apply(lambda i: process_tld(i))
data.head()
feature = ['@','?','-','=','.','#','%','+','$','!','*',',','//']
for a in feature:
    data[a] = data['url'].apply(lambda i: i.count(a))
data.head()
def abnormal_url(url):
    hostname = urlparse(url).hostname
    hostname = str(hostname)
    match = re.search(hostname, url)
    if match:
        # print match.group()
        return 1
    else:
        # print 'No matching pattern found'
        return 0
data['abnormal_url'] = data['url'].apply(lambda i: abnormal_url(i))
sns.countplot(x='abnormal_url', data=data);
def httpSecure(url):
    htp = urlparse(url).scheme
    match = str(htp)
    if match=='https':
        # print match.group()
        return 1
    else:
        # print 'No matching pattern found'
        return 0
data['https'] = data['url'].apply(lambda i: httpSecure(i))
sns.countplot(x='https', data=data);<jupyter_output><empty_output><jupyter_text>Counts the number of digit characters in a URL<jupyter_code>def digit_count(url):
    digits = 0
    for i in url:
        if i.isnumeric():
            digits = digits + 1
    return digits
data['digits']= data['url'].apply(lambda i: digit_count(i))<jupyter_output><empty_output><jupyter_text>Counts the number of letter characters in a URL<jupyter_code>def letter_count(url):
    letters = 0
    for i in url:
        if i.isalpha():
            letters = letters + 1
    return letters
data['letters']= data['url'].apply(lambda i: letter_count(i))<jupyter_output><empty_output><jupyter_text>Checks to see whether URL contains a shortening service<jupyter_code>def Shortining_Service(url):
    match = re.search('bit\.ly|goo\.gl|shorte\.st|go2l\.ink|x\.co|ow\.ly|t\.co|tinyurl|tr\.im|is\.gd|cli\.gs|'
                      'yfrog\.com|migre\.me|ff\.im|tiny\.cc|url4\.eu|twit\.ac|su\.pr|twurl\.nl|snipurl\.com|'
                      'short\.to|BudURL\.com|ping\.fm|post\.ly|Just\.as|bkite\.com|snipr\.com|fic\.kr|loopt\.us|'
                      'doiop\.com|short\.ie|kl\.am|wp\.me|rubyurl\.com|om\.ly|to\.ly|bit\.do|t\.co|lnkd\.in|'
                      'db\.tt|qr\.ae|adf\.ly|goo\.gl|bitly\.com|cur\.lv|tinyurl\.com|ow\.ly|bit\.ly|ity\.im|'
                      'q\.gs|is\.gd|po\.st|bc\.vc|twitthis\.com|u\.to|j\.mp|buzurl\.com|cutt\.us|u\.bb|yourls\.org|'
                      'x\.co|prettylinkpro\.com|scrnch\.me|filoops\.info|vzturl\.com|qr\.net|1url\.com|tweez\.me|v\.gd|'
                      'tr\.im|link\.zip\.net',
                      url)
    if match:
        return 1
    else:
        return 0
data['Shortining_Service'] = data['url'].apply(lambda x: Shortining_Service(x))
sns.countplot(x='Shortining_Service', data=data);
def having_ip_address(url):
    match = re.search(
        '(([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\.([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\.([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\.'
        '([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\/)|'  # IPv4
        '(([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\.([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\.([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\.'
        '([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\/)|'  # IPv4 with port
        '((0x[0-9a-fA-F]{1,2})\\.(0x[0-9a-fA-F]{1,2})\\.(0x[0-9a-fA-F]{1,2})\\.(0x[0-9a-fA-F]{1,2})\\/)' # IPv4 in hexadecimal
        '(?:[a-fA-F0-9]{1,4}:){7}[a-fA-F0-9]{1,4}|'
        '([0-9]+(?:\.[0-9]+){3}:[0-9]+)|'
        '((?:(?:\d|[01]?\d\d|2[0-4]\d|25[0-5])\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d|\d)(?:\/\d{1,2})?)', url)  # Ipv6
    if match:
        return 1
    else:
        return 0
data['having_ip_address'] = data['url'].apply(lambda i: having_ip_address(i))
data['having_ip_address'].value_counts()
plt.figure(figsize=(15, 15))
sns.heatmap(data.corr(), linewidths=.5)
X = data.drop(['url','type','Category','domain'],axis=1)#,'type_code'
y = data['Category']<jupyter_output><empty_output><jupyter_text>✂️ Train & Test Split<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)<jupyter_output><empty_output><jupyter_text>Training models<jupyter_code>from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import plot_roc_curve
models = [DecisionTreeClassifier,RandomForestClassifier,AdaBoostClassifier,KNeighborsClassifier,SGDClassifier,
         ExtraTreesClassifier,GaussianNB]
accuracy_test=[]
for m in models:
    print('#############################################')
    print('######-Model =>\033[07m {} \033[0m'.format(m))
    model_ = m()
    model_.fit(X_train, y_train)
    pred = model_.predict(X_test)
    acc = accuracy_score(pred, y_test)
    accuracy_test.append(acc)
    print('Test Accuracy :\033[32m \033[01m {:.2f}% \033[30m \033[0m'.format(acc*100))
    print('\033[01m              Classification_report \033[0m')
    print(classification_report(y_test, pred))
    print('\033[01m             Confusion_matrix \033[0m')
    cf_matrix = confusion_matrix(y_test, pred)
    plot_ = sns.heatmap(cf_matrix/np.sum(cf_matrix), annot=True,fmt= '0.2%')
    plt.show()
    print('\033[31m###################- End -###################\033[0m')<jupyter_output>#############################################
######-Model =>[07m <class 'sklearn.tree._classes.DecisionTreeClassifier'> [0m
Test Accuracy :[32m [01m 90.93% [30m [0m
[01m              Classification_report [0m
              precision    recall  f1-score   support

           0       0.92      0.97      0.94     85565
           1       0.93      0.96      0.94     19319
           2       0.80      0.57      0.66     18805
           3       0.95      0.91      0.93      6550

    accuracy                           0.91    130239
   macro avg       0.90      0.85      0.87    130239
weighted avg       0.90      0.91      0.90    130239

[01m             Confusion_matrix [0m<jupyter_text>Final Report<jupyter_code>output = pd.DataFrame({"Model":['Decision Tree Classifier','Random Forest Classifier',
                                'AdaBoost Classifier','KNeighbors Classifier','SGD Classifier',
                                'Extra Trees Classifier','Gaussian NB'],
                      "Accuracy":accuracy_test})
output
plt.figure(figsize=(10, 5))
plots = sns.barplot(x='Model', y='Accuracy', data=output)
for bar in plots.patches:
    plots.annotate(format(bar.get_height(), '.2f'),
                   (bar.get_x() + bar.get_width() / 2,
                    bar.get_height()), ha='center', va='center',
                   size=15, xytext=(0, 8),
                   textcoords='offset points')

plt.xlabel("Models", size=14)
plt.xticks(rotation=20);
plt.ylabel("Accuracy", size=14)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1884603945672648, 'fraction_numerical': 0.046792901754733815, 'mean_word_length': 3.2818336162988118, 'pattern_counts': {'":': 7, '<': 53, '<?xml version=': 0, '>': 55, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '646571', 'license': 'MIT', 'n_tokens_mistral': 4088, 'n_tokens_neox': 3729, 'n_words': 668, 'path': 'detection-malicious-url-using-ml-models.ipynb', 'repo_name': 'hamza-manssor/Detection-Malicious-URL-Using-ML-Models', 'size': '10087'}