<jupyter_start><jupyter_text>Data Cleaning and Feature Extraction of ` Phishing URLs `<jupyter_code>#importing required packages for this module
import pandas as pd<jupyter_output><empty_output><jupyter_text>**2.1. Phishing URLs:**<jupyter_code>#loading the phishing URLs data to dataframe
phishurl = pd.read_csv("3-phishurl.csv")
phishurl.head()<jupyter_output><empty_output><jupyter_text>**3.Feature Extraction**In this step,features are extracted from the URLs dataset.The extracted features are categorized into1.   Address Bar based Features2.   Domain based Features3.   HTML & Javascript Features **3.1.Address Bar Based Features:**Many features can be extracted that can be consided as address bar base them, below mentioned were considered for this project.*   Domain of URL *   IP Address in URL*   "@"Symbol in URL* Length of URL* Depth of URL* Redirection "//" in URL* "http/https" in Domain name* Using URL Shortening Services "TinyURL"* Prefix or Suffix "-" in DomainEach of these features are explained and the coded below:<jupyter_code>#importing required packages for this section
from urllib.parse import urlparse,urlencode
import urllib
import urllib.request
import ipaddress
import re
from bs4 import BeautifulSoup
import whois
from datetime import datetime
import requests<jupyter_output><empty_output><jupyter_text>**3.1.1.Domain of the URL**<jupyter_code># 1.Domain of the URL(Domain) 
def getDomain(url):
  domain = urlparse(url).netloc
  if re.match(r"^www.",domain):
       domain = domain.replace("www.","")
  return domain<jupyter_output><empty_output><jupyter_text>**3.1.2.IP Address in the URL**<jupyter_code># 2.Checks for IP address in URL (Have_IP)
def havingIP(url):
  try:
    ipaddress.ip_address(url)
    ip = 1
  except:
    ip = 0
  return ip<jupyter_output><empty_output><jupyter_text>**3.1.3. "@" Symbol in URL**<jupyter_code># 3.Checks the presence of @ in URL (Have_At)
def haveAtSign(url):
  if "@" in url:
    at = 1    
  else:
    at = 0    
  return at<jupyter_output><empty_output><jupyter_text>**3.1.4. Length of URL**<jupyter_code># 4.Finding the length of URL and categorizing (URL_Length)
def getLength(url):
  if len(url) < 54:
    length = 0            
  else:
    length = 1            
  return length<jupyter_output><empty_output><jupyter_text>**3.1.5. Depth of URL**<jupyter_code># 5.Gives number of '/' in URL (URL_Depth)
def getDepth(url):
  s = urlparse(url).path.split('/')
  depth = 0
  for j in range(len(s)):
    if len(s[j]) != 0:
      depth = depth+1
  return depth<jupyter_output><empty_output><jupyter_text>**3.1.6. Redirection "//" in URL**<jupyter_code># 6.Checking for redirection '//' in the url (Redirection)
def redirection(url):
  pos = url.rfind('//')
  if pos > 6:
    if pos > 7:
      return 1
    else:
      return 0
  else:
    return 0<jupyter_output><empty_output><jupyter_text>**3.1.7. "http/https" in Domain name**<jupyter_code># 7.Existence of “HTTPS” Token in the Domain Part of the URL (https_Domain)
def httpDomain(url):
  domain = urlparse(url).netloc
  if 'https' in domain:
    return 1
  else:
    return 0<jupyter_output><empty_output><jupyter_text>**3.1.8. Using URL Shortening Services “TinyURL”**<jupyter_code>#listing shortening services
shortening_services = r"bit\.ly|goo\.gl|shorte\.st|go2l\.ink|x\.co|ow\.ly|t\.co|tinyurl|tr\.im|is\.gd|cli\.gs|" \
                      r"yfrog\.com|migre\.me|ff\.im|tiny\.cc|url4\.eu|twit\.ac|su\.pr|twurl\.nl|snipurl\.com|" \
                      r"short\.to|BudURL\.com|ping\.fm|post\.ly|Just\.as|bkite\.com|snipr\.com|fic\.kr|loopt\.us|" \
                      r"doiop\.com|short\.ie|kl\.am|wp\.me|rubyurl\.com|om\.ly|to\.ly|bit\.do|t\.co|lnkd\.in|db\.tt|" \
                      r"qr\.ae|adf\.ly|goo\.gl|bitly\.com|cur\.lv|tinyurl\.com|ow\.ly|bit\.ly|ity\.im|q\.gs|is\.gd|" \
                      r"po\.st|bc\.vc|twitthis\.com|u\.to|j\.mp|buzurl\.com|cutt\.us|u\.bb|yourls\.org|x\.co|" \
                      r"prettylinkpro\.com|scrnch\.me|filoops\.info|vzturl\.com|qr\.net|1url\.com|tweez\.me|v\.gd|" \
                      r"tr\.im|link\.zip\.net"
# 8. Checking for Shortening Services in URL (Tiny_URL)
def tinyURL(url):
    match=re.search(shortening_services,url)
    if match:
        return 1
    else:
        return 0<jupyter_output><empty_output><jupyter_text>**3.1.9. Prefix or Suffix "-" in Domain**<jupyter_code># 9.Checking for Prefix or Suffix Separated by (-) in the Domain (Prefix/Suffix)
def prefixSuffix(url):
    if '-' in urlparse(url).netloc:
        return 1            # phishing
    else:
        return 0            # legitimate<jupyter_output><empty_output><jupyter_text>**3.2. Domain Based Features:**Many features can be extracted that come under this category. Out of them, below mentioned were considered for this project.*  DNS Record* Website Traffic*Age of Domain*End Period of Domain **3.2.1.DNS Record**<jupyter_code># 11.DNS Record availability (DNS_Record)
# obtained in the featureExtraction function itself<jupyter_output><empty_output><jupyter_text>**3.2.2. Web Traffic**<jupyter_code># 12.Web traffic (Web_Traffic)
def web_traffic(url):
  try:
    #Filling the whitespaces in the URL if any
    url = urllib.parse.quote(url)
    rank = BeautifulSoup(urllib.request.urlopen("http://data.alexa.com/data?cli=10&dat=s&url=" + url).read(), "xml").find(
        "REACH")['RANK']
    rank = int(rank)
  except TypeError:
        return 1
  if rank <100000:
    return 1
  else:
    return 0<jupyter_output><empty_output><jupyter_text>**3.2.3. Age of Domain**<jupyter_code># 13.Survival time of domain: The difference between termination time and creation time (Domain_Age)  
def domainAge(domain_name):
  creation_date = domain_name.creation_date
  expiration_date = domain_name.expiration_date
  if (isinstance(creation_date,str) or isinstance(expiration_date,str)):
    try:
      creation_date = datetime.strptime(creation_date,'%Y-%m-%d')
      expiration_date = datetime.strptime(expiration_date,"%Y-%m-%d")
    except:
      return 1
  if ((expiration_date is None) or (creation_date is None)):
      return 1
  elif ((type(expiration_date) is list) or (type(creation_date) is list)):
      return 1
  else:
    ageofdomain = abs((expiration_date - creation_date).days)
    if ((ageofdomain/30) < 6):
      age = 1
    else:
      age = 0
  return age<jupyter_output><empty_output><jupyter_text>**3.2.4. End Period of Domain**<jupyter_code># 14.End time of domain: The difference between termination time and current time (Domain_End) 
def domainEnd(domain_name):
  expiration_date = domain_name.expiration_date
  if isinstance(expiration_date,str):
    try:
      expiration_date = datetime.strptime(expiration_date,"%Y-%m-%d")
    except:
      return 1
  if (expiration_date is None):
      return 1
  elif (type(expiration_date) is list):
      return 1
  else:
    today = datetime.now()
    end = abs((expiration_date - today).days)
    if ((end/30) < 6):
      end = 0
    else:
      end = 1
  return end<jupyter_output><empty_output><jupyter_text>**3.3. HTML and JavaScript based Features**Many features can be extracted that come under this category. Out of them, below mentioned were considered for this project.*  IFrame Redirection*Status Bar Customization*Disabling Right Click*Website Forwarding  **3.3.1. IFrame Redirection**<jupyter_code># 15. IFrame Redirection (iFrame)
def iframe(response):
  if response == "":
      return 1
  else:
      if re.findall(r"[<iframe>|<frameBorder>]", response.text):
          return 0
      else:
          return 1<jupyter_output><empty_output><jupyter_text>**3.3.2. Status Bar Customization**<jupyter_code># 16.Checks the effect of mouse over on status bar (Mouse_Over)
def mouseOver(response): 
  if response == "" :
    return 1
  else:
    if re.findall("<script>.+onmouseover.+</script>", response.text):
      return 1
    else:
      return 0<jupyter_output><empty_output><jupyter_text>**3.3.3. Disabling Right Click**<jupyter_code># 17.Checks the status of the right click attribute (Right_Click)
def rightClick(response):
  if response == "":
    return 1
  else:
    if re.findall(r"event.button ?== ?2", response.text):
      return 0
    else:
      return 1<jupyter_output><empty_output><jupyter_text>**3.3.4. Website Forwarding**<jupyter_code># 18.Checks the number of forwardings (Web_Forwards)    
def forwarding(response):
  if response == "":
    return 1
  else:
    if len(response.history) <= 2:
      return 0
    else:
      return 1<jupyter_output><empty_output><jupyter_text>**4. Computing URL Features**<jupyter_code>#Function to extract features
def featureExtraction(url,label):

  features = []
  #Address bar based features (10)
  features.append(getDomain(url))
  features.append(havingIP(url))
  features.append(haveAtSign(url))
  features.append(getLength(url))
  features.append(getDepth(url))
  features.append(redirection(url))
  features.append(httpDomain(url))
  features.append(tinyURL(url))
  features.append(prefixSuffix(url))
  
  #Domain based features (4)
  dns = 0
  try:
    domain_name = whois.whois(urlparse(url).netloc)
  except:
    dns = 1

  features.append(dns)
  features.append(web_traffic(url))
  features.append(1 if dns == 1 else domainAge(domain_name))
  features.append(1 if dns == 1 else domainEnd(domain_name))
  
  # HTML & Javascript based features (4)
  try:
    response = requests.get(url)
  except:
    response = ""
  features.append(iframe(response))
  features.append(mouseOver(response))
  features.append(rightClick(response))
  features.append(forwarding(response))
  features.append(label)
  
  return features<jupyter_output><empty_output><jupyter_text>**4.2. Reviewing Phishing URLs:**<jupyter_code>phishurl.shape<jupyter_output><empty_output><jupyter_text>We will Reviewing URls 2500 by 2500 and storing seperate files and merge them.<jupyter_code>feature_names = ['Domain', 'Have_IP', 'Have_At', 'URL_Length', 'URL_Depth','Redirection', 
                      'https_Domain', 'TinyURL', 'Prefix/Suffix', 'DNS_Record', 'Web_Traffic', 
                      'Domain_Age', 'Domain_End', 'iFrame', 'Mouse_Over','Right_Click', 'Web_Forwards', 'Label']

label = 1<jupyter_output><empty_output><jupyter_text>0 - 5000 phishing URLs Feature extraction<jupyter_code>#Extracting the feautres & storing them in a list
phish_features = []

for i in range(0, 5000):
  url = phishurl['url'][i]
  print(i)
  phish_features.append(featureExtraction(url,label))
#converting the list to dataframe
phishing = pd.DataFrame(phish_features, columns= feature_names)
phishing.head()
# Storing the extracted legitimate URLs fatures to csv file
phishing.to_csv('5-phish_features.csv', index= False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13087248322147652, 'fraction_numerical': 0.01761744966442953, 'mean_word_length': 3.957948243992606, 'pattern_counts': {'":': 3, '<': 106, '<?xml version=': 0, '>': 103, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '53953', 'license': 'MIT', 'n_tokens_mistral': 3956, 'n_tokens_neox': 3690, 'n_words': 858, 'path': 'Constructing_ML_Model/2_Feature_Extraction_of_Phishing_URL.ipynb', 'repo_name': 'Hansajith98/Trawling-Chrome-Extention', 'size': '10728'}