<jupyter_start><jupyter_text>Worksheet 3:  Detecting Domain Generation Algorithm (DGA) Domains against DNS ANSWERSThis worksheet covers concepts covered in the second half of Module 6 - Hunting with Data Science.  It should take no more than 20-30 minutes to complete.  Please raise your hand if you get stuck.  Your objective is to reduce a dataset that has thousands of domain names and identify those created by DGA. Import the LibrariesFor this exercise, we will be using:* Pandas (http://pandas.pydata.org/pandas-docs/stable/)* Flare (https://github.com/austin-taylor/flare)* Json (https://docs.python.org/3/library/json.html)* WHOIS (https://pypi.python.org/pypi/whois)Beacon writeup:  Detect Beaconing with Flare, Elastic Stack, and Intrusion Detection Systems<jupyter_code>from flare.data_science.features import entropy
from flare.data_science.features import dga_classifier
from flare.data_science.features import domain_tld_extract
from flare.tools.alexa import Alexa
from pandas.io.json import json_normalize
from whois import whois
import pandas as pd
import json
import warnings

warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>This is an example of how to generate domain generated algorithms.<jupyter_code>def generate_domain(year, month, day):
    """Generates a domain name for the given date."""
    domain = ""

    for i in range(16):
        year = ((year ^ 8 * year) >> 11) ^ ((year & 0xFFFFFFF0) << 17)
        month = ((month ^ 4 * month) >> 25) ^ 16 * (month & 0xFFFFFFF8)
        day = ((day ^ (day << 13)) >> 19) ^ ((day & 0xFFFFFFFE) << 12)
        domain += chr(((year ^ month ^ day) % 25) + 97)

    return domain + '.com'
generate_domain(2017, 6, 23)<jupyter_output><empty_output><jupyter_text>A large portion of data science is data preparation. In this exercise, we'll take output from Suricata's eve.json file and extract the DNS records so we can find anything using DGA.  First you'll need to **unzip the large_eve_json.zip file** in the data directory and specify the path.<jupyter_code>eve_json = '../data/large_eve.json'<jupyter_output><empty_output><jupyter_text>Next read the data in and build a list<jupyter_code>all_suricata_data = [json.loads(record) for record in open(eve_json).readlines()]
len(all_suricata_data)<jupyter_output><empty_output><jupyter_text>Our output from Suricata has 746909 records, and for we are only interested in DNS records. Let's narrow our data down to records that only contain dns  Read in suricata data and load each record as json if DNS is in the key. This will help pandas json_normalize feature<jupyter_code>dns_records = [json.loads(record) for record in open(eve_json).readlines() if 'dns' in record]
len(dns_records)<jupyter_output><empty_output><jupyter_text>Down to 21484 -- much better.  Somewhere in our _21484_ records is communication from infected computers. It's up to you to narrow the results down and find the malicious DNS request.<jupyter_code>dns_records[2]<jupyter_output><empty_output><jupyter_text>The data is nested json and has varying lengths, so you will need to use the json_normalize feature<jupyter_code>suricata_df = json_normalize(dns_records)
suricata_df.shape
suricata_df.head(2)<jupyter_output><empty_output><jupyter_text>Next we need to filter out all A records<jupyter_code>a_records = suricata_df[suricata_df['dns.rrtype']==('A')][['dns.rrname','dns.rrtype']]
a_records.shape<jupyter_output><empty_output><jupyter_text>By filtering out the A records, our dataset is down to 2849.<jupyter_code>a_records['dns.rrname'].value_counts().head()<jupyter_output><empty_output><jupyter_text>Next we can figure out how many unique DNS names there are.<jupyter_code>a_records_unique = pd.DataFrame(a_records['dns.rrname'].unique(), columns=['dns_rrname'])
len(a_records_unique)<jupyter_output><empty_output><jupyter_text>Down to 177 records to process!<jupyter_code>a_records_unique.head()<jupyter_output><empty_output><jupyter_text>Next we need to train extract the top level domains (remove subdomains) using flare so we can feed it to our classifier<jupyter_code>a_records_unique['domain_tld'] = a_records_unique.dns_rrname.apply(domain_tld_extract)
a_records_unique.head()<jupyter_output><empty_output><jupyter_text>Train DGA Classifier with dictionary words, n-grams and DGA Domains<jupyter_code>dga_predictor = dga_classifier()
a_records_unique['dga_predict'] = a_records_unique.domain_tld.apply(lambda x: dga_predictor.predict(x))<jupyter_output><empty_output><jupyter_text>A quick sampling of the data shows our prediction has labelled our data.<jupyter_code>a_records_unique.sample(10)
dga_df = a_records_unique[a_records_unique.dga_predict=='dga']
dga_df<jupyter_output><empty_output><jupyter_text>Our dataset is down to 5 results! Let's run the domains through alexa to see if ny are in the top 1 million<jupyter_code>alexa = Alexa()
dga_df['in_alexa'] = dga_df.dns_rrname.apply(alexa.domain_in_alexa)
def get_creation_date(domain):
    try:
        lookup = whois(domain)
        output = lookup.get('creation_date','No results')
    except:
        output = 'No Creation Date!'
    if output is None:
        output = 'No Creation Date!'
    return output
get_creation_date('google.com')
dga_df<jupyter_output><empty_output><jupyter_text>It appears none of our domains are in Alexa, but let's check creation dates.<jupyter_code>dga_df['creation_date'] = dga_df.dns_rrname.apply(get_creation_date)
dga_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09636795035590436, 'fraction_numerical': 0.012776054024457018, 'mean_word_length': 6.144719687092569, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '170441', 'license': 'MIT', 'n_tokens_mistral': 1894, 'n_tokens_neox': 1779, 'n_words': 510, 'path': 'answers/Worksheet 3 - Hunting with Data Science - Answers.ipynb', 'repo_name': 'd3vzer0/applied_data_science_amsterdam', 'size': '5479'}