<jupyter_start><jupyter_text>Analyze Pcap  Imports & Setup<jupyter_code>from datetime import timedelta
import pandas

from steelscript.netshark.core import NetShark
from steelscript.netshark.core.types import Value, Key
from steelscript.netshark.core.filters import NetSharkFilter, TimeFilter
from steelscript.common import UserAuth

from steelscript.common.datautils import bytes2human
# optional debug logging
import logging
logger = logging.getLogger()
logging.basicConfig(format="%(asctime)s [%(levelname)-8.5s] (%(name)s) %(msg)s", level=logging.DEBUG)
logger.setLevel(logging.DEBUG)
hostname = "NETSHARK.HOSTNAME.COM"
username = "USERNAME"
password = "PASSWORD"<jupyter_output><empty_output><jupyter_text>Connect to the NetShark device<jupyter_code>netshark = NetShark(hostname, auth=UserAuth(username, password))
netshark<jupyter_output><empty_output><jupyter_text>Pick the sourceChoose the first running job.<jupyter_code>jobs = netshark.get_capture_jobs()
source = None
for j in jobs:
    if j.get_status()['state'] == 'RUNNING':
        source = j
        break
source
timefilter = TimeFilter.parse_range('last 10s')
srvport = 443
portfilter = NetSharkFilter('tcp.server_port=%d' % srvport)
export = netshark.create_export(source, timefilter, filters=[portfilter])<jupyter_output><empty_output><jupyter_text>Note that the export details does not represent the port 22 filter - it really represents the maximum possible size of the resulting packet set until the export is actually used below<jupyter_code>export.details()<jupyter_output><empty_output><jupyter_text>The `export` object is used to download the packets locally to disk.  Once the download occurs, the export object is deleted on the NetShark side of things.<jupyter_code>export.download('/tmp/telnet.pcap', overwrite=True)<jupyter_output><empty_output><jupyter_text>Analyze a pcap file using Wireshark(or more precisely, tshark -- the command line version of wireshark)The module `steelscript.wireshark` provides a Python wrapper around tshark, making it easy to inspect and perform queries on pcap files.<jupyter_code>from steelscript.wireshark.core.pcap import PcapFile

pcap = PcapFile('/tmp/telnet.pcap')
pcap.info()
print pcap.starttime
print pcap.endtime
print pcap.numpackets<jupyter_output><empty_output><jupyter_text>The `TSharkFields` class allows for searching for fields by name or protocol:<jupyter_code>from steelscript.wireshark.core.pcap import TSharkFields
fields = TSharkFields()
fields.find(name_re='^tcp.*rtt')
fields.find(name_re='^tcp.*flight')<jupyter_output><empty_output><jupyter_text>Query the pcap file for a set of packet fields<jupyter_code>df = pcap.query(['frame.time_epoch', 'ip.src', 'ip.dst', 'ip.len', 
                 'tcp.srcport', 'tcp.dstport', 
                 'tcp.flags', 'tcp.analysis.ack_rtt'], 
                 filterexpr='tcp',
                 as_dataframe=True)
df[:10]<jupyter_output><empty_output><jupyter_text>Filter for rows that have an RTT<jupyter_code>has_rtt = (~df['tcp.analysis.ack_rtt'].isnull())
df[has_rtt][:10]<jupyter_output><empty_output><jupyter_text>Assign client/server IP based on port<jupyter_code># Assign the column dst to:
#    src_true for all rows where expr is True
#    src_false for all rows where expr is False

def assign_col(df, expr, dst, src_true, src_false):
    df.ix[expr, dst]  = df.ix[expr, src_true]
    df.ix[~expr, dst] = df.ix[~expr, src_false]    
# This boolean expression assigns a True/False value for each row 
# based on the tcp.srcport.  Since we know the filter was on port 22
# than if the srcport is 22, the source is the server.  Otherwise
# the dest is the server.
srv_src = (df['tcp.srcport'] == 22)

srv_src[:10]<jupyter_output><empty_output><jupyter_text>Now we can use this "expression" to set the client/server IP and port.<jupyter_code>assign_col(df, srv_src, 'ip.cli',      'ip.dst',      'ip.src')
assign_col(df, srv_src, 'ip.srv',      'ip.src',      'ip.dst')
assign_col(df, srv_src, 'tcp.cliport', 'tcp.dstport', 'tcp.srcport')

df = df.ix[:,['frame.time_epoch', 'ip.srv', 'ip.cli', 'tcp.cliport', 
              'ip.len', 'tcp.flags', 'tcp.analysis.ack_rtt']]
df[:10]<jupyter_output><empty_output><jupyter_text>Group by conversationIn this case a unique conversation defined by the triplet: Client/Server IP and Client Port (since the server port is always 22, we don't bother)<jupyter_code>gb = df.groupby(['ip.cli', 'ip.srv', 'tcp.cliport'])<jupyter_output><empty_output><jupyter_text>The `gb` object itself is not an interesting result -- instead, you then frequently use it to aggregate rows that have the same index (the triplet)<jupyter_code>agg = gb.aggregate({'ip.len': 'sum',
                    'tcp.analysis.ack_rtt': ['min', 'mean', 'max', 'count']})
agg<jupyter_output><empty_output><jupyter_text>Lets plot some of this<jupyter_code>%pylab inline<jupyter_output><empty_output><jupyter_text>Compute the index of the row in the aggregate data that has the highest number of samples<jupyter_code>rtt_index = agg.sort(columns=[('tcp.analysis.ack_rtt', 'count')], ascending=False).index[0]<jupyter_output><empty_output><jupyter_text>Now index into the original `df` using the `rtt_index` -- this allows us to filter on only packets that were assocaited with the client/server/port index value.<jupyter_code>clisrv = df.set_index(agg.index.names)
s = clisrv[clisrv.index.isin([rtt_index])]<jupyter_output><empty_output><jupyter_text>Index this filtered data set on time and graph packet size (ie. `ip.len`)<jupyter_code>s = s.set_index('frame.time_epoch')
s.plot(y=['ip.len'])<jupyter_output><empty_output><jupyter_text>Now plot of few other interesting metrics, like computing bit rate -- the sum of ip.len across all rows in a 1 second interval:<jupyter_code>s.resample('1s', {'ip.len': 'sum'}).plot()
s.plot(y=['tcp.analysis.ack_rtt'])
s.resample('100L', {'tcp.analysis.ack_rtt': ['min', 'mean', 'max']}).plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12249705535924617, 'fraction_numerical': 0.005384485949856975, 'mean_word_length': 5.80091533180778, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '469340', 'license': 'MIT', 'n_tokens_mistral': 2054, 'n_tokens_neox': 2016, 'n_words': 513, 'path': 'notebooks/05-netshark-analyze-pcap.ipynb', 'repo_name': 'riverbed/steelscript-netshark', 'size': '5943'}