<jupyter_start><jupyter_text>AWS CloudWatch VPC Flow Logs  GraphistryAnalyze cloudwatch logs, such as using vpc flow to map an account, with GraphistryThis example directly uses the AWS CLI for cloudwatch API access. You can also work from S3 or systems like Athena.  Installs & ConfigureSet `aws_access_key_id`, `aws_secret_access_key`, `key` or pull from your env<jupyter_code>!pip install graphistry -q
!pip install awscli -q
!aws configure set region us-west-2
!aws configure set aws_access_key_id "FILL_ME_IN"
!aws configure set aws_secret_access_key "FILL_ME_IN"
import pandas as pd
import json
import graphistry

# To specify Graphistry account & server, use:
# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')
# For more options, see https://github.com/graphistry/pygraphistry#configure<jupyter_output><empty_output><jupyter_text>Record logsIf you do not already have logs, you can record VPC flow logs from your EC2 console:  * Services -> EC2 -> Network Interfaces -> select interface(s) ->  Action -> create flow log    * Send to cloudwatch; use default settings for IAM and elsewhere  * When enough data available, stop logging  Download & summarize logs* Pick a log group from available* Fetch: See AWS docs on [filter-log-events](https://docs.aws.amazon.com/cli/latest/reference/logs/filter-log-events.html)* Load into a dataframe* Compute summary stats<jupyter_code>!aws logs describe-log-groups
!aws logs filter-log-events --log-group-name VPCFlowDemo > data.json
!ls -al data.json
with open('data.json', 'r') as f:
    data = json.load(f)
df = pd.DataFrame([x['message'].split(" ") for x in data['events']])
df.columns = cols = ['version', 'accountid', 'interfaceid', 'src_ip', 'dest_ip', 'src_port', 'dest_port', 'protocol', 'packets', 'bytes', 'time_start', 'time_end', 'action', 'status']

print('# rows', len(df))
df.sample(3)
# Int->Float for precision errors
df2 = df.copy()
for c in ['packets', 'bytes']:
    df2[c] = df2[c].astype(float)

summary_df = df2\
    .groupby(['src_ip', 'dest_ip', 'interfaceid', 'dest_port', 'protocol', 'action', 'status'])\
    .agg({
        'time_start': ['min', 'max'],
        'time_end': ['min', 'max'],
        'packets': ['min', 'max', 'sum', 'count'],
        'bytes': ['min', 'max', 'sum', 'count']
    }).reset_index()
summary_df.columns = [(" ".join(x)).strip().replace(" ", "_") for x in list(summary_df.columns)]
print('# rows', len(summary_df))
summary_df.sample(3)<jupyter_output># rows 5049<jupyter_text>Plot<jupyter_code>hg = graphistry.hypergraph(
    summary_df,
    entity_types=['src_ip', 'dest_ip'], #'dest_port', 'interfaceid', 'action', ...
    direct=True)
hg['graph'].bind(edge_title='bytes_sum').plot()<jupyter_output># links 5049
# events 5049
# attrib entities 255
{'fraction_non_alphanumeric': 0.15472370766488414, 'fraction_numerical': 0.009269162210338681, 'mean_word_length': 5.1400437636761485, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '183487', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 943, 'n_tokens_neox': 920, 'n_words': 272, 'path': 'demos/demos_by_use_case/logs/aws_vpc_flow_cloudwatch/vpc_flow.ipynb', 'repo_name': 'vishalbelsare/pygraphistry', 'size': '2805'}