<jupyter_start><jupyter_text>Visualize CSV Mini-App* Jupyter: `File` -> `Make a copy`    Colab: `File` -> `Save a copy in Drive`* Run notebook cells by pressing `shift-enter`* Either edit annd run top cells one-by-one, or edit and run the self-contained version at the bottom<jupyter_code>#!pip install graphistry -q
import pandas as pd
import graphistry

# To specify Graphistry account & server, use:
# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')
# For more options, see https://github.com/graphistry/pygraphistry#configure<jupyter_output><empty_output><jupyter_text>1. Upload csvUse a file by uploading it or via URL. Run `help(pd.read_csv)` for more options. File Upload: Jupyter Notebooks* If circle on top right not green, click `kernel` -> `reconnect`* Go to file directory (`/tree`) by clicking the Jupyter logo* Navigate to the directory page containing your notebook* Press the `upload` button on the top right File Upload: Google Colab* Open the left sidebar by pressing the right arrow on the left* Go to the `Files` tab* Press `UPLOAD`* Make sure goes into `/content` File Upload: URL* Uncomment below line and put in the actual data url* Run `help(pd.read_csv)` for more options<jupyter_code>file_path = './events-1551346702.csv'
df = pd.read_csv(file_path)

print('# rows', len(df))
df.sample(min(len(df), 3))
df.columns<jupyter_output><empty_output><jupyter_text>2. Optional: Clean up CSV<jupyter_code>hits = pd.DataFrame([[c, len(df[c].unique())] for c in df.columns], columns=['col', 'num_uniq']).sort_values('num_uniq')

hits.query('num_uniq > 10 & num_uniq < 9288')
skip_nodes = ['icij_notes', 'determined_cause', 'action_classification', 'icij_notes', 'country', 'status', 'source']
nodes = [x for x in list(hits.query('num_uniq > 10 & num_uniq < 9288')['col']) if not x in skip_nodes]

nodes
df = df_orig.query('country == "USA"')<jupyter_output><empty_output><jupyter_text>3. Configure: Visualize with 3 kinds of graphsSet `mode` and the corresponding values: Mode "A". See graph from table of (src,dst) edges Mode "B". See hypergraph: Draw row as node and connect it to entities in same row* Pick which cols to make nodes* If multiple cols share same type (e.g., "src_ip", "dest_ip" are both "ip"), unify them Mode "C". See by creating multiple nodes, edges per row* Pick how different column values point to other column values * If multiple cols share same type (e.g., "src_ip", "dest_ip" are both "ip"), unify them<jupyter_code>#Pick 'A', 'B', or 'C'
mode = 'B' 
max_rows = 50000


### 'A' == mode
my_src_col = 'attackerIP'
my_dest_col = 'victimIP'



### 'B' == mode
node_cols = nodes
categories = { #optional
    #'date': [ 'create_date', 'date_initiated_by_firm', 'date_posted', 'date_terminated', 'updated_at' ]
    #'ip': ['attacker_IP', 'victimIP']
    #, 'user': ['owner', 'seller'],
}



### 'C' == mode
edges = {
      'attackerIP': [ 'victimIP', 'victimPort', 'vulnName'],
      'victimIP': [ 'victimPort'],
      'vulnName': [ 'victimIP' ]
}
categories = { #optional
      'ip': ['attackerIP', 'victimIP']
       #, user': ['owner', 'seller'], ...
}<jupyter_output><empty_output><jupyter_text>4. Plot: Upload & render!* See UI guide: https://labs.graphistry.com/graphistry/ui.html<jupyter_code>g = None
hg = None
num_rows = min(max_rows, len(df))
if mode == 'A':
    g = graphistry.edges(df.sample(num_rows)).bind(source=my_src_col, destination=my_dest_col)
elif mode == 'B':
    hg = graphistry.hypergraph(df.sample(num_rows), node_cols, opts={'CATEGORIES': categories})
    g = hg['graph']
elif mode == 'C':
    nodes = list(edges.keys())
    for dests in edges.values():
        for dest in dests:
            nodes.append(dest)
    node_cols = list(set(nodes))
    hg = graphistry.hypergraph(df.sample(num_rows), node_cols, direct=True, opts={'CATEGORIES': categories, 'EDGES': edges})
    g = hg['graph']
  
#hg
print(len(g._edges))

g.plot()<jupyter_output># links 173770
# events 35826
# attrib entities 14368
173770
Uploading 84603 kB. This may take a while...<jupyter_text>Alternative: CombinedSplit into data loading and cleaning/configuring/plotting.<jupyter_code>#!pip install graphistry -q
import pandas as pd
import graphistry
#graphistry.register(key='MY_KEY', server='labs.graphistry.com')


##########
#1. Load
file_path = './events-1551346702.csv'
df = pd.read_csv(file_path)

print(df.columns)
print('rows:', len(df))
print(df.sample(min(len(df),3)))
##########
#2. Clean
#df = df.rename(columns={'attackerIP': 'src_ip', 'victimIP: 'dest_ip', 'victimPort': 'protocol'})

hits = pd.DataFrame([[c, len(df[c].unique())] for c in df.columns], columns=['col', 'num_uniq']).sort_values('num_uniq')


skip_nodes = ['icij_notes', 'determined_cause', 'action_classification', 'icij_notes', 'country', 'status', 'source']
nodes = [x for x in list(hits.query('num_uniq > 10 & num_uniq < 9288')['col']) if not x in skip_nodes]

df = df.query('country == "USA"')
    
##########
#3. Config - Pick 'A', 'B', or 'C'
mode = 'B' 
max_rows = 50000


### 'A' == mode
my_src_col = 'attackerIP'
my_dest_col = 'victimIP'

### 'B' == mode
node_cols = nodes
categories = { #optional
    #'ip': ['src_ip', 'dest_ip']
    #, 'user': ['owner', 'seller'],
}

### 'C' == mode
edges = {
    'attackerIP': [ 'victimIP', 'victimPort', 'vulnName'],
    'victimIP': [ 'victimPort' ],
    'vulnName': ['victimIP' ]
}
categories = { #optional
    #'ip': ['attackerIP', 'victimIP']
    #, 'user': ['owner', 'seller'], ...
}

##########
#4. Plot
g = None
hg = None
num_rows = min(max_rows, len(df))
if mode == 'A':
    g = graphistry.edges(df.sample(num_rows)).bind(source=my_src_col, destination=my_dest_col)
elif mode == 'B':
    hg = graphistry.hypergraph(df.sample(num_rows), node_cols, opts={'CATEGORIES': categories})
    g = hg['graph']
elif mode == 'C':
    nodes = list(edges.keys())
    for dests in edges.values():
        for dest in dests:
            nodes.append(dest)
    node_cols = list(set(nodes))
    hg = graphistry.hypergraph(df.sample(num_rows), node_cols, direct=True, opts={'CATEGORIES': categories, 'EDGES': edges})
    g = hg['graph']
  

g.plot()<jupyter_output># links 173770
# events 35826
# attrib entities 14368
Uploading 84590 kB. This may take a while...
{'fraction_non_alphanumeric': 0.16908597862498007, 'fraction_numerical': 0.017227628010847026, 'mean_word_length': 4.800185013876041, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '214337', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2359, 'n_tokens_neox': 2230, 'n_words': 617, 'path': 'demos/more_examples/simple/tutorial_csv_mini_app_icij_implants.ipynb', 'repo_name': 'vishalbelsare/pygraphistry', 'size': '6269'}