<jupyter_start><jupyter_text>Deep Learning Toolkit for Splunk - Graph Algorithms with Rapids cuGraph This notebook contains examples for Louvain graph algorithm available in Rapids Note: By default every time you save this notebook the cells are exported into a python module which is then invoked by Splunk MLTK commands like  | fit ... | apply ... | summary . Please read the Model Development Guide in the Deep Learning Toolkit app for more information.  Stage 0 - import librariesAt stage 0 we define all imports necessary to run our subsequent code depending on various libraries.<jupyter_code># this definition exposes all python module imports that should be available in all subsequent commands
import json
import numpy as np
import pandas as pd
import cugraph
import cudf
# ...
# global constants
MODEL_DIRECTORY = "/srv/app/model/data/"
# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes
print("numpy version: " + np.__version__)
print("pandas version: " + pd.__version__)
print("cugraph version: " + cugraph.__version__)
print("cudf version: " + cudf.__version__)

# Print out GPU Name
cudf.utils.cudautils.cuda.detect()<jupyter_output>numpy version: 1.18.1
pandas version: 0.25.3
cugraph version: 0.13.0+0.gac36e8c.dirty
cudf version: 0.13.0
Found 1 CUDA devices
id 0    b'Tesla V100-SXM2-16GB'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 30
                              pci bus id: 0
Summary:
	1/1 devices are supported<jupyter_text>Stage 1 - get a data sample from SplunkIn Splunk run a search to pipe a dataset into your notebook environment. Note: mode=stage is used in the | fit command to do this. | inputlookup bitcoin_transactions.csv| rename user_id_from as src user_id_to as dest| fit MLTKContainer mode=stage algo=graph_algo_louvain from src dest into app:bitcoin_graph_louvain as graph After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword ("barebone_model" in the example above) or set to "default" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model.<jupyter_code># this cell is not executed from MLTK and should only be used for staging data into the notebook environment
def stage(name):
    with open("data/"+name+".csv", 'r') as f:
        df = pd.read_csv(f)
    with open("data/"+name+".json", 'r') as f:
        param = json.load(f)
    return df, param
# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes
df, param = stage("bitcoin_graph_louvain")
print(df[0:1])
print(param)<jupyter_output>src  dest
0    2     2
{'options': {'params': {'mode': 'stage', 'algo': 'graph_algo_louvain'}, 'feature_variables': ['src', 'dest'], 'args': ['src', 'dest'], 'model_name': 'bitcoin_graph_louvain', 'output_name': 'graph', 'algo_name': 'MLTKContainer', 'mlspl_limits': {'handle_new_cat': 'default', 'max_distinct_cat_values': '1000', 'max_distinct_cat_values_for_classifiers': '1000', 'max_distinct_cat_values_for_scoring': '1000', 'max_fit_time': '6000', 'max_inputs': '1000000000', 'max_memory_usage_mb': '16000', 'max_model_size_mb': '1500', 'max_score_time': '6000', 'streaming_apply': 'false', 'use_sampling': 'true'}, 'kfold_cv': None}, 'feature_variables': ['src', 'dest']}<jupyter_text>Stage 2 - create and initialize a model<jupyter_code># initialize your model
# available inputs: data and parameters
# returns the model object which will be used as a reference to call fit, apply and summary subsequently
def init(df,param):
    model = {}
    return model
# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes
model = init(df,param)<jupyter_output><empty_output><jupyter_text>Stage 3 - fit the model<jupyter_code># train your model
# returns a fit info json object and may modify the model object
def fit(model,df,param):
    model = {}
    return model
# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes
fit(model,df,param)<jupyter_output><empty_output><jupyter_text>Stage 4 - apply the model<jupyter_code>src_dest_name = param['feature_variables']
# apply your model
# returns the calculated results
def apply(model,df,param):

    src_dest_name = param['feature_variables']
    dfg = df[src_dest_name]
    gdf = cudf.DataFrame(dfg)

    # create graph 
    G = cugraph.Graph()
    G.from_cudf_edgelist(gdf, source='src', destination='dest', renumber=True)
    max_iter = 100
    if 'max_iter' in param['options']['params']:
        max_iter = int(param['options']['params']['max_iter'])

    # cugraph Louvain Call
    dfr, mod = cugraph.louvain(G)
    dfr = dfr.to_pandas().rename(columns={"vertex": src_dest_name[0]})                   
    df = df.join(dfr.set_index(src_dest_name[0]), on=src_dest_name[0])
    df = df.rename(columns={"partition": src_dest_name[0]+"_partition"})   
    dfr = dfr.rename(columns={src_dest_name[0]: src_dest_name[1]})
    df = df.join(dfr.set_index(src_dest_name[1]), on=src_dest_name[1])
    df = df.rename(columns={"partition": src_dest_name[1]+"_partition"})   
    model['louvain_modularity'] = mod
    return df
# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes
%time result = apply(model,df,param)
result<jupyter_output>CPU times: user 60.3 ms, sys: 7.9 ms, total: 68.2 ms
Wall time: 66.6 ms<jupyter_text>Stage 5 - save the model<jupyter_code># save model to name in expected convention "<algo_name>_<model_name>"
def save(model,name):
    # with open(MODEL_DIRECTORY + name + ".json", 'w') as file:
    #    json.dump(model, file)
    return model<jupyter_output><empty_output><jupyter_text>Stage 6 - load the model<jupyter_code># load model from name in expected convention "<algo_name>_<model_name>"
def load(name):
    model = init(None,None)
    # with open(MODEL_DIRECTORY + name + ".json", 'r') as file:
    #    model = json.load(file)
    return model<jupyter_output><empty_output><jupyter_text>Stage 7 - provide a summary of the model<jupyter_code># return a model summary
def summary(model=None):
    returns = {"version": {"pandas": pd.__version__, "cudf": cudf.__version__, "cugraph": cugraph.__version__} }
    return returns<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.107950974230044, 'fraction_numerical': 0.01744186046511628, 'mean_word_length': 4.52997393570808, 'pattern_counts': {'":': 7, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '573925', 'license': 'Apache-2.0', 'n_tokens_mistral': 2094, 'n_tokens_neox': 2002, 'n_words': 661, 'path': 'notebooks/graph_algo_louvain.ipynb', 'repo_name': 'splunk/splunk-mltk-container-docker', 'size': '6364'}