<jupyter_start><jupyter_text>HoloScope: detecting collective anoamlies of constract suspiciousnessHoloScope is topology-and-spike aware fraud detection.HoloScope detect a subgraph of highly constrast suspicousness on topological, temporal, and categorical (e.g. rating score, topic, tag) infomation. Temporal spike of retweeting a message: AbstractAs online fraudsters invest more resources, including purchasing large pools of fake user accounts and dedicated IPs, fraudulent attacks become less obvious and their detection becomes increasingly challenging. Existing approaches such as average degree maximization suffer from the bias of including more nodes than necessary, resulting in lower accuracy and increased need for manual verification. Hence, we propose HoloScope, which uses information from graph topology and temporal spikes to more accurately detect groups of fraudulent users. In terms of graph topology, we introduce contrast suspiciousness, a dynamic weighting approach, which allows us to more accurately detect fraudulent blocks, particularly low-density blocks. In terms of temporal spikes, HoloScope takes into account the sudden bursts and drops of fraudsters' attacking patterns. In addition, we provide theoretical bounds for how much this increases the time cost needed for fraudsters to conduct adversarial attacks. Additionally, from the perspective of ratings, HoloScope incorporates the deviation of rating scores in order to catch fraudsters more accurately. Moreover, HoloScope has a concise framework and sub-quadratic time complexity, making the algorithm reproducible and scalable. Extensive experiments showed that HoloScope achieved significant accuracy improvements on synthetic and real data, compared with state-of-the-art fraud detection methods.<jupyter_code>import spartan as st<jupyter_output><empty_output><jupyter_text>You can configure the backend to use GPU or CPU only. \Default is using backend cpu.<jupyter_code># load graph data
tensor_data = st.loadTensor(path = "./inputData/yelp.tensor.gz", header=None)<jupyter_output><empty_output><jupyter_text>"tensor_data.data" has multiple-colum attributes, and a single-colum values (optional). The following table shows an example of 10000 four-tuple (user, object, date, score) and the 5th-colum is the frequency. |row id |    0	|   1	|         2    	|   3 	|   4  	||-----:	|-----:	|----:	|-----------:	|----:	|-----	||    0 	|    0 	|   0 	| 2012-08-01 	|   4 	|   1 	||    1 	|    1 	|   0 	| 2014-02-13 	|   5 	|   1 	||    2 	|    2 	|   0 	| 2015-10-31 	|   5 	|   1 	||    3 	|    3 	|   0 	| 2015-12-26 	|   3 	|   1 	||    4 	|    4 	|   0 	| 2016-04-08 	|   2 	|   1 	||  ... 	|  ... 	| ... 	|        ... 	| ... 	| ... 	|| 9995 	| 4523 	| 508 	| 2013-03-06 	|   5 	|   1 	|| 9996 	|  118 	| 508 	| 2013-03-07 	|   4 	|   1 	|| 9997 	| 5884 	| 508 	| 2013-03-07 	|   1 	|   1 	|| 9998 	| 2628 	| 508 	| 2013-04-08 	|   5 	|   1 	|| 9999 	| 5885 	| 508 	| 2013-06-17 	|   5 	|   1 	|<jupyter_code>stensor = tensor_data.toSTensor(hasvalue=True, mappers={2:st.TimeMapper(timeformat='%Y-%m-%d')})
#stensor._data<jupyter_output><empty_output><jupyter_text>Sparse tensor "stensor" is a multi-mode constructed from tensor_data. users, objects, date time, and score are all mapped into $[0, N]$ integers. \This example constructs a tensor of $5886 \times 509 \times 3857 \times 6$.<jupyter_code>graph = st.Graph(stensor, bipartite=True, weighted=True, modet=2)<jupyter_output><empty_output><jupyter_text>Get a Graph instance from a sparse tensor.  Run holoscope as a single model<jupyter_code>hs = st.HoloScope(graph)
print(hs)<jupyter_output><empty_output><jupyter_text>Default parameters are:{'alg': 'fastgreedy', 'eps': 1.6, 'numSing': 10, 'qfun': 'exp', 'b': 32, 'level': 0}You can change them be passing = as the doc shows.<jupyter_code>res = hs.run(level=0, k=1)<jupyter_output><empty_output><jupyter_text>Running level can be  0: topology only;  1: topology with time;   2: topology with category (e.g. rating score);  3: all three.  Use k for number of dense blocks you want to get.  res is a list of each block. Each block constains $((rows, nnzcols), susp\_score, levelcols, nnzcol\_scores)$  Run holoscope from anomaly detection task<jupyter_code># create a anomaly detection model
ad_model = st.AnomalyDetection.create(graph, st.ADPolicy.HoloScope, 'holoscope')
# run the model
#default k=2, eps=1.6
res = ad_model.run(k=2)<jupyter_output><empty_output><jupyter_text>The results is a list of top-k suspicious blocks.For each block, the resulting tuple contains $(user~nodes, object~nodes)$,  suspicious score, and suspicious scores of all object nodes.\Then we can visualize the subgraphs as follows.<jupyter_code>#viusal of graphs by networkx
import matplotlib.pyplot as plt
for r in res:
    rows, cols = r[0]
    # to subgraph
    sg = graph.get_sub_graph(rows, cols)
    # networkx plot
    fig = st.plot_graph(sg, bipartite=True, labels=[*rows, *cols])
    fig = st.plot_graph(sg, layout='circular', bipartite=True, labels=[*rows, *cols])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1026750590086546, 'fraction_numerical': 0.04189614476789929, 'mean_word_length': 3.9464980544747084, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '179164', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1789, 'n_tokens_neox': 1609, 'n_words': 566, 'path': 'live-tutorials/Holoscope.ipynb', 'repo_name': 'sunxiaobing1999/spartan2', 'size': '5084'}