<jupyter_start><jupyter_text>Increase max cell height for network graph visualization<jupyter_code>%%html
<style>
.container { width:70% !important; }
.output_wrapper, .output {
    height:auto !important;
    max-height:1000px;
}
.output_scroll {
    box-shadow:none !important;
    webkit-box-shadow:none !important;
}
</style><jupyter_output><empty_output><jupyter_text>Toy Examples| Sample | File Size (kB) |   of export symbols |  of import symbols | of import DLL ||----|-----|----|---- |----- ||$x_1$|100|3|10 |2 ||$x_2$|10234|4|4|2 ||$x_3$|3453|6|2 |1 |<jupyter_code>x_1 = np.array([100, 3, 10, 2])
x_2 = np.array([10234, 4, 4, 2])
x_3 = np.array([3453, 6, 2, 1])
x_1
x_1.shape
pairwise.euclidean_distances([x_1, x_2, x_3])
pairwise.manhattan_distances([x_1, x_2, x_3])
pairwise.cosine_distances([x_1, x_2, x_3])
pairwise.cosine_similarity([x_1, x_2, x_3])
1 - distance.jaccard(x_1, x_2)
distance.jaccard(x_1, x_3)
distance.jaccard(x_2, x_3)<jupyter_output><empty_output><jupyter_text>Command Line Usage Install rogers, https://github.com/cylance/rogers/blob/develop/README.mdinstallation<jupyter_code>! rogers -h<jupyter_output><empty_output><jupyter_text>IndexInteract with NN indexes:+ Extract raw feature data+ Transform data into vectors with pipeline+ Fit a specific NN index+ Query NN index<jupyter_code>! rogers index -h<jupyter_output><empty_output><jupyter_text>Extract samples in a directory into the database<jupyter_code>! rogers index --dir "../samples/00/FD" extract -f<jupyter_output><empty_output><jupyter_text>Transform samples using vectorizer pipeline<jupyter_code>! rogers index transform<jupyter_output><empty_output><jupyter_text>Fit different indexes<jupyter_code>! rogers index fit pruning_tree
! rogers index fit lsh_forest
! rogers index fit hnsw
! rogers index fit ctph<jupyter_output><empty_output><jupyter_text>Query fit index<jupyter_code>! rogers --print index query lsh_forest "<KEY>"
! rogers --print index query ctph "<KEY>"
! rogers --print index query pruning_tree "<KEY>" --k 10<jupyter_output><empty_output><jupyter_text>Datasets  Variant 2015`variant2015` is a small dataset currated specifically for comparing malware similarity tools.+ `group 1`: Ziyang RAT and were originally packed with Armadillo v1.71+ `group 2`: LinseningSvr and were originally packed with Armadillo or Aspack+ `group 3`: BeepService and were originally packed with Armadillo v1.71+ `group 4`: SimpleFileMover and were originally packed with Armadillo v1.71+ `group 5`: DD Keylogger+ `group 6`: PUP by McAfee and were originally packed with Armadillo v1.71 or InstallShield 2000+ `group 7`: Backdoors by McAfee and were originally packed with Armadillo v1.xx - v2.xx+ `group 8`: SvcInstaller and were originally packed with Armadillo v1.71<jupyter_code>variant_df = pd.read_csv("../datasets/variant_2015.csv")
print("Number of samples %s" % len(variant_df))
variant_group_1 = variant_df[variant_df['variant_group'] == 1]['sha256'].tolist()[:5]
variant_df.groupby(['variant_group']).size()<jupyter_output><empty_output><jupyter_text>VT Cluster Jan 2018`vtcluster_jan2018` is a moderately sized dataset pulled from the VT clusters endpoint in January 2018.<jupyter_code>vt_df = pd.read_csv("../datasets/vt_clusters_jan2018.csv")
print("Number of samples %s" % len(vt_df))
vt_df.groupby(['vt_cluster']).size()
sample_vt_df = vt_df.sample(n=100)
db = r.store.Database()

variant_seed_samples = db.load_samples(variant_df['sha256'])
variant_group_1_samples = db.load_samples(variant_group_1)
vt_seed_samples = db.load_samples(sample_vt_df['sha256'])

x = variant_seed_samples[0]
x1 = vt_seed_samples[0]
x2 = vt_seed_samples[1]<jupyter_output><empty_output><jupyter_text>Features<jupyter_code>x1.features
x2.contextual_features()
hnsw_idx = r.index.init('hnsw')
hnsw_idx.load()

lsh_idx = r.index.init('lsh_forest')
lsh_idx.load()

pruning_idx = r.index.init('pruning_tree')
pruning_idx.load()

ssdeep_idx = r.index.init('ctph')
ssdeep_idx.load()

brute_idx = r.index.init('bruteforce')
brute_idx.load()<jupyter_output><empty_output><jupyter_text>Vector PlotsPlot vectors for `vtcluster_jan2018` to understand cosine similarity.<jupyter_code>from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from scipy import stats


xs, _ = hnsw_idx.transform(variant_seed_samples)

pca = PCA(n_components=2)

xs_t = pca.fit_transform(xs)
X, Y = zip(*np.zeros((xs_t.shape[0], 2)))
U, V = zip(*xs_t)

plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([-1, 1.1])
ax.set_ylim([-1, 1])
plt.draw()
plt.show()<jupyter_output><empty_output><jupyter_text>Index Usage<jupyter_code>neighbors = lsh_idx.query_samples([x], k=5, include_neighbors=False)
print("%s is in variant group %s" % (x.sha256, x.contextual_features()['variant_group']))
r.visualize.plt_neighbor_graph(neighbors)
neighbors = hnsw_idx.query_samples([x], k=5, include_neighbors=False)
print("%s is in variant group %s" % (x.sha256, x.contextual_features()['variant_group']))
r.visualize.plt_neighbor_graph(neighbors)
neighbors = lsh_idx.query_samples(variant_seed_samples, k=5, include_neighbors=False)
r.visualize.plt_neighbor_graph(neighbors)
neighbors = pruning_idx.query_samples(variant_seed_samples, k=5, include_neighbors=True)
r.visualize.plt_neighbor_graph(neighbors)
neighbors = hnsw_idx.query_samples(variant_seed_samples, k=5, include_neighbors=True)
r.visualize.plt_neighbor_graph(neighbors)
neighbors = ssdeep_idx.query_samples(variant_seed_samples, k=5, include_neighbors=True)
r.visualize.plt_neighbor_graph(neighbors, normalize=False)
neighbors = ssdeep_idx.query_samples(vt_seed_samples, k=5, include_neighbors=True)
r.visualize.plt_neighbor_graph(neighbors, normalize=False)<jupyter_output><empty_output><jupyter_text>Experiment Precision@k is the proportion of retrieved documents that are relevant to the query over k.$Precision@k = \frac{\textrm{relevant} \ \cap \ \textrm{retrieved}}{k}$<jupyter_code>def precision_at_k_method(method):
    for k in (5, 10, 50, 100):
        results = method.query_samples(list(vt_seed_samples), k=k)
        precision_per_query = []
        for ret in results:
            q = ret['query']
            neigbors = ret['neighbors']

            label = str(q.contextual_features()['vt_cluster'])
            label_key = 'vt_cluster'

            relevant = 0

            for nbr in neigbors:
                nbr, _ = nbr
                context = nbr.contextual_features()
                if label_key in context and str(context[label_key]) == label:
                    relevant += 1
            precision_per_query.append(relevant / float(k))
        print(method.name, k, np.mean(precision_per_query))
for method in (pruning_idx, hnsw_idx, lsh_idx, ssdeep_idx, brute_idx):
    precision_at_k_method(method)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12716085555230003, 'fraction_numerical': 0.029006738939349547, 'mean_word_length': 5.938008130081301, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 62, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '201288', 'license': 'Apache-2.0', 'n_tokens_mistral': 2675, 'n_tokens_neox': 2472, 'n_words': 508, 'path': 'docs/notebooks/2018-acod/ACoD 2018.ipynb', 'repo_name': 'cylance/rogers', 'size': '7003'}