<jupyter_start><jupyter_text>Create training dataset for anomaly detection modelIn this notebook We are going to create training dataset from node embeddings feature group and register to Hopsworks Feature Store. ![Training Dataset](./images/create_training_dataset.png)  Create a connection to hsfs<jupyter_code>import hsfs
from hops import hdfs
# Create a connection
connection = hsfs.connection()
# Get the feature store handle for the project's feature store
fs = connection.get_feature_store()<jupyter_output>Starting Spark application<jupyter_text>Retrieve alert nodes feature group from hsfs<jupyter_code>alert_nodes_fg = fs.get_feature_group("alert_nodes_fg", 1)
node_embeddings_fg = fs.get_feature_group("node_embeddings_fg", 1)<jupyter_output><empty_output><jupyter_text>Prepare training datasets for anomaly detection  In the next notebook we are going to train [gan for anomaly detection](https://arxiv.org/pdf/1905.11034.pdf). Durring training step  we will provide only features of accounts that have never been reported for money laundering behaviour.  But we will disclose previously reported accounts to the model only in evaluation step.<jupyter_code>non_sar_emb_query = node_embeddings_fg.select(["embedding"])\
                                      .join(alert_nodes_fg.select(["is_sar"])\
                                      .filter(alert_nodes_fg.is_sar == 0))
non_sar_emb_query.show(5)
non_sar_emb_query.read().count()
non_sar_td = fs.create_training_dataset(name="gan_non_sar_training_df",
                                       version=1,
                                       data_format="tfrecord",
                                       label=["is_sar"], 
                                       statistics_config=False, 
                                       splits={'train': 0.8, 'test': 0.2},
                                       coalesce=True,
                                       description="non sar dataset for gan training")
non_sar_td.save(non_sar_emb_query)<jupyter_output><empty_output><jupyter_text>For testing and evaluation we will include known SAR nodes to measure anomaly score<jupyter_code>non_sar_td = fs.get_training_dataset("gan_non_sar_training_df", 2)
non_sar_test_df = non_sar_td.read(split="test")
sar_emb_query = node_embeddings_fg.select(["embedding"])\
                                  .join(alert_nodes_fg.select(["is_sar"])\
                                  .filter(alert_nodes_fg.is_sar == 1))
sar_df = sar_emb_query.read()
sar_df = sar_df.select(*non_sar_test_df.columns)
eval_df = non_sar_test_df.union(sar_df)
eval_df.cache()
eval_df.show()
non_sar_test_df.count()
sar_df.count()
eval_df.count()
gan_eval_ds = fs.create_training_dataset(name="gan_eval_df",
                                       version=1,
                                       data_format="tfrecord",
                                       label=["is_sar"], 
                                       statistics_config=False, 
                                       coalesce = True,
                                       description="evaluation dataset for gan training")
gan_eval_ds.save(eval_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08183254344391785, 'fraction_numerical': 0.006635071090047393, 'mean_word_length': 2.43756786102063, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '181629', 'license': 'Apache-2.0', 'n_tokens_mistral': 939, 'n_tokens_neox': 860, 'n_words': 191, 'path': '6_create_anomaly_detection_td.ipynb', 'repo_name': 'logicalclocks/AMLend2end', 'size': '3165'}