<commit_before># Malware Detection using Serverless Spark through Airflow.

**Goal** - Data Preparation and Model Training for Detecting Malware in Network Dataset.

Following are the lob modules:

[1. Understanding Data](instructions/06c_malware_detection_airflow_execution.md#1-understanding-data)<br>
[2. Solution Diagram](instructions/06c_malware_detection_airflow_execution.md#2-solution-diagram)<br>
[3. Uploading DAG files to DAGs folder](instructions/06c_malware_detection_airflow_execution.md#3-uploading-dag-files-to-dags-folder)<br>
[4. Execution of Airflow DAG](instructions/06c_malware_detection_airflow_execution.md#4-execution-of-airflow-dag)<br>
[5. BQ output tables](instructions/06c_malware_detection_airflow_execution.md#5-bq-output-tables)<br>
[6. Logging](instructions/06c_malware_detection_airflow_execution.md#6-logging)<br>

<br>

## 1. Understanding Data

The dataset used for this project is [networks traffic data](01-datasets/network_traffic_dataset.csv). <br>

The dataset contains the following features:

- duration - length (number of seconds) of the connection
- protocol_type - type of the protocol, e.g. tcp, udp, etc.
- service - network service on the destination, e.g., http, telnet, etc.
- flag - normal or error status of the connection
- src_bytes - number of data bytes from source to destination
- dst_bytes - number of data bytes from destination to source
- land - 1 if connection is from/to the same host/port; 0 otherwise
- wrong_fragment - number of wrong fragments
- urgent - number of urgent packets
- hot - number of 'hot' indicators
- num_failed_logins - number of failed login attempts
- logged_in - 1 if successfully logged in; 0 otherwise
- num_compromised - number of compromised conditions
- root_shell - 1 if root shell is obtained; 0 otherwise
- su_attempted - 1 if su root command attempted; 0 otherwise
- num_root - number of root accesses
- num_file_creations - number of file creation operations
- num_shells - number of shell prompts
- num_access_files - number of operations on access control files
- num_outbound_cm ds - number of outbound commands in an ftp session
- is_hot_login - 1 if the login belongs to the hot list; 0 otherwise<commit_msg>fix dataset link<commit_after># Malware Detection using Serverless Spark through Airflow.

**Goal** - Data Preparation and Model Training for Detecting Malware in Network Dataset.

Following are the lab modules:

[1. Understanding Data](instructions/06c_malware_detection_airflow_execution.md#1-understanding-data)<br>
[2. Solution Diagram](instructions/06c_malware_detection_airflow_execution.md#2-solution-diagram)<br>
[3. Uploading DAG files to DAGs folder](instructions/06c_malware_detection_airflow_execution.md#3-uploading-dag-files-to-dags-folder)<br>
[4. Execution of Airflow DAG](instructions/06c_malware_detection_airflow_execution.md#4-execution-of-airflow-dag)<br>
[5. BQ output tables](instructions/06c_malware_detection_airflow_execution.md#5-bq-output-tables)<br>
[6. Logging](instructions/06c_malware_detection_airflow_execution.md#6-logging)<br>

<br>

## 1. Understanding Data

The dataset used for this project is [networks traffic data](../01-datasets/network_traffic_dataset.csv). <br>

The dataset contains the following features:

- duration - length (number of seconds) of the connection
- protocol_type - type of the protocol, e.g. tcp, udp, etc.
- service - network service on the destination, e.g., http, telnet, etc.
- flag - normal or error status of the connection
- src_bytes - number of data bytes from source to destination
- dst_bytes - number of data bytes from destination to source
- land - 1 if connection is from/to the same host/port; 0 otherwise
- wrong_fragment - number of wrong fragments
- urgent - number of urgent packets
- hot - number of 'hot' indicators
- num_failed_logins - number of failed login attempts
- logged_in - 1 if successfully logged in; 0 otherwise
- num_compromised - number of compromised conditions
- root_shell - 1 if root shell is obtained; 0 otherwise
- su_attempted - 1 if su root command attempted; 0 otherwise
- num_root - number of root accesses
- num_file_creations - number of file creation operations
- num_shells - number of shell prompts
- num_access_files - number of operations on access control files
- num_outbound_cm ds - number of outbound commands in an ftp session
- is_hot_login - 1 if the login belongs to the hot list; 0 otherwise
{'fraction_non_alphanumeric': 0.0782942238267148, 'fraction_numerical': 0.016696750902527077, 'mean_word_length': 4.832894736842105, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1465604', 'n_tokens_mistral': 1380, 'n_tokens_neox': 1287, 'n_words': 465}