<jupyter_start><jupyter_text>Creation of features for chain of events for attacks<jupyter_code>df = pd.read_csv('../data/processed/merged_dataset_pivoted.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Lateral Movement (Link: https://www.rapid7.com/resources/using-windows-event-logs-to-detect-lateral-movement/):Using window event logs to detect lateral movement:Authentication Events (all):1. Event_528 <-- Successful Login2. Event_529 <-- Unsucessful Login3. Event_4624 and Event_4625 <-- Two methods of Lateral Movement (Windows NT5 and Nt6 Operating Systems)- SMB: 552, 4648- Scheduled Tasks: 602, 4698- PS Exec: 601, 4697 <-- System Admin Tool to execute code remote- SSH: app logs <-- Less common in windows environment<jupyter_code>df[["event_4624", "event_4625", "event_4648", "event_4698"]].head()
df["total_authN_events"] = df["event_4624"] + df["event_4625"] # ".Logon Type:[\W](3|10).*"
df.head()<jupyter_output><empty_output><jupyter_text>- PS Exec not there- No columns for "event_552", "event_528", "event_529", "event_601", "event_602", and "event_4697"- Have some of the cases for SMB, Scheduled Tasks <-- Look into regenerating dataset to get features- Have some of the cases for authentication events <-- Same as above  Ransomware, malware and cobalt strike (Link: https://www.beyondtrust.com/blog/entry/windows-server-events-monitor):Ransomware:- event_8or- event_22 <-- Not Contained in the dataframeHacker Presence:- event_104 <-- Event Log was Cleared- event_1102 <-- Audit Log was Cleared- event_4719 <-- System Audit Policy Changed<jupyter_code>columns = list(df.columns)
features = columns[4:]
print(features)<jupyter_output>['event_1', 'event_3', 'event_8', 'event_10', 'event_11', 'event_12', 'event_4624', 'event_4625', 'event_4648', 'event_4658', 'event_4661', 'event_4663', 'event_4672', 'event_4698', 'event_4768', 'event_5140', 'event_5145', 'event_5156', 'event_5158', 'total_authN_events']<jupyter_text>APT- event_4674 <-- Account Name, Service, Process, Object- event_4688 <-- Account Name, ProcessType Ratio:- event_4624 <-- Logon- event_4627 <-- Group Membership- event_4658 <-- Handle to an object- event_4768 <-- Kerberos AuthN- event_4769 <-- Kerberos AuthN - Services- event_4672 <-- Assignment of Admin Rights- event_4776 <-- Kerberos Service TicketNo event ids found.  Detect Pass the Hash Attacks (Link: https://stealthbits.com/blog/how-to-detect-pass-the-hash-attacks/)Workstation Logs (source host):- event_4648- event_4624- event_4672- sysmon event 10Target Server Logs (target host):- event_4624- event_4672Domain Controller:- event 4768- event 4769- event 4776<jupyter_code>df[["event_4624", "event_4672", "event_4648", "event_10"]].head()
df["hash_attack"] = df["event_4648"] + df["event_4624"] + df["event_4672"] + df["event_10"]
df.head()<jupyter_output><empty_output><jupyter_text>Common Incident Response Scenario - Phishing (Link: https://www.netscylla.com/blog/2020/02/01/Threat-hunting-with-Windows-Event-Logs.html):- event_1 <-- Process Creation- event_11 <-- FileCreate- event_15 <-- FileCreateStreamHash<jupyter_code>df["phishing"] = df["event_1"] + df["event_11"]
pd.unique(df["phishing"].values.ravel())
df.head()
columns = list(df.columns)
features = columns[4:]
df[features] = df[features].div(df.total_events, axis = 0)
# Separating out the features
X = df.loc[:, features].values
y = df.loc[:, ['is_malicious']].values
print(X)
print(y)
df.to_csv("../data/processed/final_df.csv", index = False)
X = StandardScaler().fit_transform(X)
pca = PCA(n_components=2)
pcs = pca.fit_transform(X)
pca_df = pd.DataFrame(data = pcs, columns = ['pc1', 'pc2'])
fig = plt.figure(figsize = (8,8))
ax = fig.add_subplot(1,1,1) 
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('Features in 2D', fontsize = 20)

targets = [0, 1]
colors = ['r', 'b']
for target, color in zip(targets,colors):
    indicesToKeep = df['is_malicious'] == target
    ax.scatter(pca_df.loc[indicesToKeep, 'pc1']
               , pca_df.loc[indicesToKeep, 'pc2']
               , c = color
               , s = 50)
    
# ax.legend(targets)
# ax.scatter(pca_df['pc1']
#                , pca_df['pc2']
#                , s = 50)
ax.grid()
smote = SMOTE(random_state=0, sampling_strategy="minority")
X_os, y_os = smote.fit_resample(X, y)
X_train_os, X_test_os, y_train_os, y_test_os = train_test_split(X_os, y_os, test_size = 0.2, random_state=2)
# Decision Tree Classifier

# Create Decision Tree classifer object
clf_os = DecisionTreeClassifier()

# Train Decision Tree Classifer
clf_os.fit(X_train_os, y_train_os)

# Predict the response for the test dataset
y_pred_os = clf_os.predict(X_test_os)

# Check Prediction
print(classification_report(y_test_os, y_pred_os))
# print(confusion_matrix(y_test_os, y_pred_os))
print("Accuracy:", accuracy_score(y_test_os, y_pred_os))
print("Label 0:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 0))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 0))
print("Label 1:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 1))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 1))
classes = np.unique(y_pred_os)
fig,ax = plt.subplots()
cm = metrics.confusion_matrix(y_test_os,y_pred_os,labels=classes)
sns.heatmap(cm, annot=True,fmt='d',cmap=plt.cm.Blues,cbar=False)
ax.set(xlabel="Pred",ylabel="True",title="Confusion Matrix")
ax.set_yticklabels(labels=classes,rotation=0)
plt.show()
# K Neighbors Classifier

# Create K Neighbors Classifier Object
neigh = KNeighborsClassifier(n_neighbors=3)

# Train K Neighbors Classifier Object
neigh.fit(X_train_os,y_train_os.ravel())

# Predict the response for test dataset
y_pred_os = neigh.predict(X_test_os)

# Check Prediction
print(classification_report(y_test_os, y_pred_os))
# print(confusion_matrix(y_test_os, y_pred_os))
print("Accuracy:", accuracy_score(y_test_os, y_pred_os))
print("Label 0:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 0))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 0))
print("Label 1:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 1))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 1))
classes = np.unique(y_pred_os)
fig,ax = plt.subplots()
cm = metrics.confusion_matrix(y_test_os,y_pred_os,labels=classes)
sns.heatmap(cm, annot=True,fmt='d',cmap=plt.cm.Blues,cbar=False)
ax.set(xlabel="Pred",ylabel="True",title="Confusion Matrix")
ax.set_yticklabels(labels=classes,rotation=0)
plt.show()
# Logistic Regression

# instantiate model
model = LogisticRegression()

# fit 
model.fit(X_train_os,y_train_os)

# predict
y_pred_os = model.predict(X_test_os)

# Check Prediction
print(classification_report(y_test_os, y_pred_os))
# print(confusion_matrix(y_test_os, y_pred_os))
print("Accuracy:", accuracy_score(y_test_os, y_pred_os))
print("Label 0:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 0))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 0))
print("Label 1:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 1))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 1))
classes = np.unique(y_pred_os)
fig,ax = plt.subplots()
cm = metrics.confusion_matrix(y_test_os,y_pred_os,labels=classes)
sns.heatmap(cm, annot=True,fmt='d',cmap=plt.cm.Blues,cbar=False)
ax.set(xlabel="Pred",ylabel="True",title="Confusion Matrix")
ax.set_yticklabels(labels=classes,rotation=0)
plt.show()
# Random Forest:

# Instantiate Model
random_model = RandomForestClassifier()

# Fit
random_model.fit(X_train_os, y_train_os)

# Predict
y_pred_os = random_model.predict(X_test_os)

# Check Prediction
print(classification_report(y_test_os, y_pred_os))
# print(confusion_matrix(y_test_os, y_pred_os))
print("Accuracy:", accuracy_score(y_test_os, y_pred_os))
print("Label 0:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 0))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 0))
print("Label 1:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 1))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 1))
classes = np.unique(y_pred_os)
fig,ax = plt.subplots()
cm = metrics.confusion_matrix(y_test_os,y_pred_os,labels=classes)
sns.heatmap(cm, annot=True,fmt='d',cmap=plt.cm.Blues,cbar=False)
ax.set(xlabel="Pred",ylabel="True",title="Confusion Matrix")
ax.set_yticklabels(labels=classes,rotation=0)
plt.show()
# Gradient Boosting Classifier

# Instantiate Model
gb = GradientBoostingClassifier()

# Fit
gb.fit(X_train_os, y_train_os)

# Predict
y_pred_os = gb.predict(X_test_os)

# Check Prediction
print(classification_report(y_test_os, y_pred_os))
# print(confusion_matrix(y_test_os, y_pred_os))
print("Accuracy:", accuracy_score(y_test_os, y_pred_os))
print("Label 0:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 0))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 0))
print("Label 1:")
print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 1))
print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 1))
classes = np.unique(y_pred_os)
fig,ax = plt.subplots()
cm = metrics.confusion_matrix(y_test_os,y_pred_os,labels=classes)
sns.heatmap(cm, annot=True,fmt='d',cmap=plt.cm.Blues,cbar=False)
ax.set(xlabel="Pred",ylabel="True",title="Confusion Matrix")
ax.set_yticklabels(labels=classes,rotation=0)
plt.show()
# # sklearn.svm.SVC (Support Vector Classification)

# svc = SVC(gamma="auto")

# svc.fit(X_train_os, y_train_os)

# # Predict
# y_pred_os = svc.predict(X_test_os)

# # Check Prediction
# print(classification_report(y_test_os, y_pred_os))
# # print(confusion_matrix(y_test_os, y_pred_os))
# print("Accuracy:", accuracy_score(y_test_os, y_pred_os))
# print("Label 0:")
# print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 0))
# print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 0))
# print("Label 1:")
# print("Precision:", precision_score(y_test_os, y_pred_os, pos_label = 1))
# print("Recall:", recall_score(y_test_os, y_pred_os, pos_label = 1))
# classes = np.unique(y_pred_os)
# fig,ax = plt.subplots()
# cm = metrics.confusion_matrix(y_test_os,y_pred_os,labels=classes)
# sns.heatmap(cm, annot=True,fmt='d',cmap=plt.cm.Blues,cbar=False)
# ax.set(xlabel="Pred",ylabel="True",title="Confusion Matrix")
# ax.set_yticklabels(labels=classes,rotation=0)
# plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15033301617507136, 'fraction_numerical': 0.033587059942911515, 'mean_word_length': 7.135448916408668, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 20, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '212465', 'license': 'FTL', 'n_tokens_mistral': 4281, 'n_tokens_neox': 3953, 'n_words': 732, 'path': 'notebooks/6.0-ss-test-models.ipynb', 'repo_name': 'vekshan/anomaly_detection_active_directory_logs', 'size': '10510'}