<jupyter_start><jupyter_text><jupyter_code>from google.colab import files
uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>from google.colab import filesuploaded = files.upload()<jupyter_code>from google.colab import files
uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>Data sourceTaken from chapter 5 of Machine learning cookbook for cyber securityData dictionarynumber in brackets states the number of described features"…" - stands for multiple optional names that match the given pattern.…ip, …port(4): IP and port of client / srverpackets_…(3): Number of packets sent by client / server / bothack_…(3): Number of ACK packets sent by client / server / bothpacketsAB_ratio(1): Ratio between packets sent by client and sent by serverasn_…(2): Number of autonomous systems served as client, serverpush_…(3): Number of packets with PSH flag sent by client / server / bothbytes_…(3): Number of bytes sent by client / server / bothreset_…(3): Number of packets with RST flag sent by client / server / bothbytesAB_ratio(1): Ratio between number of bytes sent and number of bytes receivedsslcountcertificates(1): Number of SSL certificatescap_date(1): date of data capturing startsslcountclient_…(6): Client: Number of supported SSL cipher algorithms / ciphersuites / compressions / eliptic curves / key exchange algorithms / MAC algorithmscountry_…(2): Number of countries systems served as client / serversslcountserver_…(6): Server: Number of supported SSL cipher algorithms / ciphersuites / compressions / eliptic curves / key exchange algorithms / MAC algorithmsdaysTime(1): When during the day communication was establishedssldomserver_ciphersuite(1): Number of SSL versionsdns_alexaRank(1): DNS response server Alexa rankssldomserver_compression(5): Dominated SSL ciphersuite / eliptic curve / server name / server rank / versiondnscountaddresses(4): Number of adresses / answer / authoritative / additional fields in DNS responsesslhandshakeduration_…(10): SSL handshake duration: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropydnscountcanon_names(1): Number of canonical names in DNS responsesslratio…(7): Ratio between ssl sessions and: expired certificates / client cipher algorithms / ciphersuits / eliptic curves / client key exchange algorithms / client MAC algorithms / server namesdns_flag(1): DNS response flags combinationssslreqbytes_…(10): Number of request bytes: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropydnshostname(1): DNS host namesslrespbytes_…(10): Number of response bytes: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropydnsminttl(1): DNS response minimal time-to-livestart(1): session start (date-time)dnsprebad_requests(1): Number of preceding bad DNS responsestcpanalysis…(6): TCP: Number of packets with Keep Alive packets / lost segments / packets received out of order / retransmitted packets / reused ports / duplicake ACKsdns_time(1): Time took to receive DNS responsettlA…(10): TCP packet time-to-live sent by client: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropydsfield…(2): Differentiated Services (DS) field sent by client / serverttl_…(10): TCP packet time-to-live: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyduration(1): Session durationttlB…(10): TCP packet time-to-live sent by server: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyhttpbytes…(10): Number of bytes sent by client over HTTP: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropy.urg_…(3): Packets with URG flag sent by client / server / bothhttpcookiecount(1): Total number of cookie valuesweekDay(1): day of week (Sunday, Monday, …)httpcookievalues_…(10): Number of cookie values: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropydomain / subdomain / suffix(3): Dminated host's URL: domain / subdomain / suffixhttpcount…(6): HTTP: Number of hosts / unique content types used in request / unique response codes / unique response content types / transactions / unique user agentsisadhttp(1): subdomain of HTTP dominated host includes ad-related keywordshttpdom…(8): Dominated HTTP: browser / browser version / host / host Alexa rank / operating system / operating system version / request contetn type / response code / response contetn type /iscdnhttp(1): subdomain of HTTP dominated host includes CDN-related keywordshttpdomis_bot(1): Is most of HTTP connections created by known botiscloudhttp(1): subdomain of HTTP dominated host includes cloud-related keywordshttp_GET(1): Number of HTTP requests submited with GET methodis_…(3): session protocol is DNS / HTTP / SSLhttphas…(4): Does HTTP request have a location / referrer / content type / user agent fieldisghttp(1): subdomain of HTTP dominated host includes g-related keywordshttphasrespcontenttype(1): Does HTTP response have a content type fieldisimghttp(1): subdomain of HTTP dominated host includes image-related keywordshttpinterarrivel_…(10): HTTP request-response inter arrival time: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyismhttp(1): subdomain of HTTP dominated host includes mobile-related keywordshttp_POST(1): Number of HTTP requests submited with POST methodismakersite_http(1): subdomain of HTTP dominated host is of a maker's sitehttpreqbytes_…(10): HTTP request bytes: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyismediahttp(1): subdomain of HTTP dominated host includes media-related keywordshttprespbytes_…(10): HTTP response bytes: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyisnumericurl_http(1): subdomain of HTTP dominated host is numerichttptime…(10): Time took to HTTP server to return response: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyisnumericurlwithport_http(1): subdomain of HTTP dominated host is numeric plus port namelabel(1): malware labelistvhttp(1): HTTP dominated host has TV-related keywordslabelSS(1): malware labelBissystem_port(1): destination port is in the range of [1, 1023]packetinterarrivelA…(10): Client packets inter arival time: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyBisuser_port(1): destination port is in the range of [1024, 49151]packetinterarrivel_…(10): Packets inter arival time: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyBisdynamicandorprivateport(1): destination port is in the range of [49152, 65535]packetinterarrivelB…(10): Server packets inter arival time: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropyBportis_…(10): Destination port is one of recent top 10 most frequent: 80, 23, etc.packetsizeA_…(10): Client packets size: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropysubdomainis…(10): subdomain of HTTP dominated host is one of recent top 10 most frequentpacketsize…(10): Packets size: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropydomainis…(10): domain of HTTP dominated host is one of recent top 10 most frequentpacketsizeB_…(10): Server packets size: Minimum value, quartile 1, average, median (quartile 2), sum, quartile 3, maximum value, standard deviation, variance, entropysuffixis…(4): suffix of HTTP dominated host is one of recent top 4most frequent: com, net, etc.---<jupyter_code>import math
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential 
from keras.models import Sequential
import pandas as pd
import os
training_data = pd.read_csv("iot_device_test.csv")
testing_data = pd.read_csv("iot_device_train.csv")
training_data.head()
training_data["device_category"].unique()
for i in testing_data["device_category"].unique():
    if i not in training_data["device_category"].unique():
        print(i)
for i in training_data["device_category"].unique():
    if i not in testing_data["device_category"].unique():
        print(i)
len(training_data["device_category"].unique())
#length of training data device cateogry<jupyter_output><empty_output><jupyter_text>we will removing the water sensor dataset as its mnot present in the both cateogries<jupyter_code>len(testing_data["device_category"].unique())
#length of testing data device cateogry
index_names = testing_data[testing_data['device_category'] == 'water_sensor'].index
len(index_names)
testing_data.drop(index_names, inplace=True)
testing_data["device_category"].unique()
#we are dividing the x train and the y train data using the loc function
x_train, y_train = (
    training_data.loc[:, training_data.columns != "device_category"].values,
    training_data["device_category"],
)
x_test, y_test = (
    testing_data.loc[:, testing_data.columns != "device_category"].values,
    testing_data["device_category"],
)
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le.fit(training_data["device_category"].unique())
y_train_encoded = le.transform(y_train)
y_test_encoded = le.transform(y_test)
from xgboost import XGBClassifier

model = XGBClassifier(n_estimators=1000)
model.fit(x_train, y_train_encoded)
from sklearn.preprocessing import StandardScaler

predictions = model.predict(x_test)

rmse = np.sqrt(np.mean(predictions - y_test_encoded)**2)
rmse
model.score(x_test, y_test_encoded)
from sklearn.neighbors import KNeighborsClassifier
classifier= KNeighborsClassifier(n_neighbors=50, metric='minkowski', p=2 )  
classifier.fit(x_train, y_train_encoded) 
classifier.score(x_test,y_test_encoded)
from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(x_train, y_train_encoded)
clf.score(x_test,y_test_encoded)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08508845829823083, 'fraction_numerical': 0.01862772629411214, 'mean_word_length': 6.497543859649123, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '304624', 'license': 'MIT', 'n_tokens_mistral': 3122, 'n_tokens_neox': 2884, 'n_words': 1146, 'path': 'Ntccmoel.ipynb', 'repo_name': 'adityasaxena-crypto/Ntcc-model-aditya', 'size': '10683'}