<jupyter_start><jupyter_text>The following definitions have been taken from the Kaggle Competition Description:https://www.kaggle.com/c/microsoft-malware-prediction/dataUnavailable or self-documenting column names are marked with an "NA".- MachineIdentifier - Individual machine ID- ProductName - Defender state information e.g. win8defender- EngineVersion - Defender state information e.g. 1.1.12603.0- AppVersion - Defender state information e.g. 4.9.10586.0- AvSigVersion - Defender state information e.g. 1.217.1014.0- IsBeta - Defender state information e.g. false- RtpStateBitfield - NA- IsSxsPassiveMode - NA- DefaultBrowsersIdentifier - ID for the machine's default browser- AVProductStatesIdentifier - ID for the specific configuration of a user's antivirus software- AVProductsInstalled - NA- AVProductsEnabled - NA- HasTpm - True if machine has tpm- CountryIdentifier - ID for the country the machine is located in- CityIdentifier - ID for the city the machine is located in- OrganizationIdentifier - ID for the organization the machine belongs in, organization ID is mapped to both specific companies and broad industries- GeoNameIdentifier - ID for the geographic region a machine is located in- LocaleEnglishNameIdentifier - English name of Locale ID of the current user- Platform - Calculates platform name (of OS related properties and processor property)- Processor - This is the process architecture of the installed operating system- OsVer - Version of the current operating system- OsBuild - Build of the current operating system- OsSuite - Product suite mask for the current operating system.- OsPlatformSubRelease - Returns the OS Platform sub-release (Windows Vista, Windows 7, Windows 8, TH1, TH2)- OsBuildLab - Build lab that generated the current OS. Example: 9600.17630.amd64fre.winblue_r7.150109-2022- SkuEdition - The goal of this feature is to use the Product Type defined in the MSDN to map to a 'SKU-Edition' name that is useful in population reporting. The valid Product Type are defined in %sdxroot%\data\windowseditions.xml. This API has been used since Vista and Server 2008, so there are many Product Types that do not apply to Windows 10. The 'SKU-Edition' is a string value that is in one of three classes of results. The design must hand each class.- IsProtected - This is a calculated field derived from the Spynet Report's AV Products field. Returns: a. TRUE if there is at least one active and up-to-date antivirus product running on this machine. b. FALSE if there is no active AV product on this machine, or if the AV is active, but is not receiving the latest updates. c. null if there are no Anti Virus Products in the report. Returns: Whether a machine is protected.- AutoSampleOptIn - This is the SubmitSamplesConsent value passed in from the service, available on CAMP 9+- PuaMode - Pua Enabled mode from the service- SMode - This field is set to true when the device is known to be in 'S Mode', as in, Windows 10 S mode, where only Microsoft Store apps can be installed- IeVerIdentifier - NA- SmartScreen - This is the SmartScreen enabled string value from registry. This is obtained by checking in order, HKLM\SOFTWARE\Policies\Microsoft\Windows\System\SmartScreenEnabled and HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\SmartScreenEnabled. If the value exists but is blank, the value "ExistsNotSet" is sent in telemetry.- Firewall - This attribute is true (1) for Windows 8.1 and above if windows firewall is enabled, as reported by the service.- UacLuaenable - This attribute reports whether or not the "administrator in Admin Approval Mode" user type is disabled or enabled in UAC. The value reported is obtained by reading the regkey HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\EnableLUA.- Census_MDC2FormFactor - A grouping based on a combination of Device Census level hardware characteristics. The logic used to define Form Factor is rooted in business and industry standards and aligns with how people think about their device. (Examples: Smartphone, Small Tablet, All in One, Convertible...)- Census_DeviceFamily - AKA DeviceClass. Indicates the type of device that an edition of the OS is intended for. Example values: Windows.Desktop, Windows.Mobile, and iOS.Phone- Census_OEMNameIdentifier - NA- Census_OEMModelIdentifier - NA- Census_ProcessorCoreCount - Number of logical cores in the processor- Census_ProcessorManufacturerIdentifier - NA- Census_ProcessorModelIdentifier - NA- Census_ProcessorClass - A classification of processors into high/medium/low. Initially used for Pricing Level SKU. No longer maintained and updated- Census_PrimaryDiskTotalCapacity - Amount of disk space on primary disk of the machine in MB- Census_PrimaryDiskTypeName - Friendly name of Primary Disk Type - HDD or SSD- Census_SystemVolumeTotalCapacity - The size of the partition that the System volume is installed on in MB- Census_HasOpticalDiskDrive - True indicates that the machine has an optical disk drive (CD/DVD)- Census_TotalPhysicalRAM - Retrieves the physical RAM in MB- Census_ChassisTypeName - Retrieves a numeric representation of what type of chassis the machine has. A value of 0 means xx- Census_InternalPrimaryDiagonalDisplaySizeInInches - Retrieves the physical diagonal length in inches of the primary display- Census_InternalPrimaryDisplayResolutionHorizontal - Retrieves the number of pixels in the horizontal direction of the internal display.- Census_InternalPrimaryDisplayResolutionVertical - Retrieves the number of pixels in the vertical direction of the internal display- Census_PowerPlatformRoleName - Indicates the OEM preferred power management profile. This value helps identify the basic form factor of the device- Census_InternalBatteryType - NA- Census_InternalBatteryNumberOfCharges - NA- Census_OSVersion - Numeric OS version Example - 10.0.10130.0- Census_OSArchitecture - Architecture on which the OS is based. Derived from OSVersionFull. Example - amd64- Census_OSBranch - Branch of the OS extracted from the OsVersionFull. Example - OsBranch = fbl_partner_eeap where OsVersion = 6.4.9813.0.amd64fre.fbl_partner_eeap.140810-0005- Census_OSBuildNumber - OS Build number extracted from the OsVersionFull. Example - OsBuildNumber = 10512 or 10240- Census_OSBuildRevision - OS Build revision extracted from the OsVersionFull. Example - OsBuildRevision = 1000 or 16458- Census_OSEdition - Edition of the current OS. Sourced from HKLM\Software\Microsoft\Windows NT\CurrentVersion@EditionID in registry. Example: Enterprise- Census_OSSkuName - OS edition friendly name (currently Windows only)- Census_OSInstallTypeName - Friendly description of what install was used on the machine i.e. clean- Census_OSInstallLanguageIdentifier - NA- Census_OSUILocaleIdentifier - NA- Census_OSWUAutoUpdateOptionsName - Friendly name of the WindowsUpdate auto-update settings on the machine.- Census_IsPortableOperatingSystem - Indicates whether OS is booted up and running via Windows-To-Go on a USB stick.- Census_GenuineStateName - Friendly name of OSGenuineStateID. 0 = Genuine- Census_ActivationChannel - Retail license key or Volume license key for a machine.- Census_IsFlightingInternal - NA- Census_IsFlightsDisabled - Indicates if the machine is participating in flighting.- Census_FlightRing - The ring that the device user would like to receive flights for. This might be different from the ring of the OS which is currently installed if the user changes the ring after getting a flight from a different ring.- Census_ThresholdOptIn - NA- Census_FirmwareManufacturerIdentifier - NA- Census_FirmwareVersionIdentifier - NA- Census_IsSecureBootEnabled - Indicates if Secure Boot mode is enabled.- Census_IsWIMBootEnabled - NA- Census_IsVirtualDevice - Identifies a Virtual Machine (machine learning model)- Census_IsTouchEnabled - Is this a touch device ?- Census_IsPenCapable - Is the device capable of pen input ?- Census_IsAlwaysOnAlwaysConnectedCapable - Retreives information about whether the battery enables the device to be AlwaysOnAlwaysConnected .- Wdft_IsGamer - Indicates whether the device is a gamer device or not based on its hardware combination.- Wdft_RegionIdentifier - NA<jupyter_code># Notebooks which have been considered in the following analyisis are as follows:
# [1] https://www.kaggle.com/dimitreoliveira/malware-detection-eda-and-lgbm
# [2] https://www.kaggle.com/dimitreoliveira/malware-detection-extended-eda
# [3] https://www.kaggle.com/jiegeng94/everyone-do-this-at-the-beginning
# [4] 
# In order to optimize memory usage it is useful to make certain features get certain types
# the dictionary below has been taken from https://www.kaggle.com/theoviel/load-the-totality-of-the-data
dtypes = {
        'MachineIdentifier':                                    'category',
        'ProductName':                                          'category',
        'EngineVersion':                                        'category',
        'AppVersion':                                           'category',
        'AvSigVersion':                                         'category',
        'IsBeta':                                               'int8',
        'RtpStateBitfield':                                     'float16',
        'IsSxsPassiveMode':                                     'int8',
        'DefaultBrowsersIdentifier':                            'float32',
        'AVProductStatesIdentifier':                            'float32',
        'AVProductsInstalled':                                  'float16',
        'AVProductsEnabled':                                    'float16',
        'HasTpm':                                               'int8',
        'CountryIdentifier':                                    'int16',
        'CityIdentifier':                                       'float32',
        'OrganizationIdentifier':                               'float16',
        'GeoNameIdentifier':                                    'float16',
        'LocaleEnglishNameIdentifier':                          'int16',
        'Platform':                                             'category',
        'Processor':                                            'category',
        'OsVer':                                                'category',
        'OsBuild':                                              'int16',
        'OsSuite':                                              'int16',
        'OsPlatformSubRelease':                                 'category',
        'OsBuildLab':                                           'category',
        'SkuEdition':                                           'category',
        'IsProtected':                                          'float16',
        'AutoSampleOptIn':                                      'int8',
        'PuaMode':                                              'category',
        'SMode':                                                'float16',
        'IeVerIdentifier':                                      'float16',
        'SmartScreen':                                          'category',
        'Firewall':                                             'float16',
        'UacLuaenable':                                         'float64', # was 'float32'
        'Census_MDC2FormFactor':                                'category',
        'Census_DeviceFamily':                                  'category',
        'Census_OEMNameIdentifier':                             'float32', # was 'float16'
        'Census_OEMModelIdentifier':                            'float32',
        'Census_ProcessorCoreCount':                            'float16',
        'Census_ProcessorManufacturerIdentifier':               'float16',
        'Census_ProcessorModelIdentifier':                      'float32', # was 'float16'
        'Census_ProcessorClass':                                'category',
        'Census_PrimaryDiskTotalCapacity':                      'float64', # was 'float32'
        'Census_PrimaryDiskTypeName':                           'category',
        'Census_SystemVolumeTotalCapacity':                     'float64', # was 'float32'
        'Census_HasOpticalDiskDrive':                           'int8',
        'Census_TotalPhysicalRAM':                              'float32',
        'Census_ChassisTypeName':                               'category',
        'Census_InternalPrimaryDiagonalDisplaySizeInInches':    'float32', # was 'float16'
        'Census_InternalPrimaryDisplayResolutionHorizontal':    'float32', # was 'float16'
        'Census_InternalPrimaryDisplayResolutionVertical':      'float32', # was 'float16'
        'Census_PowerPlatformRoleName':                         'category',
        'Census_InternalBatteryType':                           'category',
        'Census_InternalBatteryNumberOfCharges':                'float64', # was 'float32'
        'Census_OSVersion':                                     'category',
        'Census_OSArchitecture':                                'category',
        'Census_OSBranch':                                      'category',
        'Census_OSBuildNumber':                                 'int16',
        'Census_OSBuildRevision':                               'int32',
        'Census_OSEdition':                                     'category',
        'Census_OSSkuName':                                     'category',
        'Census_OSInstallTypeName':                             'category',
        'Census_OSInstallLanguageIdentifier':                   'float16',
        'Census_OSUILocaleIdentifier':                          'int16',
        'Census_OSWUAutoUpdateOptionsName':                     'category',
        'Census_IsPortableOperatingSystem':                     'int8',
        'Census_GenuineStateName':                              'category',
        'Census_ActivationChannel':                             'category',
        'Census_IsFlightingInternal':                           'float16',
        'Census_IsFlightsDisabled':                             'float16',
        'Census_FlightRing':                                    'category',
        'Census_ThresholdOptIn':                                'float16',
        'Census_FirmwareManufacturerIdentifier':                'float16',
        'Census_FirmwareVersionIdentifier':                     'float32',
        'Census_IsSecureBootEnabled':                           'int8',
        'Census_IsWIMBootEnabled':                              'float16',
        'Census_IsVirtualDevice':                               'float16',
        'Census_IsTouchEnabled':                                'int8',
        'Census_IsPenCapable':                                  'int8',
        'Census_IsAlwaysOnAlwaysConnectedCapable':              'float16',
        'Wdft_IsGamer':                                         'float16',
        'Wdft_RegionIdentifier':                                'float16',
        'HasDetections':                                        'int8'
        }
malware = pd.read_csv('/content/gdrive/My Drive/Notebooks Colab/train.csv', dtype=dtypes)
malware_test = pd.read_csv('/content/gdrive/My Drive/Notebooks Colab/test.csv', dtype=dtypes)
malware.head()
# Let's determine the type of each feature
# Easiest to do for binary types:

bins = [col for col in malware.columns if malware[col].nunique()==2]

# Now it is tricky to distinguish between the numerical and categorical data - 
# a lot of categorical features have numerical values so we need to distinguish them by inspection?
# According to [1], the following are true numericals:

nums = ['Census_ProcessorCoreCount',
        'Census_PrimaryDiskTotalCapacity',
        'Census_SystemVolumeTotalCapacity',
        'Census_TotalPhysicalRAM',
        'Census_InternalPrimaryDiagonalDisplaySizeInInches',
        'Census_InternalPrimaryDisplayResolutionHorizontal',
        'Census_InternalPrimaryDisplayResolutionVertical',
        'Census_InternalBatteryNumberOfCharges']

cats = [col for col in malware.columns if col not in bins if col not in nums]
feature_dict = [['Binary Features', len(bins)], ["Numerical Features", len(nums)], 
               ["Categorical Features", len(cats)]]
categories = pd.DataFrame(feature_dict, columns=['Feature Type', 'Feature Count'])
len(cats) + len(bins) + len(nums)
plt.figure(figsize = (10,10))
sns.barplot(data = categories.sort_values(by = 'Feature Count', ascending = False),
            x = "Feature Count", y = "Feature Type")
# as noted in the presentation, most of the features are categorical
# are all rows distinct?
len(malware) == malware['MachineIdentifier'].nunique()
# Look for missing data:
# calculate the percentage of missing values in each column
missing_data = (malware.isnull().sum()/malware.shape[0]).sort_values(ascending = False)
print(f"There are {len(missing_data[missing_data == 0])} columns with no missing data.")
print(f"There are {len(missing_data[missing_data != 0])} columns with some data missing.")
# we see that more than 50% of the features have some missing data
# get the first 20 columns with the highest % of missing values
missing_data.head(20).sort_values().plot(kind = "barh", figsize = (10,20), fontsize = 30)
# add a horizontal line for clearer comparison
plt.vlines(x = 1, ymin = -1, ymax = len(missing_data))

#plt.xlabel(size = 100)
missing_data.head(20)<jupyter_output><empty_output><jupyter_text>We have three columns with over 95% missing data. These are:- **PuaMode** -  Pua Enabled mode from the service (PUA are Potentially Unwanted Applications). By default in Windows 10 (version 2004 and later), Microsoft Defender Antivirus blocks apps that are considered PUA, for Enterprise (E5) devices.- **Census_ProcessorClass** - Census_ProcessorClass - A classification of processors into high/medium/low. Initially used for Pricing Level SKU. No longer maintained and updated.- **DefaultBrowsersIdentifier** - ID for the machine's default browser.It seems that while these could potentially be important in our analysis, the fact that there are so many missing values simply disqualifies them for further consideration?<jupyter_code>for feat in list(missing_data.head(3).index):
    print(malware[feat].value_counts())
    print(malware[feat].nunique())
    print("\n")
    
#  --- MISSING DATA (>95%) ----

# NUCLEAR OPTION: ANNIHILATE ALL
# to_drop = list(missing_data.head(3).index)
# malware.drop(to_drop, inplace=True, axis=1)
    
# OPTION A: PULLING THE TRIGGER FOR SELECTED FEATURES
# This seems like a good choice for the PuaMode and Census_ProcessorClass? 
# to_drop = ['PuaMode', 'Census_ProcessorClass']
# malware.drop(labels = to_drop, axis=1, inplace = True)

# OPTION B: FIXING?
# 'DefaultBrowsersIdentifier' are denoted with numbers. Would it make sense to use some number for
# the missing data that we will qualify as "OTHER" or "UNKNOWN"? People who care about browsers
# may behave safer on the web etc.
# malware["DefaultBrowsersIdentifier"].fillna(777, inplace = True)<jupyter_output><empty_output><jupyter_text>The italic text in the brackets next to the features are the definitions from Kaggle6 columns have roughly more than a third of the data missing. These are:- **Census_IsFlightingInternal** (*NA*, 83% missing) - what do they mean by flighting?? No idea what it represents- **Census_InternalBatteryType** (*NA*, 71% missing) - can battery type be important? - **Census_ThresholdOptIn** (*NA*, 64% missing) - I don't know what that is- **Census_IsWIMBootEnabled** (*NA*, 63% missing) - something system-boot related? I don't know- **SmartScreen** (*This is the SmartScreen enabled string value from registry. This is obtained by checking in order, HKLM\SOFTWARE\Policies\Microsoft\Windows\System\SmartScreenEnabled and HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\SmartScreenEnabled. If the value exists but is blank, the value "ExistsNotSet" is sent in telemetry*, 36% missing) - - **OrganizationIdentifier** (*ID for the organization the machine belongs in, organization ID is mapped to both specific companies and broad industries*, 31% missing) - this may be important as some organizations might be a more common target of malware attacks due to the nature of their businesses, however we cannot really 'fill' those values with random values or deduce them? Would need to encode them as "OTHER" or "UNKNOWN"A lot of notebooks on this copetition throw away features with >70% missing data<jupyter_code>for feat in list(missing_data.iloc[3:9].index):
    print(malware[feat].value_counts())
    print(malware[feat].nunique())
    print("\n")

# NUCLEAR OPTION: ANNIHILATE ALL
# to_drop = list(missing_data.iloc[3:9].index)
# malware.drop(to_drop, inplace=True, axis=1)

# for 'SmartScreen', we see that some categories are doubled becasue of sloppiness - this could be fixed
# and another class denoting 'UNKNOWN'/'OTHER' could be added?
# screen_fix = {"Off", "Prompt", "Warn", "RequireAdmin", "On", }
# Another thing we might want too look at is to consider the cardinality of the data
cardinals = pd.DataFrame([[col, malware[col].nunique()] for col in cats if col != 'MachineIdentifier'],
                          columns = ['Feature', 'Cardinality'])
cardinals['Cardinality'] = cardinals['Cardinality'].astype(float)
cardinals.sort_values(by = "Cardinality", ascending = True).plot(kind = 'barh', x = "Feature",
                                               y = "Cardinality",
                                               figsize = (15,40), fontsize = 40)
cardinals.sort_values(by = "Cardinality", ascending = True)
# In categorical features, consider whether there is a majority class which 'dominates' 
# over the whole columns - this includes both binary and non-binary categorical features

domination = pd.DataFrame(
    data = [[col, malware[col].value_counts(normalize = True).values[0], 
             malware[col].nunique()] for col in malware.columns if col not in nums], 
    columns = ['Feature', 'domination of main class', "Cardinality"]
)
domination.sort_values(by = "domination of main class", ascending = True).plot(kind = 'barh', x = "Feature",
                                               y = "domination of main class",
                                               figsize = (15,40), fontsize = 30)
# add a horizontal line for clearer comparison
plt.vlines(x = 1, ymin = -1, ymax = len(domination))


# some guys on Kaggle remove features with >98% majority class dominance
# for both binaries and non-binaries<jupyter_output><empty_output><jupyter_text>So the possible bad aspects of our training dataset are:- some columns with missing parts of data- columns with big variety of categories- some columns with a very dominant one type of class (majority category covers more than >90% of occurences[3] - skeweness?)However, it is also important to consider how the training set is related to the test set (or is it? thanks Rodrigo for the insight):<jupyter_code>#####################################
######## QUESTIONABLE PART ##########
#####################################

'''
card_test = pd.DataFrame(
    data = [[col, 
             malware[col].nunique(),
             malware_test[col].nunique()] for col in cats if col != "MachineIdentifier"],
    columns = ['Feature', "Card. Train", "Card. Test"]
)

card_test['Abs. % Difference'] = 100*abs((card_test['Card. Train']-card_test['Card. Test'])/card_test['Card. Train'])
card_test.sort_values(by="Abs. % Difference", ascending = False)

# we see that some features have different classes in the train and test sets
# some have more classes in the train, some have more classes in the test 
# this seems important but what can we do about it?
'''
##################################################################################
###################### FURTHER IDEAS FOR REMOVING COLUMNS ########################
##################################################################################
#
# - We have Country/City/GeoName Identifiers - Contry has NO missing values
#   while City/GeoName have some (~3% and <0.1%). Surely GeoName can be tossed away because
#   a given country obviously corresponds to a particular geographical region... What about 
#   cities? 
# - The ID column is to be dropped in the end of course
# - more ideas
# Is our dataset balanced for training?
plt.figure(figsize = (8,8))
sns.countplot(data = malware, x = "HasDetections")
# there is almost a 50:50 split of PWNED and non-PWNED machine, which is good
# Look for bias in binary features?
for binary in bins:
    if binary != 'HasDetections':
        plt.figure()
        sns.countplot(data = malware, x = binary, hue = 'HasDetections')
        plt.show()
# all but the infamous 'PuaMode' seem fine, yet another reason to get rid of it
# the dominances of major class are also visible here:

cond = 0.95
bin_dom = domination[(domination['domination of main class'] > cond) & (domination['Cardinality'] == 2)]
print(bin_dom)
print("\n")
print(f"There are {len(bins)} binary features, {len(bin_dom)} of them have the majority class in > {100*cond}% occurences")

def entropy(series):
    pX = series.value_counts(normalize=True, dropna=False)
    return -np.sum(pX * np.log2(pX))

def condEntropy(df, X, Y):
    """Compute the conditional H(X|Y) in dataframe df
    Args:
        df: a dataframe
        X: the name of the conditioning columt
        Y: the name of the column whose entropy we wish to compute
    """
    pY = df[Y].value_counts(normalize=True, dropna=False)
    HXgivenY = np.array(df.groupby(Y, dropna=False).agg({X: entropy}).reindex(pY.index)).reshape(-1)
    return np.sum(np.array(pY) * HXgivenY)

entropyTarget = entropy(malware['HasDetections'])
mutualInfos = {}
for column in malware:
    # column = 'PuaMode'
    if column != 'MachineIdentifier' and column != 'HasDetections':
        if dtypes[column] == 'category' and malware[column].isnull().values.any():
            malware[column] = malware[column].cat.add_categories("NaN")
            malware[column].fillna("NaN", inplace=True)       
        mi = entropyTarget - condEntropy(malware, 'HasDetections', column)
        mutualInfos[column] = mi
        with open('/content/gdrive/My Drive/Notebooks Colab/mutualInfos.pickle', 'wb') as handle:
            pickle.dump(mutualInfos, handle, protocol=pickle.HIGHEST_PROTOCOL)
        print('I(target, {}) = {}'.format(column, mi))
import itertools

columns = malware.columns.values.tolist()
columns.remove('HasDetections')
columns.remove('MachineIdentifier')

featureMIs = {}
entropies = {}

for (var1, var2) in itertools.combinations(columns, 2):
    if var1 != var2:
        entropies[var1] = entropy(malware[var1])
        featureMIs[f"{var1},{var2}"] = entropies[var1] - condEntropy(malware, var1, var2)
        print(f'I({var1}, {var2}) = {featureMIs[f"{var1},{var2}"]}')
        with open('/content/gdrive/My Drive/Notebooks Colab/featureMIs.pickle', 'wb') as handle:
            pickle.dump(featureMIs, handle, protocol=pickle.HIGHEST_PROTOCOL)
        with open('/content/gdrive/My Drive/Notebooks Colab/entropies.pickle', 'wb') as handle:
            pickle.dump(entropies, handle, protocol=pickle.HIGHEST_PROTOCOL)<jupyter_output><empty_output><jupyter_text>Preprocessing Some data cleansing and appropriate formatting. There are some variables that currently are type int although they are not ordinal, but I thought their cardinality was way too large to be categorical considering that to use scikit models we'll need to one-hot encode them.<jupyter_code>preProcessing = {
    'EngineVersion': lambda x: x.str.replace('.', '').astype(np.uint32),
    'AppVersion': lambda x: x.str.replace('.', '').astype(np.uint64),
    'AvSigVersion': lambda x: x.str.replace('.', '').str.replace('&#x17;', '').astype(np.uint32),
    'RtpStateBitfield': lambda x: x.astype('category'),
    'DefaultBrowsersIdentifier': lambda x: x.astype(np.uint16),
    'AVProductStatesIdentifier': lambda x: x.astype(np.uint32),
    'AVProductsInstalled': lambda x: x.astype('category'),
    'AVProductsEnabled': lambda x: x.astype('category'),
    'CityIdentifier': lambda x: x.astype(np.uint32),
    'OrganizationIdentifier': lambda x: x.astype(np.uint8),
    'GeoNameIdentifier': lambda x: x.astype(np.uint16),
    'OsVer': lambda x: x.str.replace('.', '').astype(np.uint32),
    'IsProtected': lambda x: x.astype(np.uint8),
    'PuaMode': lambda x: x.astype(np.uint8),
    'SMode': lambda x: x.astype(np.uint8),
    'IeVerIdentifier': lambda x: x.astype(np.uint16),
    'Firewall': lambda x: x.astype(np.uint8),
    'UacLuaenable': lambda x: x.astype('category'),
    'Census_OEMNameIdentifier': lambda x: x.astype(np.uint16),
    'Census_OEMModelIdentifier': lambda x: x.astype(np.uint32),
    'Census_ProcessorCoreCount': lambda x: x.astype(np.uint8),
    'Census_ProcessorManufacturerIdentifier': lambda x: x.astype('category'),
    'Census_ProcessorModelIdentifier': lambda x: x.astype(np.uint16),
    'Census_PrimaryDiskTotalCapacity': lambda x: x.astype(np.uint64),
    'Census_SystemVolumeTotalCapacity': lambda x: x.astype(np.uint32),
    'Census_TotalPhysicalRAM': lambda x: x.astype(np.uint32),
    'Census_InternalPrimaryDisplayResolutionHorizontal': lambda x: x.astype(np.int32),  # (Has missing values as -1)
    'Census_InternalPrimaryDisplayResolutionVertical': lambda x: x.astype(np.int16),  # (Has missing values as -1)
    'Census_InternalBatteryNumberOfCharges': lambda x: x.astype(np.uint32),
    'Census_OSVersion': lambda x: x.str.replace('.', '').astype(np.uint64),
    'Census_OSInstallLanguageIdentifier': lambda x: x.astype('category'),
    'Census_IsFlightingInternal': lambda x: x.astype(np.uint8),
    'Census_IsFlightsDisabled': lambda x: x.astype(np.uint8),
    'Census_ThresholdOptIn': lambda x: x.astype(np.uint8),
    'Census_FirmwareManufacturerIdentifier': lambda x: x.astype(np.uint16),
    'Census_FirmwareVersionIdentifier': lambda x: x.astype(np.uint32),
    'Census_IsWIMBootEnabled': lambda x: x.astype(np.uint8),
    'Census_IsVirtualDevice': lambda x: x.astype(np.uint8),
    'Census_IsAlwaysOnAlwaysConnectedCapable': lambda x: x.astype(np.uint8),
    'Wdft_IsGamer': lambda x: x.astype(np.uint8),
    'Wdft_RegionIdentifier': lambda x: x.astype('category'),
}<jupyter_output><empty_output><jupyter_text>We apply the preprocessing functions. The try-catch is to deal with the casting to integers of variables with NaN values, that is to reason for so many senseless floats.<jupyter_code>malwareProcessed = malware.copy()
malwareProcessed.drop('MachineIdentifier', inplace=True, axis=1)

for column in malwareProcessed:
    if column in preProcessing:
        try:
            malwareProcessed[column] = preProcessing[column](malwareProcessed[column])
        except Exception as e:
            if str(e) == "Cannot convert non-finite values (NA or inf) to integer":
                malwareProcessed[column] = malwareProcessed[column].fillna(malwareProcessed[column].mode()[0])
                malwareProcessed[column] = preProcessing[column](malwareProcessed[column])<jupyter_output><empty_output><jupyter_text>Checking the data type and cardinality of all features:<jupyter_code>for column in malwareProcessed:
    print(f"{column} ({malwareProcessed[column].dtype}): {malwareProcessed[column].nunique()}")<jupyter_output>ProductName (category): 6
EngineVersion (uint32): 70
AppVersion (uint64): 120
AvSigVersion (uint32): 9357
IsBeta (int8): 2
RtpStateBitfield (category): 8
IsSxsPassiveMode (int8): 2
DefaultBrowsersIdentifier (uint16): 1757
AVProductStatesIdentifier (uint32): 23492
AVProductsInstalled (category): 6
AVProductsEnabled (category): 6
HasTpm (int8): 2
CountryIdentifier (int16): 222
CityIdentifier (uint32): 105817
OrganizationIdentifier (uint8): 50
GeoNameIdentifier (uint16): 289
LocaleEnglishNameIdentifier (int16): 278
Platform (category): 4
Processor (category): 3
OsVer (uint32): 44
OsBuild (int16): 78
OsSuite (int16): 13
OsPlatformSubRelease (category): 9
OsBuildLab (category): 673
SkuEdition (category): 8
IsProtected (uint8): 2
AutoSampleOptIn (int8): 2
PuaMode (category): 2
SMode (uint8): 2
IeVerIdentifier (uint16): 294
SmartScreen (category): 21
Firewall (uint8): 2
UacLuaenable (category): 10
Census_MDC2FormFactor (category): 14
Census_DeviceFamily (category): 3
Census_OEMNameIdentifier [...]<jupyter_text>Dealing with NaNs according to each data-type:<jupyter_code>for column in malwareProcessed:
    if hasattr(malwareProcessed[column], 'cat'):
        malwareProcessed[column] = malwareProcessed[column].cat.add_categories("NaN")
        malwareProcessed[column].fillna("NaN", inplace=True)
    elif malwareProcessed[column].dtype in ['uint8', 'int8', 'uint16', 'int16', 'uint32', 'int32', 'uint64', 'int64']:
        malwareProcessed[column].fillna(malwareProcessed[column].mode(), inplace=True)
    elif malwareProcessed[column].dtype in ['float16', 'float32']:
        malwareProcessed[column].fillna(malwareProcessed[column].mean(), inplace=True)<jupyter_output><empty_output><jupyter_text>Now we verify that there are no more NaNs :)<jupyter_code>for column in malwareProcessed:
    if malwareProcessed[column].isnull().any():
        print(f"¡Mierda!, {column} ({malwareProcessed[column].dtype}) has still NaN values! D:")<jupyter_output><empty_output><jupyter_text>Let's save a pickle with the processed data set to check if pickles are indeed faster:<jupyter_code>with open('/content/gdrive/My Drive/Notebooks Colab/cleanDF.pickle', 'wb') as handle:
    pickle.dump(malwareProcessed, handle, protocol=pickle.HIGHEST_PROTOCOL)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10221298084063568, 'fraction_numerical': 0.016159215802762514, 'mean_word_length': 3.1021079566223957, 'pattern_counts': {'":': 1, '<': 41, '<?xml version=': 0, '>': 46, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 67, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '577491', 'license': 'MIT', 'n_tokens_mistral': 9540, 'n_tokens_neox': 8979, 'n_words': 2947, 'path': 'Initial_EDA_Colab.ipynb', 'repo_name': 'rcofrer/MLWR', 'size': '33665'}