<jupyter_start><jupyter_text>**SONAR_MEASURES**This notebook the cleaning of the attributes of the table `SONAR_MEASURES`.First, we import the libraries we need and, then, we read the corresponding csv.<jupyter_code>import pandas as pd
import numpy as np
sonarMeasures = pd.read_csv("../../../data/interim/DataPreparation/SelectData/SONAR_MEASURES_select.csv")
print(sonarMeasures.shape)
sonarMeasures.head()<jupyter_output>(55629, 23)<jupyter_text>We define a function that returns, given two lists, their intersection.<jupyter_code>def intersection(l1, l2):
  temp = set(l2)
  l3 = [value for value in l1 if value in temp]
  return l3<jupyter_output><empty_output><jupyter_text>Next, for each attribute, we treat the missing values.  commitHash<jupyter_code>len(sonarMeasures.commitHash.unique())
commitHashNan = list(np.where(sonarMeasures.commitHash.isna()))[0]
len(commitHashNan)<jupyter_output><empty_output><jupyter_text>projectID<jupyter_code>len(sonarMeasures.projectID.unique())
projectIDNan = list(np.where(sonarMeasures.projectID.isna()))[0]
len(projectIDNan)<jupyter_output><empty_output><jupyter_text>functions<jupyter_code>functionsNan = list(np.where(sonarMeasures.functions.isna()))[0]
len(functionsNan)<jupyter_output><empty_output><jupyter_text>commentLinesDensity<jupyter_code>commentLinesDensityNan = list(np.where(sonarMeasures.commentLinesDensity.isna()))[0]
len(commentLinesDensityNan)<jupyter_output><empty_output><jupyter_text>complexity<jupyter_code>complexityNan = list(np.where(sonarMeasures.complexity.isna()))[0]
len(complexityNan)<jupyter_output><empty_output><jupyter_text>functionComplexity<jupyter_code>functionComplexityNan = list(np.where(sonarMeasures.functionComplexity.isna()))[0]
len(functionComplexityNan)<jupyter_output><empty_output><jupyter_text>duplicatedLinesDensity<jupyter_code>duplicatedLinesDensityNan = list(np.where(sonarMeasures.duplicatedLinesDensity.isna()))[0]
len(duplicatedLinesDensityNan)<jupyter_output><empty_output><jupyter_text>violations<jupyter_code>violationsNan = list(np.where(sonarMeasures.violations.isna()))[0]
len(violationsNan)<jupyter_output><empty_output><jupyter_text>blockerViolations<jupyter_code>blockerViolationsNan = list(np.where(sonarMeasures.blockerViolations.isna()))[0]
len(blockerViolationsNan)<jupyter_output><empty_output><jupyter_text>criticalViolations<jupyter_code>criticalViolationsNan = list(np.where(sonarMeasures.criticalViolations.isna()))[0]
len(criticalViolationsNan)<jupyter_output><empty_output><jupyter_text>infoViolations<jupyter_code>infoViolationsNan = list(np.where(sonarMeasures.infoViolations.isna()))[0]
len(infoViolationsNan)<jupyter_output><empty_output><jupyter_text>majorViolations<jupyter_code>majorViolationsNan = list(np.where(sonarMeasures.majorViolations.isna()))[0]
len(majorViolationsNan)<jupyter_output><empty_output><jupyter_text>minorViolations<jupyter_code>minorViolationsNan = list(np.where(sonarMeasures.minorViolations.isna()))[0]
len(minorViolationsNan)<jupyter_output><empty_output><jupyter_text>codeSmells<jupyter_code>codeSmellsNan = list(np.where(sonarMeasures.codeSmells.isna()))[0]
len(codeSmellsNan)<jupyter_output><empty_output><jupyter_text>bugs<jupyter_code>bugsNan = list(np.where(sonarMeasures.bugs.isna()))[0]
len(bugsNan)<jupyter_output><empty_output><jupyter_text>vulnerabilities<jupyter_code>vulnerabilitiesNan = list(np.where(sonarMeasures.vulnerabilities.isna()))[0]
len(vulnerabilitiesNan)<jupyter_output><empty_output><jupyter_text>cognitiveComplexity<jupyter_code>cognitiveComplexityNan = list(np.where(sonarMeasures.cognitiveComplexity.isna()))[0]
len(cognitiveComplexityNan)<jupyter_output><empty_output><jupyter_text>ncloc<jupyter_code>nclocNan = list(np.where(sonarMeasures.ncloc.isna()))[0]
len(nclocNan)<jupyter_output><empty_output><jupyter_text>sqaleIndex<jupyter_code>sqaleIndexNan = list(np.where(sonarMeasures.sqaleIndex.isna()))[0]
len(sqaleIndexNan)<jupyter_output><empty_output><jupyter_text>sqaleDebtRatio<jupyter_code>sqaleDebtRatioNan = list(np.where(sonarMeasures.sqaleDebtRatio.isna()))[0]
len(sqaleDebtRatioNan)<jupyter_output><empty_output><jupyter_text>reliabilityRemediationEffort<jupyter_code>reliabilityRemediationEffortNan = list(np.where(sonarMeasures.reliabilityRemediationEffort.isna()))[0]
len(reliabilityRemediationEffortNan)<jupyter_output><empty_output><jupyter_text>securityRemediationEffort<jupyter_code>securityRemediationEffortNan = list(np.where(sonarMeasures.securityRemediationEffort.isna()))[0]
len(securityRemediationEffortNan)<jupyter_output><empty_output><jupyter_text>--- We save it into a new csv.<jupyter_code>sonarMeasures.to_csv('../../../data/interim/DataPreparation/CleanData/SONAR_MEASURES_clean.csv', header=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1266329540665824, 'fraction_numerical': 0.007374631268436578, 'mean_word_length': 16.779026217228463, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 100, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '97680', 'license': 'MIT', 'n_tokens_mistral': 1887, 'n_tokens_neox': 1776, 'n_words': 125, 'path': 'notebooks/2-DataPreparation/2-CleanData/2-DB-SONAR-MEASURES.ipynb', 'repo_name': 'chus-chus/softwareDevTypes', 'size': '4746'}