<jupyter_start><jupyter_text>**SONAR_MEASURES_difference**This notebook the creation of the table `SONAR_MEASURES_difference`, that contains the difference between consecutive commits of the same project in order to know the incorporations made by each committer.First, we import the libraries we need and, then, we read the corresponding csv files.<jupyter_code>import pandas as pd
import numpy as np
from datetime import datetime
sonarMeasures = pd.read_csv("../../../data/interim/DataPreparation/CleanData/SONAR_MEASURES_clean.csv").iloc[:, 2:]
print(sonarMeasures.shape)
sonarMeasures.head()
gitCommits = pd.read_csv("../../../data/interim/DataPreparation/CleanData/GIT_COMMITS_clean.csv").iloc[:, 2:]
print(gitCommits.shape)
gitCommits.head()<jupyter_output>(140653, 5)<jupyter_text>Firstly, we convert the attribute `committerDate` to the correct timestamps format:<jupyter_code>gitCommits['committerDate'] =  pd.to_datetime(gitCommits['committerDate'], format='%Y-%m-%dT%H:%M:%SZ')
gitCommits.head()<jupyter_output><empty_output><jupyter_text>To create this new attributes, we need to compute the difference of each value in a commit with the previous commit of the same project. Therefore, we will know the changes made by each developer in the project.So, we need to know the time when the commit was made in order to know the order among the different commits.That is why we have to join this two tables, `SONAR_MEASURES` and `GIT_COMMITS`:<jupyter_code>print(sonarMeasures.shape)
newDF = pd.merge(sonarMeasures, gitCommits,  how='left', left_on=['commitHash','projectID'], right_on = ['commitHash','projectID'])
print(newDF.shape)
newDF.head()
newDFNaN = list(np.where(newDF.committerDate.isna()))[0]
len(newDFNaN)<jupyter_output><empty_output><jupyter_text>As we can see that, after the join of both tables, we have 4 records with missing values (because the pair `(commitHash,projectID)` from `SONAR_MEASURES` is not in `GIT_COMMITS`). So these lines have to be deleted.<jupyter_code>newDF = newDF.drop(newDFNaN)
newDF.shape
projectID = newDF.projectID.unique()
projectID.size
newDFsorted = newDF.sort_values(by=['projectID', 'committerDate'])
newDFsorted.head()
newDFsorted.shape[0]<jupyter_output><empty_output><jupyter_text>We define a function that computes the difference between two rows if they are of the same project:<jupyter_code>newDFsortedCopy = newDFsorted.copy()
project = newDFsorted.iloc[0,1]
for index, row in newDFsorted.iterrows():
  if index < 55625:
    if project == newDFsorted.iloc[index,1]:
      r = newDFsortedCopy.iloc[index-1:index+1,2:22].diff().iloc[1,:]
      newDFsorted.iloc[index:index+1,2:22] = np.array(r)
    else:
      project = newDFsorted.iloc[index,1]
newDFsorted.head()
newDFsorted.shape<jupyter_output><empty_output><jupyter_text>We take the same attributes that we had originally in `SONAR_MEASURES`:<jupyter_code>sonarMeasuresDifference = newDFsorted.iloc[:,:22]
sonarMeasuresDifference.head()
print(sonarMeasuresDifference.shape)
print(sonarMeasures.shape)<jupyter_output>(55625, 22)
(55629, 22)<jupyter_text>We save the results in a new csv file:<jupyter_code>sonarMeasuresDifference.to_csv('../../../data/interim/DataPreparation/ConstructData/SONAR_MEASURES_difference.csv', header=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11615853658536586, 'fraction_numerical': 0.014329268292682927, 'mean_word_length': 6.982968369829684, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '620012', 'license': 'MIT', 'n_tokens_mistral': 1150, 'n_tokens_neox': 1097, 'n_words': 270, 'path': 'notebooks/2-DataPreparation/3-ConstructData/8-DB-SONAR-MEASURES-difference.ipynb', 'repo_name': 'chus-chus/softwareDevTypes', 'size': '3280'}