<jupyter_start><jupyter_text>**DATA SELECTION** First, we select the tables that contains the variables of interest<jupyter_code>path1 = "../data/raw/TDD/GIT_COMMITS.csv"
path2 = "../data/raw/TDD/SONAR_ANALYSIS.csv"
path3 = "../data/raw/TDD/SONAR_MEASURES.csv"
db_commits = pd.read_csv(path1,lineterminator='\n')
db_analysis = pd.read_csv(path2)
db_measures = pd.read_csv(path3)<jupyter_output><empty_output><jupyter_text>Secondly, for each table, we select those variables:    · GIT_COMMITS: project id, commit hash, commit message, author and committer date    · SONAR_ANALYSIS: revision and anlysis key    · SONAR_MESURES: analysis key, complexity, violations and development cost<jupyter_code>db_commits = db_commits[["PROJECT_ID", "COMMIT_HASH", "COMMIT_MESSAGE", "AUTHOR",  "COMMITTER_DATE"]]
db_commits.head()
db_analysis = db_analysis[["REVISION", "ANALYSIS_KEY"]]
db_analysis.head()
db_measures = db_measures[["analysis_key", "complexity", "violations", "development_cost"]]
db_measures.head()<jupyter_output><empty_output><jupyter_text>**DATA CLEANING** We proceed to clean the tables by deleting the NA only in the sonar_measures table as we want to keep the time consistency when computing the invervals between the selected variables.<jupyter_code>db_measures = db_measures[db_measures["complexity"].notna()]
db_sonar0 = db_analysis.merge(db_measures, left_on="ANALYSIS_KEY", right_on="analysis_key")
db_sonar = db_sonar0.rename(columns = {'REVISION' : 'COMMIT_HASH'})
db_sonar = db_sonar[["COMMIT_HASH", "complexity", "violations", "development_cost"]]
db_sonar.head()
db_commits
db_sonar
db_merged = pd.merge(db_commits, db_sonar, how = 'left', on = 'COMMIT_HASH', indicator = True)
db_merged.head()
db_merged = db_merged.sort_values(by=["PROJECT_ID","COMMITTER_DATE"]).reset_index()

db_merged['inc_complexity'] = float("Nan")
db_merged['inc_violations'] = float("Nan")
db_merged['inc_development_cost'] = float("Nan")

db_merged.append
for i in range(1,db_merged.shape[0]):

    #first we make sure that both entries are from the same project (if not leave with the Nan value in the increment variable)
    if (db_merged['PROJECT_ID'][i] == db_merged['PROJECT_ID'][i-1]):

        for inc_variable in [["complexity",'inc_complexity'],["violations", "inc_violations"],["development_cost", "inc_development_cost"]]:
            variable_act = db_merged[inc_variable[0]][i] #value for the variable in the row i
            variable_past =  db_merged[inc_variable[0]][i-1] #value for the variable in the row before i
            
            if pd.notna(variable_act) and pd.notna(variable_past): #both entries available
                db_merged[inc_variable[1]][i] = variable_act - variable_past
            else:
                break

db_increases = db_merged[db_merged['inc_complexity'].notna() & db_merged['inc_violations'].notna() & db_merged['inc_development_cost'].notna()]

    
clean_db_merged = db_merged.dropna() #we delete al the NAs in the table

final_db = clean_db_merged[["PROJECT_ID", "COMMIT_HASH", "COMMIT_MESSAGE", "AUTHOR", "COMMITTER_DATE", "inc_complexity", "inc_violations", "inc_development_cost"]]
final_db.head()
print(np.mean(final_db["inc_complexity"]),np.var(final_db["inc_complexity"]),max(final_db["inc_complexity"]),min(final_db["inc_complexity"]))
print(np.mean(final_db["inc_violations"]),np.var(final_db["inc_violations"]),max(final_db["inc_violations"]),min(final_db["inc_violations"]))
print(np.mean(final_db["inc_development_cost"]),np.var(final_db["inc_development_cost"]),max(final_db["inc_development_cost"]),min(final_db["inc_development_cost"]))
final_db.to_csv('../data/processed/predictionDB.csv', index='False') #export!
print(final_db.shape)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1414033208355651, 'fraction_numerical': 0.004017139796464917, 'mean_word_length': 6.946808510638298, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '521481', 'license': 'FTL', 'n_tokens_mistral': 1369, 'n_tokens_neox': 1295, 'n_words': 242, 'path': 'notebooks/prep.ipynb', 'repo_name': 'megaelius/EPAA', 'size': '3734'}