<jupyter_start><jupyter_text>Import dataframes<jupyter_code>allBuilds = pd.read_csv("../csv/allBuilds.csv", index_col=0)
pr_commits = pd.read_csv("../csv/pr_commits.csv", index_col=0)
pr_commits<jupyter_output><empty_output><jupyter_text>Import jobs<jupyter_code>job_files = glob.glob("../jobs/*.pkl")
jobs = []
for job_file in job_files:
    with open(job_file, "rb") as f:
        jobs = jobs + pickle.load(f)
len(jobs)
pp.pprint(jobs[0])
job_tuples = []
for job in jobs:
    job_values = []
    for field, subfields in fields.items():
        if not subfields:
            job_values.append(job[field])
            continue
        for subfield in subfields:
            job_values.append(job[field][subfield])
    job_tuples.append(tuple(job_values))
len(job_tuples)
columns = []
for field, subfields in fields.items():
        if not subfields:
            columns.append(field)
            continue
        for subfield in subfields:
            columns.append(f"{field}{subfield}")
columns
jobsDf = pd.DataFrame(job_tuples, columns = columns)
jobsDf.to_csv(f"{csv_folder}/allJobs.csv")
jobsDf<jupyter_output><empty_output><jupyter_text>**Jobs with no commits**<jupyter_code>jobsDf["commitsha"].isna().sum()
job_id_build_id = pd.read_csv("../csv/job_id_build_id.csv", index_col=0)
job_id_build_id
len(job_id_build_id.job_id.unique())
x = job_id_build_id[~job_id_build_id.job_id.isin(jobsDf.id.unique())]
x<jupyter_output><empty_output><jupyter_text>We have all jobs!  Pull request tests<jupyter_code>pr_number = 2104
pull_req = f"refs/pull/{pr_number}/merge"
pr_sha = pr_commits[pr_commits.pr_number == pr_number].commitsha.unique()
pr_sha
jobsDf[jobsDf.commitsha.isin(pr_sha)]
jobsDf[jobsDf.commitref == pull_req]<jupyter_output><empty_output><jupyter_text>Many more jobs associated with that PR than the ones retrieved with the PR list of commits. Why? **PR Jobs to investigate**<jupyter_code>jobsDf[(jobsDf.commitref == pull_req) & ~(jobsDf.commitsha.isin(pr_sha))]
len(jobsDf[jobsDf.commitref == pull_req]) - len(jobsDf[(jobsDf.commitref == pull_req) & ~(jobsDf.commitsha.isin(pr_sha))])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1382228490832158, 'fraction_numerical': 0.0037611659614480487, 'mean_word_length': 5.1502890173410405, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '127320', 'license': 'MIT', 'n_tokens_mistral': 840, 'n_tokens_neox': 812, 'n_words': 126, 'path': 'notebooks/parser_and_validation/Job parser.ipynb', 'repo_name': 'IncioMan/ci_data_issue_prediction', 'size': '2127'}