<jupyter_start><jupyter_text>| Field          | Description                                                     | Type of variable ||----------------|-----------------------------------------------------------------|------------------|| fixVersions    | Search for issues that are assigned to a particular fix version | list of versions || issuetype      | Type of issue                                                   | Categorical      || project        | The project the issue belongs to                                | String           || resolution     | The resolution of the issue                                     | Categorical      || resolutiondate | The date the issue was resolved                                 | Date             || workratio      | calculated as workRatio = (timeSpent / originalEstimate) x 100  | Number           || lastViewed     | The date at which the issue was last viewed                     | Date             || watches        | Users watching the issue                                        | ?                || created        | The date of the creation of the issue                           | Date             || priority       | Priority assigned to the issue                                  | Categorical      || labels         | Labels assigned to the issue                                    | Strings          || versions       | The versions affected by the issue                              | List             || assignee       | The user the issue is assigned to                               | User             || status         | The status of the issue                                         | Categorical      || description    | The description of the issue                                    | String           || summary        | A summary of the issue reported                                 | String           || creator        | The user that created the issue                                 | User             || subtasks       | The subtasks of the issue                                       | List of issues   || reporter       | The user who reported the issue. May be the same as the creator | User             || duedate        | The date the issue is due to be resolved                        | Date             |  Import tags<jupyter_code>tags = pd.read_csv(f"{csv_folder}/tags_updated.csv", index_col=0)
tags.Date = pd.to_datetime(tags.Date)
tags = tags[tags.Date > "2015-03-10"]
tags["month"] = tags.Date.dt.month
tags["year"] = tags.Date.dt.year
tags.head()<jupyter_output><empty_output><jupyter_text>Import issues<jupyter_code>issues = pd.read_csv(f"{csv_folder}/issues_community.csv", index_col=0)
for date_field in ["created", "duedate", "lastViewed", "resolutiondate", "updated"]:
    issues[date_field] = pd.to_datetime(issues[date_field], errors="coerce")
issues.head()
issues.shape<jupyter_output><empty_output><jupyter_text>We filter only issues in the timeframe in which the project was using TravisCI  Import versions<jupyter_code>versions = pd.read_csv(f"{csv_folder}/issues_versions.csv", index_col=0)
versions.head()<jupyter_output><empty_output><jupyter_text>Import fixVersions<jupyter_code>fixversions = pd.read_csv(f"{csv_folder}/issues_fixversions.csv", index_col=0)
fixversions = fixversions[fixversions.issue_id.isin(issues.issue_id)]
fixversions.head(5)<jupyter_output><empty_output><jupyter_text>Import labels<jupyter_code>labels = pd.read_csv(f"{csv_folder}/issues_labels.csv", index_col=0)
labels = labels[labels.issue_id.isin(issues.issue_id)]
labels.head()<jupyter_output><empty_output><jupyter_text>Import components<jupyter_code>components = pd.read_csv(f"{csv_folder}/issues_components.csv", index_col=0)
components = components[components.issue_id.isin(issues.issue_id)]
components.head()<jupyter_output><empty_output><jupyter_text>Import subtasks<jupyter_code>substaks = pd.read_csv(f"{csv_folder}/issues_subtasks.csv", index_col=0)
substaks = substaks[substaks.issue_id.isin(issues.issue_id)]
substaks.head()<jupyter_output><empty_output><jupyter_text>Analysis  Shape<jupyter_code>issues.shape
fixversions.shape
versions.shape
labels.shape
substaks.shape<jupyter_output><empty_output><jupyter_text>Select only issues affecting the analysed release<jupyter_code>versions_issues = versions.merge(issues[["issue_id","issuetype_name"]], on="issue_id")
versions_issues[versions_issues.version_name > "5.2"].version_name.nunique()<jupyter_output><empty_output><jupyter_text>Columns<jupyter_code>issues.columns
versions.columns
fixversions.columns
labels.columns
substaks.columns<jupyter_output><empty_output><jupyter_text>Describe<jupyter_code>issues.describe()
issues.describe(include = ['O'])<jupyter_output><empty_output><jupyter_text>* Votes has 75% percentile to 0 (most of the issues did not get votes) but has also a value of 14* There are 6 issuetypes* There are 7 statuses    We therefore safely remove one of each columns and keep the one in the more human readable form<jupyter_code>duplicated_columns = ["issuetype", "assignee_key", "creator_key", "resolution", "priority", "reporter_key", "resolution_id"]
issues = issues[set(issues.columns) - set(duplicated_columns)]
fixversions.describe()
versions.describe()
labels.describe()
substaks.describe()<jupyter_output><empty_output><jupyter_text>We do not detect anomalies in the statistical summaries of the other datasets  Filter only the needed columns<jupyter_code>issues_cols = ["issue_id", "issuetype_name", "resolution_name","description", "resolutiondate", "created", "priority_name", "versions", "status", "edition"]
issues = issues[issues_cols]<jupyter_output><empty_output><jupyter_text>Missing values<jupyter_code>issues.isna().sum().sort_values(ascending=False).head(15)
issues[issues.resolution_name.isna()].status.value_counts()<jupyter_output><empty_output><jupyter_text>Drop edition columns<jupyter_code>issues = issues.drop(columns=["edition"])
fixversions.isna().sum()
versions.isna().sum()
labels.isna().sum()
substaks.isna().sum()<jupyter_output><empty_output><jupyter_text>* There are missing values only in the issues dataset* All fields presenting missing values are fields which are optional in an issue. We therefore not worry about their absence even though that has to be taken into account when working with those fields.  In depth analysis  IssueType<jupyter_code>issues.issuetype_name.unique()
issues.issuetype_name.value_counts()
plt.figure()
ax = sns.countplot(x="issuetype_name", data=issues[["issue_id", "issuetype_name"]],\
              order=issues.issuetype_name.value_counts().index)
ax.set(xlabel="issue type")
plt.savefig(f"{figure_folder}/issues/issuetype_dist.png", transparent=True, bbox_inches='tight', pad_inches=0.2)
plt.show()
issues.issuetype_name.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>* We see that issues can only have one of these 6 issue types.* 54% is Improvement, followed by 22% Bug **Focus only on bugs**<jupyter_code>issues = issues[issues.issuetype_name=="Bug"]
issues.shape<jupyter_output><empty_output><jupyter_text>Versions<jupyter_code>versions_merged = versions.merge(issues, on="issue_id")
versions_merged.version_name.sort_values().unique()
len(versions_merged.version_name.unique())<jupyter_output><empty_output><jupyter_text>**How many issues have a version**<jupyter_code>versions_merged.issue_id.nunique() / issues.issue_id.nunique()<jupyter_output><empty_output><jupyter_text>**Issues with version and critical or blocker priority**<jupyter_code>len(versions_merged[versions_merged.priority_name.isin(["Blocker", "Critical"])])<jupyter_output><empty_output><jupyter_text>**Number of versions per tags**<jupyter_code>sns.countplot(versions.groupby("issue_id").count().version_name.values)<jupyter_output>/Users/alexincerti/Library/Python/3.8/lib/python/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>**Distribution of missing tags on Jira**<jupyter_code>#Versions on Jira have a slightly different format at times. This functions formats the versions to match Jira's
def polish_version(version):
    if "latest" in version:
        return None
    splits = version.split(".")
    major = splits[0]
    release = splits[1]
    if(len(splits)>2):
        modification = splits[2]
        if(modification != "0"):
            return f"{major}.{release}.{modification}"
    """if("-" in release):
        release = release.split("-")[0]"""
    return f"{major}.{release}"
releases = set(map(lambda version: polish_version(version), tags.Tag.unique()))<jupyter_output><empty_output><jupyter_text>Type of version<jupyter_code>def type_of_version(version):
    if "RC" in version: 
        return "Release candidates"
    if "M" in version:
        return "Milestones"
    return "Normal releases"
pd.Series(map(type_of_version,versions_merged.version_name.unique())).value_counts(normalize=True).plot(kind="bar")
version_type = pd.DataFrame({"type": map(type_of_version,versions_merged.version_name.unique()),
              "version": versions_merged.version_name.unique()})
version_type.type.value_counts()
version_type[version_type.type=="Milestones"]
versions_issues[versions_issues.version_name.isin(["6.6-M1", "6.5-M1"])]
3/ len(versions_issues) * 100<jupyter_output><empty_output><jupyter_text>**Tags present on Jira but not on Github**<jupyter_code>len(set(versions.version_name.sort_values().unique()).difference(set(releases)))
len(set(set(versions.version_name.sort_values().unique())).intersection(set(releases)))<jupyter_output><empty_output><jupyter_text>**Report time distribution**<jupyter_code>all_tags = pd.read_csv(f"{csv_folder}/tags.csv", index_col=0)
all_tags.shape
all_tags = all_tags[all_tags.Tag.apply(lambda x: not "RC" in x and not "M" in x)]
all_tags["jira_version"] = all_tags.Tag.apply(polish_version)
all_tags.Date = pd.to_datetime(all_tags.Date)
versions_merged_all_tags = versions_merged.merge(all_tags[["jira_version", "Date"]], left_on="version_name", right_on="jira_version")
versions_merged_all_tags.created = pd.to_datetime(versions_merged_all_tags.created)
report_time = (versions_merged_all_tags.created.dt.tz_localize(None) - versions_merged_all_tags.Date)\
            / np.timedelta64(1, 'D')
plt.figure()
ax = sns.boxplot(x=report_time)
ax.set(xlabel="days from issue creation since the release")
plt.savefig(f"{figure_folder}/issues/opening_time_since_release.png", transparent=True, bbox_inches='tight', pad_inches=0.2)
plt.show()
report_time.describe()
report_time.quantile(0.9)<jupyter_output><empty_output><jupyter_text>**End of chapter for now, export csv**<jupyter_code>issues.to_csv(f"{csv_folder}/issues_community_final.csv")<jupyter_output><empty_output><jupyter_text>**Top versions**<jupyter_code>top_versions = versions_merged.groupby("version_name").count().issue_id.reset_index().sort_values(by="issue_id", ascending=False)
ax = sns.barplot(x="version_name", y="issue_id", data=top_versions[["issue_id", "version_name"]].head(30))
top_versions.head(10).version_name.values<jupyter_output><empty_output><jupyter_text>**Distribution of issuetypes for each version**<jupyter_code>versions_sorted = versions_merged.sort_values(by="version_name").version_name.unique()
f, axes = plt.subplots(4,1, figsize=(15,9), sharey=True)
for i in range(4):
    data = versions_merged[versions_merged.version_name.isin(versions_sorted[32*i:32*(i+1)])]
    ax = sns.countplot(ax=axes[i], x="version_name", 
                       hue="issuetype_name", data=data.sort_values(by="version_name").head(3000),
                      palette={"Bug":"r","Improvement":"orange",
                               "New Feature": "green",
                               "Task": "yellow",
                               "Sub-task": "pink",
                              "Documentation": "grey"})
    ax.tick_params(axis='x', labelrotation= 60)
    if i > 0:
        ax.get_legend().remove()<jupyter_output><empty_output><jupyter_text>**Time distribution of creation**<jupyter_code>issues["created_month"] = issues.created.dt.month
issues["created_year"] = issues.created.dt.year
time_group = ["created_year", "created_month"]
f, axes = plt.subplots(len(issues.issuetype_name.unique()),1, figsize=(15,5), sharex=True)
x = issues.groupby(time_group+["issuetype_name"]).size().unstack(fill_value=0).reset_index()
for i, issuetype in enumerate(issues.issuetype_name.unique()):
    ax = sns.barplot(x=x.index, y=x[issuetype], ax=axes)
    ax.set_xticklabels(list(x[time_group[1]].astype(int).astype(str) + "/" + x[time_group[0]].astype(int).astype(str)),
                      rotation=90)
    ax.set_ylabel(issuetype)<jupyter_output><empty_output><jupyter_text>* New Feature, Improvement have decresed in number over time* Documentation has increased after 1/2019* Bug has been quite consistentTo note that the y-axis is not the same for the different plots. We are interested in seeing their trend overtime and using the same axis would penalize the types with fewer observations.   Resolution<jupyter_code>issues.resolution_name.unique()
ax = sns.countplot(x="resolution_name", data=issues[["issue_id", "resolution_name"]])
ax.tick_params(axis='x', labelrotation= 60)
ax = sns.countplot(issues[~(issues.resolution_name == "Fixed") & ~(issues.resolution_name == "Won't Fix")].resolution_name)
ax.tick_params(axis='x', labelrotation= 60)
issues.resolution_name.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>* 95% of the resolution are Fixed* 0.01% are duplicates (may need to be removed)  Priority<jupyter_code>issues.priority_name.unique()
ax = sns.countplot(x="priority_name", data=issues[["issue_id", "priority_name"]])
issues.priority_name.isna().sum()
issues.priority_name.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Every issue has a priority. With issue with Major priority representing the 93%  Labels<jupyter_code>labels_merge = labels.merge(issues, on="issue_id")<jupyter_output><empty_output><jupyter_text>**Number of labels**<jupyter_code>labels.label = labels.label.str.lower()
len(labels.label.unique())<jupyter_output><empty_output><jupyter_text>**Number of issues with their number of labels**<jupyter_code>sns.countplot(labels_merge.groupby("issue_id").count().label)<jupyter_output>/Users/alexincerti/Library/Python/3.8/lib/python/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>**Top labels**<jupyter_code>vals = labels.label.value_counts().head(20)
sns.barplot(x = vals.values, y=vals.index)<jupyter_output><empty_output><jupyter_text>* Lambda label seems to be the most popular ones by far  FixVersions<jupyter_code>fixversions = fixversions[fixversions.fixVersion_name > "4.4"]
fixversions_merged = fixversions.merge(issues, on="issue_id")
fixversions.fixVersion_name.unique()
len(fixversions.fixVersion_name.unique())
set(fixversions.fixVersion_name.unique()) - set(tags.Tag.apply(polish_version))
set(fixversions.fixVersion_name.unique()) - set(versions.version_name.unique())<jupyter_output><empty_output><jupyter_text>**How many fixversions each issue has?**<jupyter_code>fixversions.groupby("issue_id").fixVersion_id.count().value_counts()
sns.countplot(fixversions.groupby("issue_id").fixVersion_id.count())<jupyter_output>/Users/alexincerti/Library/Python/3.8/lib/python/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>**Percentage of issues with a fixversion**<jupyter_code>len(fixversions_merged.issue_id)/(len(issues.issue_id))*100<jupyter_output><empty_output><jupyter_text>**Top fixversions**<jupyter_code>top_fixversions = fixversions.groupby("fixVersion_name").count().issue_id.reset_index().sort_values(by="issue_id", ascending=False)
ax = sns.barplot(x="fixVersion_name", y="issue_id", data=top_fixversions[["issue_id", "fixVersion_name"]].head(30))
top_fixversions.head(10).fixVersion_name.values<jupyter_output><empty_output><jupyter_text>**Distribution of issuetypes for each version**<jupyter_code>fixversions_sorted = fixversions.sort_values(by="fixVersion_name").fixVersion_name.unique()
f, axes = plt.subplots(2,1, figsize=(15,9), sharey=True)
for i in range(2):
    data = fixversions_merged[fixversions_merged.fixVersion_name.isin(fixversions_sorted[32*i:32*(i+1)])]
    ax = sns.countplot(ax=axes[i], x="fixVersion_name", 
                       hue="issuetype_name", data=data.sort_values(by="fixVersion_name").head(3000),
                      palette={"Bug":"r","Improvement":"orange",
                               "New Feature": "green",
                               "Task": "yellow",
                               "Sub-task": "pink",
                              "Documentation": "grey"})
    ax.tick_params(axis='x', labelrotation= 60)
    if i > 0:
        ax.get_legend().remove()<jupyter_output><empty_output><jupyter_text>**Percentage of issuestype with a version**<jupyter_code>data = []
for issuetype in issues.issuetype_name.unique():
    issuetype_ids = issues[issues.issuetype_name == issuetype].issue_id
    data.append((issuetype, len(set(issuetype_ids).intersection(set(fixversions.issue_id)))\
                 /len(issuetype_ids)))
sns.barplot(x="issuetype", y="percentage_of_issue_with_fixversion", data=pd.DataFrame(data, columns=["issuetype", "percentage_of_issue_with_fixversion"]))
data<jupyter_output><empty_output><jupyter_text>* We have 70 (but 13 not on Github) fix versions we can use for our Travis analysis* 92% of issues have a fixversion* The issuetypes with most issues with a fixversion are Bug and New Feature  No version issuesIdeally all bugs would be labeled with a version. Why is it not the case?<jupyter_code>no_version_issues = issues[~issues.issue_id.isin(versions.issue_id.unique())]
bug_no_version = no_version_issues[no_version_issues.issuetype_name == "Bug"]
bug_no_version.head()
len(bug_no_version)<jupyter_output><empty_output><jupyter_text>**Issues with no version and their resolution type distribution**<jupyter_code>no_version_issues.resolution_name.value_counts(normalize=True)
sns.countplot(no_version_issues.resolution_name)<jupyter_output>/Users/alexincerti/Library/Python/3.8/lib/python/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>**Percentage of issues with no versions over the years**<jupyter_code>x = no_version_issues.created_year.value_counts(normalize=True).sort_index()
sns.barplot(x.index, x.values)<jupyter_output>/Users/alexincerti/Library/Python/3.8/lib/python/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>**Bugs with no version resolution type distribution**<jupyter_code>bug_no_version.resolution_name.value_counts()
bug_no_version.resolution_name.value_counts(normalize=True)
sns.countplot(bug_no_version.resolution_name)<jupyter_output>/Users/alexincerti/Library/Python/3.8/lib/python/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>* 94% of the issues with no version have been fixed* 94% of the issues with no version AND of type Bug have been fixedThis means that for almost 415 bug issues a solution has been found. This can possibly be exploited for identifying in which version the bug was introduced.  Status<jupyter_code>issues.status.unique()
ax = sns.countplot(x="status", data=issues[["issue_id", "status"]])<jupyter_output><empty_output><jupyter_text>**Ratio of issues with version over time**<jupyter_code>versions_merged["created_month"] = versions_merged.created.dt.month
versions_merged["created_year"] = versions_merged.created.dt.year
time_group = ["created_year", "created_month"]
y = versions_merged.groupby(time_group+["issuetype_name"]).size().unstack(fill_value=0).reset_index()
x = no_version_issues.groupby(time_group+["issuetype_name"]).size().unstack(fill_value=0).reset_index()
x["Bug"] = x.Bug / (x.Bug + y.Bug)
no_version_issues.columns
x = x[(x.created_year > 2014) & (x.created_month > 2)]
time_group = ["created_year", "created_month"]
f, axes = plt.subplots(len(issues.issuetype_name.unique()),1, figsize=(15,5), sharex=True)
#x = no_version_issues.groupby(time_group+["issuetype_name"]).size().unstack(fill_value=0).reset_index()
ax = sns.barplot(x=x.index, y=x[issuetype], ax=axes)
ax.set_xticklabels(list(x[time_group[1]].astype(int).astype(str) + "/" + x[time_group[0]].astype(int).astype(str)),
                  rotation=90)
ax.set(ylabel="Bugs with no version / All bugs reported")
plt.savefig(f"{figure_folder}/issues/no_ver_bugs_time.png", transparent=True, bbox_inches='tight', pad_inches=0.2)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12693343067025598, 'fraction_numerical': 0.009581603321622485, 'mean_word_length': 4.882447665056361, 'pattern_counts': {'":': 14, '<': 183, '<?xml version=': 0, '>': 191, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '47', 'dir': 'jupyter-structured-clean-dedup', 'id': '649918', 'license': 'MIT', 'n_tokens_mistral': 7169, 'n_tokens_neox': 6824, 'n_words': 1352, 'path': 'notebooks/analysis/Jira_issue_analysis_community.ipynb', 'repo_name': 'IncioMan/ci_data_issue_prediction', 'size': '21917'}