software engineering researchers to study and improve collaborative editing support in crowd-shared platforms.
Data Availability. The datasets analyzed during this study are available in our online appendix [1]}.
Deviations of this study from our registered report [2]} are discussed in Section .
Structure of the Article.
The rest of this article is structured as follows. Section  represents a catalog of edit rejection reasons. In Section , we discuss a model that predicts rejected edits with potential reasons. Section  discusses our online tool named EditEx, its architecture, and analyzes its effectiveness. Feature ranking, the reasons behind the misclassifications of our model, and the implications of our study are discussed in Section . Section  focuses on the threats to validity, Section  discusses the related work and finally, Section  concludes our study.
A Catalog of Edit Rejection Reasons
SO introduces an edit system to improve the quality of the posts. However, edits may not always be satisfactory and thus can be rejected. Wang et al. [3]} conduct an initial investigation on rejected edits of answers by rollback and expose 12 potential reasons behind rejections. We extend their investigation by manually investigating rejected edits by rollbacks of questions and answers. This section discusses our manual investigation process first and later summarizes the identified edit rejection reasons.
Dataset Preparation
We downloaded the September 2019 (which was the start time of the journey of this study) data dump of SO from the Stack Exchange site [4]}. The data dump stores the history of all events (e.g., rollback and edit body) of the posts. In this study, we only investigate the revisions where users made edits to the body of posts. Our data dump contains a total of 1,116,473 rejected edits (72,159 questions + 44,314 answers) by rollbacks and 26,604,779 accepted edits (13,624,495 questions + 12,980,284 answers). We manually analyze rejected edits to explore the rejection reasons. However, accepted edits will be used later to train and test our machine learning classifiers in Section REF . We randomly sampled a statistically significant sample size from rejected and accepted edits.
To achieve a confidence level of 95% with a confidence interval of 5% [5]}, we randomly sampled – (1) 382 from 72,159 rollback edits of questions, (2) 382 from 44,314 rollback edits of answers, (3) 385 from 13,624,495 accepted edits of questions, and (4) 385 from 12,980,284 accepted edits of answers. We use the following formula to compute the size of our random sample.
\(\frac{Nz^2p(1-p)}{e^2N+z^2p(1-p)}\) 
where N is the population size (e.g., 72,159), z is the Z-score corresponding to a particular confidence level (e.g., 1.96 for a confidence level of 95%), e is the confidence interval (e.g., 5%), and p is population proportion (e.g., 0.5) [3]}.
The data dump only stores the latest contents of the questions and answers after each revision. However, we need both the texts before and after rollback/acceptance to analyze which suggested edits are rejected/accepted. Therefore, we collect the PostId and RevisionGUID for each of our randomly selected revisions from the data dump. We manually save the web pages using PostId from the SO site containing each post's revision history. Next, we find the target revision from the history using the RevisionGUID. Note that RevisionGUID is a unique ID used to find a particular revision. Each revision contains the text before and after rollback/acceptance. Finally, we extract those target texts.
<TABLE>
Edit Rejection Reasons
Table REF  summarizes the rollback edit reasons.
We (two authors of this paper) manually investigate the randomly selected 764 (382 questions + 382 answers) rejected edits by rollbacks. We consider the rollback reasons identified by Wang et al. [3]} as the baseline during our analysis. However, we discuss the rollback edit reasons in multiple interactive sessions. We then analyze 200 rollback edits (100 questions + 100 answers) from our selected dataset and label the reasons. For a given rollback edit, we meticulously analyze the texts before and after rollback to see the edits that cause a rollback. Our in-depth investigation exposes a total of nineteen potential reasons. Twelve of them were identified by Wang et al., and the remaining seven are new reasons. The new reasons are – (1) status update, (2) gratitude add/remove, (3) greetings add/remove, (4) signature add/remove, (5) deprecation note add/remove, (6) duplication note add/remove, and (7) community trust. We then measure the agreement using Cohen's Kappa [8]}, [9]}. The value of \(\kappa \) was \(0.98\) , which means the strength of the agreement is almost perfect. Next, we resolve the remaining few disagreements by discussion. However, the agreement level indicates that any coder can do the rest of the labeling without introducing individual bias. Thus, the first author of this paper analyzes the remaining dataset and manually labels the reasons.

A Model to Predict Potential Rejection of Suggested Edits
The identification of undesired editing that causes rejection is essential to promote quality editing. However, manual differentiation of undesired and accepted edits can waste a lot of time and effort of users. Thus, this study aims to assist SO users by offering them automated support while editing a post. Specifically, we attempt to exploit cues in the textual contents of suggested edits to build a classifier that can automatically determine the rejected edits with potential rejection reasons. In this section, we answer the research question as follows.
[colframe=black!50,colback=white,left=0pt,right=1pt,top=1pt,bottom=1pt, boxrule=1pt,arc=1pt, width=4.7in, center]
\(RQ_1\) ) To what extent do our classifiers predict the rejected edits with the potential reasons?
For \(RQ_1\) , we define the following null hypothesis.
[colframe=black!50,colback=white,left=0pt,right=1pt,top=1pt,bottom=1pt, boxrule=1pt,arc=1pt, width=4.7in, center]
\(H_1\) : The accuracy of our developed classifiers is not better than a random classifier with 50% accuracy.
<FIGURE>Fig. REF  shows the workflow of how we can detect the potential rejected edits of SO with reasons. Our prediction pipeline includes three main components: Feature Extractor, Rejected Edit Predictor and Rejection Reason Classifier. First, the feature extractor takes the body text of the original post, the post with suggested edits, and the user's information (e.g., reputation) as inputs. Then, it produces a feature vector based on the predictor variables. Second, the rejected edit predictor takes the feature vectors as input and outputs a dichotomous variable, rejected. The value of rejected is 1 if the predictor determines that the suggested edit will most likely be rejected, and it is 0 otherwise. Third, the rejection reason classifier takes the corresponding feature vector and texts as inputs if the value of rejected is 1. Then, it outputs the potential reasons for rejection. Finally, we measure the performances of the rejected edit predictor and rejection reason classifier.
<TABLE>Feature Extractor
Table REF  summarizes the features employed to predict whether suggested edits will be rejected or not. We extracted fifteen texts & user-based features to predict the potential rejected edits. Each feature is connected to one or multiple reasons behind a rejected edit.
Note that we discarded emotion, which was included in our registered report [2]}, and added reputation. The reasons behind such decisions are discussed in Section .
However, this section discusses how we extract the features as follows.
Text/Code Formatting.
Text or code formatting refers to the changes in their presentation styles. For example, consider the text with HTML tags – \(<\) p\(>\)  I am using \(<\) b\(>\)  C# \(<\) /b\(>\)  programming language \(<\) /p\(>\) . Here, C# is formatted as bold. Someone can reject the bold format of C# by removing the \(<\) b\(>\) ...\(<\) /b\(>\)  tags. However, in both cases, the content remains unchanged. The extracted content will be “I am using C# programming language”. We thus detect the text/code formatting in the following ways.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Determine Text Formatting, boxrule=0.0pt, fonttitle=]
\(textFormat =\)  Tbr != null \(\&\&\)  Tar != null
     \(\&\&\)  De(Tbr, Tar) \(==\)  0
          \(\&\&\)  De (Tbrwt,Tarwt) \(!=\)  0
where Tbrwt: Texts before rollback with HTML tags (e.g., \(<\) p\(>\)  I am using \(<\) b\(>\)  C# \(<\) /b\(>\)  programming language \(<\) /p\(>\) ),
Tbr: Texts before rollback (e.g., I am using C# programming language),
Tarwt: Texts after rollback with HTML tags (e.g., \(<\) p\(>\)  I am using C# programming language \(<\) /p\(>\) ),
Tar: Texts after rollback (e.g., I am using C# programming language), and
De: Levenshtein [11]}, [12]} editing distance. However, we remove new lines, strip leading & trailing spaces from Tbr & Tar, and make them lowercase before processing.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Determine Code Formatting, boxrule=0.0pt, fonttitle=]
\(codeFormat =\)  Cbr != null \(\&\&\)  Car != null
     \(\&\&\)  De(Cbr, Car) \(==\)  0
          \(\&\&\)  De (Cbrwt,Carwt) \(!=\)  0
where Cbrwt: Code before rollback with HTML tags,
Cbr: Code before rollback,
Carwt: Code after rollback with HTML tags, and
Tar: Code after rollback. Like text formatting, we remove new lines, strip leading & trailing spaces from Cbr & Car, and make them lowercase before processing.
Text/Code Modification.
Text/code modification refers to text/code addition/removal or change of existing text/code.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Determine Text Modification, boxrule=0.0pt, fonttitle=]
if (Tbr != null \(\&\&\)  Tar != null \(\&\&\)  De(Tbr, Tar) \(!=\)  0)\(\lbrace \)
     \(textModification =\)  De(Tbr, Tar)
\(\rbrace \) 
else
     \(textModification = False\)
We normalize textModification with respect to the character length of Tbr since Tbr varies.
We determine the code modification in the same way as text modification.
Deface Post.
Remove the texts or code entirely.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Deface Text, boxrule=0.0pt, fonttitle=]
\(defaceText =\)  (Tbr != null \(\&\&\)  Tar == null) \(||\)  (Tar != null \(\&\&\)  Tbr == null)
We determine the deface code in the same way as the deface text. However, deface post will be True if defaceText or defaceCode becomes True.
Complete Change of Post.
Users change texts or code segments entirely.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Determine Complete Change of Text, boxrule=0.0pt, fonttitle=]
\(completeChangeText =\)  Tbr != null \(\&\&\)  Tar != null
     \(\&\&\)  (De(Tbr, Tar) \(==\)  Len(Tbr) \(||\)  De(Tbr, Tar) \(==\)  Len(Tar))
We determine the complete code change in the same way as the text change. However, complete change of post will be True if completeChangeText or completeChangeCode becomes True.
Status.
To detect the addition/deletion of status, we attempt to find a keyword match to Tbr/Tar from a keywords list, Lkw. Here, Lkw = \(\lbrace \)edit, update, note, ps\(\rbrace \)  [13]}.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Determine Status, boxrule=0.0pt, fonttitle=]
status = False
for (keyword in Lkw)\(\lbrace \)
     if ((Match(keyword, Tbr) == \(True\)  \(\&\&\)  Match(keyword, Tar) == \(False\) ) \(||\)
     (Match(keyword, Tar) == \(True\)  \(\&\&\)  Match(keyword, Tbr) == \(False\) ))\(\lbrace \)
          status = True
          return status
     \(\rbrace \)
\(\rbrace \)
return status
Gratitude.
We detect the addition/removal of gratitude similarly to status. However, here the keyword list, Lkw = \(\lbrace \)welcome, thanks, sorry, appreciated, thank, ty (i.e., thank you), thx, regards, tia (i.e., thanks in advance)\(\rbrace \)  [13]}.
Greeting.
We detect the addition/removal of greeting similarly to status. However, here the keyword list, Lkw = \(\lbrace \)hi, hello, hey, dear, greetings, hai, guys, hii, howdy, hiya, hay, heya, hola, hihi, salutations\(\rbrace \)  [15]}.
Reference Modification.
To detect reference modification, we extract the values of the href attribute of \(<\) a\(>\)  tag from Tbrwt & Tarwt.
Then insert the hyperlinks into two lists – LSTbr: list of hyperlinks found from texts before rollback, and LSTar: list of hyperlinks found from texts after rollback.
[colframe=black!10, colback=black!2, left=1pt,right=1pt,top=1pt,bottom=1pt,arc=0mm, title = Determine Reference Modification, boxrule=0.0pt, fonttitle=]
referenceModification = False
for (reference in LSTbr)\(\lbrace \)
     if (Contain(reference, LSTar) == \(False\) )\(\lbrace \)
          referenceModification = True
          return referenceModification
     \(\rbrace \)
\(\rbrace \)
for (reference in LSTar)\(\lbrace \)
     if (Contain(reference, LSTbr) == \(False\) )\(\lbrace \)
          referenceModification = True
          return referenceModification
     \(\rbrace \)
\(\rbrace \)
return status
Inactive Hyperlink.
To detect the inactive (e.g., broken/dead) hyperlink, we check the HTTP response of each of the hyperlinks of LSTar. We decide whether a hyperlink is inactive or not based on the response code.
Signature.
To detect the addition/removal of the signature, we extract and store the full name, first part, and last part (if any) of two users (who suggested edit & who rolled back) into a list, Lname. We then detect the addition/removal of the signature similarly to status.
Deprecation Note.
We detect the addition/removal of deprecation notes similarly to status. However, here the keyword list, Lkw = \(\lbrace \)deprecation, deprecate, old code\(\rbrace \)  [13]}.
Duplication Note.
We detect the addition/removal of duplication notes similarly to status. However, here the keyword list, Lkw = \(\lbrace \)duplicate, duplication\(\rbrace \)  [13]}.
Reputation Score.
We compute the reputation score of users to estimate – (1) how much the community trusts them and (2) whether they follow the editing guidelines.
The official data dump of SO only reports the latest reputation scores of the users, which are not appropriate for our analysis. We thus use the snapshot of user activities (e.g., votes, acceptances, bounties) to compute the reputation score of users during their editing of posts. In particular, we use a standard equation provided by the SO to calculate the reputation score [18]}.

Rejected Edit Predictor
In this section, we first describe machine learning classifiers (Section REF ) and their evaluation setup (Section REF ). Then we evaluate the performance of our classifiers in identifying rejected edits (Section REF ). Finally, we construct the baseline models and report their performance in Section REF .
Machine Learning Models
The relationship between edit categories (rejected/accepted) and their corresponding feature values might be complex. Thus, we choose the following four popular machine learning classification techniques with different learning strategies to identify the potential rejected edits. They are widely used in the relevant studies [19]}, [20]}, [21]}, [22]}.
Decision Trees (DT) is a non-parametric supervised machine learning technique for classification and regression. Non-parametric means it does not make any assumptions about the underlying data distribution. The intuition behind decision trees is that simple decision rules are inferred from the dataset features and continually split the training set until all data points belonging to each class are isolated. In particular, this technique employs different heuristics (e.g., entropy, information gain) to decide which feature to be used for the subsequent split of the training set. The commonly used decision trees are ID3, C4.5, and CART. However, ID3 can only be used when features are categorical. C4.5 and CART are extensions of ID3, which can work with features of both categorical and continuous data. Here, we use CART since our extracted features have both continuous and categorical values.
Random Forest (RF)
is a supervised machine learning technique. The `forest' it builds is an ensemble of many decision trees, usually trained with the `bagging' method. The underlying principle behind the ensemble model is that a group of weak learners come together to form a strong learner. Ensemble learners thus improve the performance of single classifiers by inducing several classifiers and combining them to obtain a new classifier that outperforms every one of them [23]}. RF is scalable to any number of dimensions and usually has acceptable performance. However, it adds additional randomness to the model while growing the trees. For example, instead of searching for the most important feature while splitting nodes, it searches for the best feature among a random subset of features. RF thus prevents the overfitting of datasets by creating random subsets of the features.
K-Nearest Neighbors(KNN)
is a non-parametric method employed in classification and regression problems [24]}. It does not use the training data points to perform any generalization. Thus, the KNN's training phase is much faster than other classification algorithms. In KNN, K represents the number of nearest neighbors, which is the core factor in deciding a data point's label (i.e., class). This technique finds the K closest neighbors of a target point using distance measures (e.g., Euclidean distance). Then, each neighbor votes for their class, and the class with the most votes is taken as the prediction.
eXtreme Gradient Boosting (XGBoost) is a scalable tree boosting technique that predicts a target variable by combining an ensemble of estimates from a set of more simplistic and weaker models [25]}. It is a supervised learning algorithm and can be employed for both classification and regression. XGBoost is an extension to gradient boosted decision trees (GBM) with improved speed and performance. However, it is faster than other algorithms because of its parallel and distributed computing. XGBoost performs well because of its robust handling of various data types, relationships, distributions, and a variety of hyperparameters. In addition, XGBoost has inbuilt cross-validation and a variety of regularizations, which helps reduce overfitting.

Model Evaluation Setup
Dataset Selection. We used the randomly sampled dataset (rejected & accepted) from Section REF  to train and test our machine learning classifiers. However, we keep the training and testing data separate since we have one time-dependent feature (i.e., reputation score). For example, consider a user suggested edits to multiple posts at different times. The reputation score of users increases over time. Thus, the reputation score of that user will be lower while suggesting prior edits than the later edits. Therefore, we use earlier edits as the training set and later edits as the test set to ensure that past data is not predicted based on future data. In particular, we take 70% samples that were edited relatively earlier to train the machine learning classifiers and use the remaining 30% to test them.
<TABLE>Performance Metrics. The selection of the evaluation criteria is vital to guarantee a reliable assessment of the prediction models. In a binary classification problem like rejected edit prediction, a confusion matrix (e.g., Table REF ) records the correctly and incorrectly recognized examples of each class. Therefore, we can obtain several metrics from the given confusion matrix to independently evaluate models' performance for both positive and negative classes.
The machine learning community often measures the classification accuracy as a simple scalar performance metric for binary classification. Classification accuracy measures the ratio of correctly classified edits into rejected & accepted classes with respect to all classified edits. However, according to [26]}, accuracy might lead to an incorrect conclusion as the measure is highly sensitive to changes in data. In such cases, precision is a useful metric to capture the effect on a classifier performance of having a larger number of negative examples [27]}. In particular, precision measures the ratio of correctly classified edits into a class (i.e., rejected/accepted) with respect to all edits classified into that class. However, [26]} argued that precision is still sensitive to changes in the data distribution, and it cannot assert how many positive examples are classified incorrectly.
Unlike precision, recall is not sensitive to data distribution that measures the ratio of correctly classified edits with respect to the actually observed edits as true instances. However, any assessment based solely on recall would be inadequate, as it provides no insight into how many examples are incorrectly classified as positives.
Therefore, neither precision nor recall can provide a reliable assessment of classification performance [29]}. However, these individual scalar metrics can be combined to build more reliable classification performance measures. Specifically, these aggregated performance metrics include the F-measure that represents the harmonic mean of precision and recall. We thus measure precision, recall, F1-score, and overall accuracy to conclude the models' performance better. They can be measured as follows.
     Precision \(= \frac{TP}{TP+FP}\)                 Recall \(= \frac{TP}{TP+FN}\)
     F1-Score \(\frac{2 \times Precision \times Recall}{Precision + Recall}\)         Accuracy \(= \frac{TP+TN}{TP+FP+TN+FN}\)
<FIGURE>
Model Performance Evaluation
We experiment with our models to see how well the classification models perform based on our features. Fig. REF  summarizes the performance of our models. Our primary focus is to predict the rejected edits. We see that our models can predict the rejected edits with 62.3%–69.1% precision. The random forest can predict the rejected edits more precisely than the other three models. Its precision is 69.1%.
On the other hand, XGBoost shows the highest recall (i.e., 71.8%). However, the highest F1-score is achieved by the random forest model in predicting rejected edits. The precision to predict the accepted edits ranges from 59.1%–70.5%. Like rejected edits, the random forest model achieves the highest precision in predicting accepted edits. On the contrary, the k-nearest neighbors shows the lowest precision.
From Fig. REF , we see that the overall accuracy of the models is more than 60%. The highest accuracy is about 70%. Our experimental result shows that the random forest performs best, whereas the k-nearest neighbors shows the lowest performance. The k-nearest neighbors could suffer from the high dimensional data. However, according to the experimental result, XGBoost slightly outperforms decision trees but underperforms the random forest. We thus further investigate why XGBoost does not outperform random forest. We analyze the predicted class of XGBoost and random forest models against our test dataset. We find that XGBoost misclassified 31 samples (15 accepted + 16 rejected), which random forests classified correctly.
In our dataset, addition/removal of duplication notes get rejected more than 94% times, and signatures get rejected 100% times. However, a few samples are classified as accepted by XGBoost even after the addition/removal of duplication notes or signatures. On the contrary, random forest classified them correctly as rejected. Besides, XGBoost classified several samples with trivial text changes as rejected, which were incorrect. Our analysis shows that trivial changes have more chance of being accepted than rejected. However, random forest classified the target class correctly in those cases. Hence, the above reasons could explain why XGBoost slightly underperforms random forest. Therefore, we select the random forest model to deploy with our online tool.
<FIGURE>Our models could capture unnecessary details or too specific relationships within the training dataset and thus suffer from overfitting. However, overfit models are not very stable since they fail to generalize well to the data. Moreover, overfit models generally perform poorly on unseen (e.g., test) data. In particular, a overfit model has a substantial difference between the accuracy of the training and test dataset. That is why we can never trust an overfit model and put it into deployment. We thus attempt to reduce overfitting before deploying our model. In particular, we tuned the critical parameters of the model to balance the accuracy between training and test datasets. Then, we set the values of the parameters to avoid model overfitting.
<TABLE>Fig. REF  shows the model accuracies on the training and test dataset in contrast to example critical parameters of the model. For example, we run the random forest model over the depth of the tree from 1 to 20. As shown in Fig. REF , the training accuracy improves with the depth of the tree. However, the difference between training and test accuracies increases when the depth exceeds five. We thus set the depth value as five while training the random forest model. We set the depth values for the decision trees and XGBoost models similarly. Their depth values are five and three. In particular, we attempt to balance the training and test accuracies when determining depth values.
On the other hand, Fig. REF  shows the training and test accuracies of the k-nearest neighbors in contrast to the number of neighbors. We vary the number of neighbors from 1 to 50, evaluate the model on the train and test datasets for each number of neighbors, and report the accuracy. We see that performance on the test set improves initially and then worsens, and performance on the training set continues to degrade. As shown in Fig. REF , the training accuracy is dropping to converge with the line for the test set. However, the training and testing accuracies are very close when the number of neighbors is 46. We thus choose the parameter value is 46 (i.e., number of neighbors \(= 46\) ).
<FIGURE><FIGURE>
Performance of Baseline Model
To the best of our knowledge, there was no existing machine learning model to identify the rejected edits with reasons at the time when we conducted this study. However, we construct baseline models that reject/accept trivial edits and evaluate their performance. We thus first attempt to categorize the types of edits conducted in the samples of our dataset. We calculate the Levenshtein distance between the original post's content (text + code) and the post with suggested edits. We normalize the distance with respect to the character length of the original posts. Then, we classify the edits into four categories based on the edit distance. They are – trivial (distance \(\le \)  lower quartile), small (lower quartile \(>\)  distance \(\le \)  median), medium (median \(>\)  distance \(\le \)  upper quartile) and major (distance \(>\)  upper quartile) edit.
Fig. REF  shows the percentage and count of rejected and accepted edits for each category. We see that 62.8% of trivial edits get accepted (Fig. REF ) in our dataset. On the contrary, such a statistic is only 37.2% for rejected edits. However, the percentage of rejected edits is higher than the accepted edits for the remaining three categories. We then develop a rule-based classifier that rejects trivial edits. Fig. REF  summarizes the performance of the classifier. The precision of identifying rejected edits is below 40%. The classifier also has poor recall (i.e., 19.1%) in identifying rejected edits with 43.7% overall accuracy. Next, we build a classifier that rejects non-trivial (i.e., small, medium & major) edits. That is, it accepts trivial edits. Fig. REF  shows its performance. This classifier shows a higher recall in identifying rejected edits than our proposed models. However, the precision, F1-score, and overall accuracy are significantly lower than our best-performing model. Such performances suggest that rejected edits cannot be identified reasonably well only based-on edit categories.
<FIGURE>
Rejection Reason Classifier
In the previous section, we evaluate the performance of the machine learning models in predicting the rejected edits. This section analyzes how accurately the rejection reason classifier can identify the potential reasons for those rejections.
Our rejected reason classifier can almost accurately identify several rejection reasons by applying the same approach as we extracted features REF . For example, our manual investigation identified nine keywords (e.g., thanks, welcome) (Section REF ) that were utilized to identify gratitude. However, our further analysis finds that the addition or removal of gratitude is rejected 85.5% times (e.g., Fig. REF ) in our dataset. We thus used this lightweight keyword-based technique to identify the reason “gratitude add/remove” when the edits are predicted as rejected. However, to avoid multiple computations, we look back at the feature vector to check the feature value of gratitude.
Similarly, we identify the following potential edits rejection reasons by analyzing the feature vector.
[colframe=black!50, colback=white,left=1pt,right=1pt,top=1pt,bottom=1pt, arc=1pt]
Undesired text/code formatting, complete change of post, deface post, gratitude add/remove, greetings add/remove, signature add/remove, deprecation note add/remove, duplication note add/remove, undesired reference modification
<TABLE>For a remaining few rejection reasons, such as undesired text/code addition/removal, we primarily apply n-grams and pos-tagging-based techniques. Unfortunately, we did not find satisfactory performance (e.g., accuracy \(<\)  50%).
Our further investigation suggests that both desired and undesired texts contain similar words, phrases, or patterns. Therefore, we cannot distinguish them using words, phrases, or pos-based patterns. However, the character length of added or removed text/code can identify the undesired addition or removal of text/code reasonably well. We thus extract the added/removed text or code from our manually analyzed dataset using the appropriate HTML tags. In particular, we extract the contents of HTML elements with a class attribute with either diff-add & diff-delete values. We measure the length of added/removed characters of text or code. We then normalize the length with respect to the total length of the text/code of revisions. Next, we separate samples into two classes according to our manual label – (1) undesired vs. desired text addition, (2) undesired vs. desired text removal, (3) undesired vs. desired code addition, and (4) undesired vs. desired code removal. We developed four random forest classifiers to identify the undesired text/code addition/removal. However, we resolve the class imbalance problem using Synthetic Minority Oversampling Technique (SMOTE) [30]}.
Table REF  shows the performance of the classifiers. The precision of identifying the undesired text/code addition/removal is about 62.1%–69.2%. The recall is slightly lower for undesired text addition/removal than undesired code addition/removal. However, the overall accuracy is more than 63%, except for undesired text removal.
We then evaluate the overall performance of the rejection reason classifier in identifying the potential rejection reasons. We experiment with the rejection reason classifier using our test dataset. In particular, we label each sample in a file as follows.
<TABLE>(1) Identified: the reasons detected by the rejection reason classifier.
(2) Expected: the actual reasons based on our manual analysis.
We then create a confusion matrix to analyze the performance of the rejection reason classifier as follows.
(i) True Positive (TP) = `identified' reasons = `expected' reasons
(ii) False Positive (FP) = (`identified' reasons \(\ne \)  `expected' reasons) or (`identified' reasons but `expected' no reasons)
(iii) True Negative (TN) = `identified' no reasons and `expected' no reasons, and
(iv) False Negative (FN) = `identified' no reasons but `expected' one or more reasons.
Using the above matrix, we compute four standard metrics
(Precision, Recall, F1-score, and Accuracy) to compute the performance of the model to identify the rejected reasons.
Table REF  shows the confusion matrix and performance of rejection reason classifier to identify the potential reasons for edit rejections. Our analysis shows that our rejection reason classifier can identify the potential rejection reasons with 62.3% precision, 67.2% recall, 64.7% F1-score, and 66.7% overall accuracy.
We further analyze in which cases our rejection reason classifier fails to identify rejection reasons. We find that our model mainly fails to identify all the reasons when there were multiple reasons to reject suggested edits. In that cases, our model can partly identify the reasons. For example, there are three reasons for rejection. However, our model identifies two of them accurately.
Note that we identify the potential rejection reason as community mistrust when – (1) our rejection reason classifier cannot identify any reason and (2) the reputation score of the user who suggested edits is below 2K. We choose a reputation score \(< 2K\)  since users with a reputation score \(< 2K\)  have no privilege to edit posts instantly.
However, our rejection reason classifier cannot identify a couple of reasons (e.g., incorrect text/code change). We will discuss them in Section .
[colframe=black!50,colback=white,left=0pt,right=1pt,top=1pt,bottom=1pt, boxrule=1pt,arc=1pt, width=4.7in, center]
Summary of \(RQ_1\)  and hypothesis testing: We developed four machine learning classifiers based on 15 texts & user-based features to predict rejected edits. Our best-performing classifier (i.e., random forest) can predict the rejected edits with 69.1% precision, 71.2% recall, 70.1% F1-score, and 69.8% overall accuracy. We then develop a classifier to identify the potential reasons behind rejections. Our rejected reason classifier can identify the potential rejection reasons with overall 66.7% accuracy.
Such findings reject our null hypothesis (i.e., \(H_1\)), which hypothesized that the accuracy of our developed classifiers is not better than 50%.

EditEx: A Recommender for Early Fixes to Suggested Edits
We can assess the actual impact of our developed classifier if it can automatically assist users during their editing of SO posts. Editing is a time-consuming and largely voluntary activity in SO. Therefore, users can be assisted with a tool that can recommend potential fixes to their undesired edits. We thus focus on introducing an online tool, namely EditEx, that interacts with our classifier, identifies the potential rejection reasons from the suggested edits, and helps to reduce the likelihood of rejection of suggested edits. We then assess the tool's effectiveness via real-world usage by users and answer the research question as follows.
[colframe=black!70,colback=white,left=0pt,right=1pt,top=1pt,bottom=1pt, boxrule=1pt,arc=1pt, width=4.7in, center]
\(RQ_2\) ) To what extent can EditEx be helpful to users avoid rejection in their edits?
For \(RQ_2\) , we define the following null hypothesis.
[colframe=black!70,colback=white,left=0pt,right=1pt,top=1pt,bottom=1pt, boxrule=1pt,arc=1pt, width=4.7in, center]
\(H_2\) : Our EditEx tool is not effective in helping users avoid rejected edits.
EditEx Architecture
Fig. REF  shows an overview of the EditEx architecture. EditEx has two parts: client and server. On the client-side, users get the EditEx interface. EditEx interface comprises two buttons: EditEx and Suggestion. EditEx enables users to edit the SO posts. On the other hand, users can check whether the edits will be rejected/accepted and the potential rejection reasons upon rejection by clicking the Suggestion button.
<FIGURE>When users click the Suggestion button, the client-side script captures the necessary data required to extract the features for the machine learning model. In particular, it captures – (1) text before edit, (2) text after edit, (3) reputation, and (4) the name of the user who suggests edits. Then it sends this data to the server-side application. Client-side script is written by JavaScript, and server-side application is developed by Java. However, the server-side application captures all the feature values (Section REF ) using the data sent by the client. The feature values form a feature vector. The feature vector is then passed to the classification model to predict whether the edit will be rejected or not.
The potential rejection reasons are identified upon rejection based on the feature vector and texts (before and after edit). Then, the decision of the edit (i.e., rejected/accepted) and the helpful suggestions that notify the rejection reasons (if rejected) are sent to the client-side script. Finally, the client-side script offers users the result and suggestions.
<FIGURE>
Effectiveness Evaluation Plan of EditEx
Fig. REF  shows the overview of how EditEx is introduced and its effectiveness is evaluated. Users can get the EditEx interface integrated with the SO's existing edit system after installing EditEx. Then, they can suggest edits to posts using EditEx to improve their quality. We measure the effectiveness of EditEx in two ways.
First, we recruited 20 participants and divided them into two groups. The control group edits posts using the existing SO edit system of SO, whereas the treatment group uses EditEx to edit posts. When both groups complete their edits, we compare the success rates (e.g., fewer rejection ratios). Second, we survey participants after completing their edits and analyze their feedback.

Study Design
In this section, we first discuss how we recruit our study participants (Section REF ). We then explain the formation of control vs. treatment groups (Section REF ). Next, we discuss the two phases of our study – task-based evaluation and survey-based feedback collection phase (Section REF ).
Study Participants
We recruit 20 participants who satisfy our constraint (i.e., participants must have editing experience in SO posts) in the following two ways.

Snowball Approach:
We use convenience sampling to bootstrap the snowball [31]}. We first contacted a few software developers who are known to us, easily reachable, and working in software companies worldwide. We discuss our study goals and share the survey with them. We then adopt a snowballing method [32]} to disseminate the survey to some of their colleagues with similar experiences. We asked to share the survey with those who could be interested in editing posts and participating in our survey. In this process, we receive information (e.g., email) from 22 participants who show their interest in our study. However, 12 of them finally confirmed their participation.


Open Circular:
We post a description of this study and our research goals in the specialized Facebook groups to find potential participants. We target the groups where professional software developers discuss their programming problems. We also use LinkedIn as a research tool to reach potential participants because it is one of the largest professional social networks in the world. We get contacts of 20 participants from this open circular who are willing to participate and satisfy our constraints (e.g., must have editing experience). However, some participants did not respond to us when we contacted them. We finally confirmed eight participants in this process.

In the end, we were able to recruit a total of 20 developers (12 from snowball + 8 from open circular) who were eligible based on our study constraint. Half of them have 3–5 years, 40% (8 out of 20) have two years (or less), and 10% (2 out of 20) have 6–8 years of software development experience. Note that 35% of participants were from different software industries, and the remaining 65% of them were from academia. They worked as developers, technical leads, grad students, and faculty members worldwide (e.g., Canada, Germany, Bangladesh).
In our registered report [2]}, we proposed to recruit 30 participants using a snowball approach. However, we discussed these discrepancies in the number of participants and recruitment approach in Section .

Formation of Control and Treatment Groups
As mentioned above (Section REF ), we recruit 20 participants for our study. We pick 10 of them for the treatment group and 10 for the control group as follows.
<TABLE>
Treatment Group: Each participant in this group was assisted in their editing of SO posts by our developed EditEx tool. However, the participant could also access the standard SO edit system.


Control Group: Each participant in this group edited SO posts by using the standard SO edit system only.

Table REF  shows the experience and professions of control and treatment groups. We attempt to equate profession and experience between two groups to minimize the subjective biases. For example, control and treatment groups have participants with low and high software development experience. Also, each group contains participants from academia (e.g., faculty members) and software industries.

Execution Plan
<TABLE>First, we conduct a task-based analysis (Section REF ) by asking each participant (control + treatment group) to edit ten posts. However, we set a task list and asked each participant to suggest edits to posts following the task list. Table REF  lists the tasks. Tasks T1–T6 were set based on the identified reasons that might cause rejections. For example, the treatment and control groups were asked to add gratitude (e.g., thank you) associated with other edits. In this case, EditEx warns the treatment group against adding such gratitude. Therefore, participants of the treatment group proceed with the remaining edits and avoid adding gratitude.
On the other hand, the control group did not receive such a warning. Thus, they suggested edits with gratitude. We attempt to see the effectiveness of EditEx in preventing the commonly rejected reasons while suggesting edits from tasks T1–T6. However, the control group might get more rejections for tasks T1–T6. On the contrary, such tasks favor the treatment group. Such a scenario can exaggerate EditEx's effectiveness in reducing edit rejections. Therefore, we also ask the participants to edit posts arbitrarily (e.g., Table REF , T7). That means participants suggest edits to posts that are not related to any rejection reasons to limit the bias of this study.
We circulated the editing guidelines of SO to each participant of the control and treatment groups. We asked participants to follow the guidelines in suggesting edits. After suggesting edits, participants wait until they get the decision (rejected/accepted) on those suggested edits from the edit reviewers. However, users with a reputation score \(\ge 2K\)  can edit posts instantly. Those edits are neither added to the review queue nor reviewed by experts. However, a rollback can reject their edits. Unfortunately, such a rollback even may take a few months. We thus ask participants with a reputation score \(\ge 2K\)  to create a new account. Edits suggested by their new account undergo an expert review. This decision also confirms the same privilege level of all the participants. However, our target was to quickly get decisions (rejected/accepted) on the suggested edits and avoid undecided edits.
Second, we conduct an online survey (Section REF ) to listen to the participants about their experience in suggesting edits to SO posts with/without our tool EditEx.
Kitchenham and Pfleeger [34]} suggest considering six main steps for a personal opinion survey. They are – setting survey objectives, designing the survey, developing the survey instrument (i.e., the questionnaire), evaluating the survey instrument, and obtaining and analyzing data. We primarily follow their guidelines to survey participants. However, we also consider the guidance and ethical issues from the established best practices [35]}, [36]}. For example, we take participants' consent before starting the survey. Besides, we confirm to participants that their provided information must be treated confidentially. Our survey includes different types of questions (e.g., multiple-choice, free-text answers). However, we inform about the estimated time (i.e., approximately 10 minutes) required to complete the survey to the participants. Our survey comprises the parts as follows.

Consent and Prerequisite. This part confirms participants' consent to participate in this survey and agreement to process their data.


Participants' Information. In this part, we collect participants' information such as experience, current profession, organization, country, and editing experience in SO posts.


Workload Assessment. This section assesses the cognitive workload of the control and treatment groups in suggesting edits to SO posts. We leverage the NASA Task Load Index (TLX) (non-weighted) to estimate subjective workload [37]}, [38]}, [39]}, [40]}. In particular, we assess how much effort participants had to exert mentally and physically to use EditEx and the standard edit system of SO. Participants were asked to rate their scores on an interval scale ranging from low (1) to high (10) [41]} in the following six dimensions – (1) mental demand, (2) physical demand, (3) temporal demand, (4) effort, (5) performance, and (6) frustration.


Usefulness Analysis. In this section, we measure the participants' confidence in suggesting edits using EditEx (treatment group) and SO edit system (control group). In addition, we ask the treatment group to rate the usefulness of the suggestions of EditEx. In particular, we ask the following two questions to the treatment group participants and employ a 5-point Likert scale (i.e., 1–5) to estimate their consent [42]}, [43]}.


How useful did you find the suggestions from EditEx? (5-point Likert scale)

How confident were you to follow the EditEx suggestions? (5-point Likert)

We ask the following question to measure the confidence level of the participants in the control group.

How confident are you to edit posts using the SO editing system? (5-point Likert scale)



Suggestions to Improve EditEx. Finally, we seek participants' recommendations to improve the effectiveness of EditEx. We ask them the question as follows.


What are your recommendations to further improve EditEx? (Text)


We added the survey form and its responses in anonymized CSV form in our replication package [1]}.

Results from the User Study on Editing of SO Posts
Participants were asked to complete ten edits. However, several participants (especially from the control group) could not edit ten posts due to three main challenges.

SO's edit queue often remains full, and thus participants could not edit posts according to their schedule.

Participants could not edit many posts (e.g., more than three) simultaneously. SO restricts them from suggesting further edits before receiving a decision (rejected/accepted) on the pending ones.

SO does not allow its users to edit for a period when consecutive edits are being rejected.

<FIGURE>Fig. REF  shows the task completion ratio of control and treatment groups. As mentioned above, we asked each group to edit 100 posts (10 for each). However, participants from the control group were able to edit 83 posts in total (i.e., completion ratio 83%), whereas the treatment group edited 94 posts.
<FIGURE>We then attempt to see the rejection ratio of suggested edits. First, we collect information from each participant (treatment & control group) when all of their suggested edits got decisions (e.g., rejected/accepted). In particular, we asked how many edits they suggested and how many of them got rejected. We also collect their editing details to examine whether they follow the given task list (e.g., Table REF ) or not. Then we count the total number of suggested and rejected edits of the treatment and control groups. Finally, we calculate the rejection ratio of each group.
As shown in Fig. REF , the rejection ratio of the edits suggested by the treatment group is only 16% (15 out of 94). On the contrary, such a statistic is 65.1% (54 out of 83) for the control group. Overall, the percentage of rejected edits who used EditEx is about 49% lower than those who used the standard editing system of SO. Such a finding gives us a preliminary validation that our tool helps users to prevent their suggested edits from being rejected.
<FIGURE>Tasks T1–T6 were set based on our identified rejection reasons. However, EditEx alerts the treatment group participants while suggesting T1–T6. Such alerts might help them to avoid edits that cause rejections. On the contrary, the control group did not receive any alerts from the SO edit system while suggesting T1–T6. Therefore, the overall edit rejection ratio of the control group is much higher than the treatment group. We thus attempt to compare the rejection ratio of task T7 (i.e., free-form editing) between control and treatment groups. According to our analysis, the control group suggests 23 free-form edits. Among them, six were rejected. Since EditEx prevents T1-T6, the treatment group could not suggest T1-T6. Therefore, we consider all of their tasks as T7. However, we found that they suggested two edits out of task T7 (i.e., T1–T6). Among the remaining 92, 13 edits were rejected. As shown in Fig. REF , the rejection ratio of the control group for T7 is 26.1%. On the other hand, such ratio for the treatment group is 14.1%.
Therefore, EditEx not only assists users in avoiding edits that are usually rejected but also assists them in conducting regular (i.e., free-form) edits.

Results from Survey of User Study Participants
We received 20 valid survey responses (10 treatment + 10 control). We report the survey responses as follows.

Workload assessment during the completion of the editing tasks,

Usefulness ratings of EditEx suggestions, and

Improvement suggestions by the study participants for EditEx.

Workload Assessment During Edit Task Completion
Fig. REF  shows the box plots of the NASA TLX cognitive workload score on a scale of ten. We compute the average workload of each participant. We first sum up the ratings of each of the six dimensions (e.g., mental demand) and then divide it by the number of dimensions. In particular, we use the equation to compute the average workload of each participant as follows.
\(A_{wl} = \frac{1}{D_T}\Bigg [\sum _{i=1}^{D_{T}} R_{i}\Bigg ]\) 
where \(R_i\)  denotes the rating (1–10) of ith dimension, \(D_T\)  represents the total dimensions (here, \(D_T = 6\) ).
<FIGURE>Fig. REF  shows the box plots that represent the average cognitive workload of each participant from the treatment and control groups. We see that the median subjective workload for the treatment group is about half that of the control group. That is, EditEx strongly supports the SO standard editing system to reduce the users' workload required to suggest edits to posts.
We then attempt to find whether the workload difference between treatment and control groups is statistically significant. We use the Mann-Whitney-Wilcoxon statistical significance test [45]} and find a statistical significance p-value (p-value \(\simeq \)  \(0.0\)  \(<\)  \(0.05\) ). We also use Cliff's delta test [46]} to determine the effect size and find a large effect size (Cliff's d \(= -0.97\)  (large)) with \(95\%\)  confidence. Given this evidence, EditEx helps the users to edit posts by significantly reducing their workload.

Usefulness of EditEx Suggestions
Table REF  shows the participants' assessment of the effectiveness of EditEx and the SO edit system. We see that participants find the suggestions of EditEx influential (3.41 \(\leqslant \)  score \(\leqslant \)  4.20) in avoiding the potential rejections. The Likert score (i.e., 4.0) also shows that they were confident to follow the suggestions given by EditEx. When we asked the reason behind their confidence level, one participant responded that EditEx suggests the common reasons behind the unsuccessful attempts of edit. These suggestions help to identify those and fix them. On the contrary, the participants who use the SO edit system (i.e., control group) were moderately confident (2.61 \(\leqslant \)  score \(\leqslant \)  3.40) in suggesting their edits. To explain the reasons, one participant stated that I was not sure whether my edits are good or bad, so my confidence was low. I cannot understand much from the reviewers' comments why my edits were actually rejected. The reasons were too general. Such findings indicate that EditEx is not only able to provide valuable suggestions but also make users more confident in suggesting edits.
<TABLE>
Recommendations for EditEx Improvements
We analyzed the recommendations of all the participants and summarized them into three categories. We see that participants recommended – (1) enhancing existing functionalities, (2) improving Graphical User Interface (GUI), and (3) notification & installation system. We discuss their recommendations below.

Enhance Functionality


Besides suggesting the rejection reasons, EditEx could also estimate a score based on the quality of the edit.

EditEx can be enhanced by adding a few features related to natural language processing, such as identifying incorrect spelling, sentence complexity.

EditEx should detect minor changes (e.g., adding an article) that do not significantly improve the quality of the posts.

Participants also suggested enhancing the capability of EditEx in such a way that it can identify more potential reasons that might cause rejection.

Participants recommend paraphrasing the notification sentences (e.g., edits may get rejected due to low reputation) to convey them more positively.

Improve Graphical User Interface


A few participants recommend improving the GUI of EditEx. For example, the notification system of the potential rejection reasons could be more appealing. In addition, Suggest Me button should appear beside the edit window to avoid scrolling.

Add Notification & Improve Installation System


One of the main barriers to suggesting edit is that the queue remains full most of the time. Participants thus suggest that EditEx should notify them when the queue becomes free to suggest edits to avoid frequent manual checking.

EditEx uses Tampermonkey to add userscripts for integrating it into the SO edit system. They appreciate it since Tampermonkey is popular and easy to use. However, they suggest deploying EditEx as a standalone browser plug-in in future.

[colframe=black!50,colback=white,left=0pt,right=1pt,top=1pt,bottom=1pt, boxrule=1pt,arc=1pt, width=4.7in, center]
Summary of \(RQ_2\)  and hypothesis testing: Results from the user study on the editing of SO posts show that EditEx can support the SO edit system to prevent overall 49% of rejected edits, including the commonly rejected ones. However, it can prevent 12% rejections even in regular free-form edits. On the other hand, results from the survey of user study participants show that users who use EditEx found its suggestions influential while editing posts. Therefore, they were confident to follow the suggestions. As a result, EditEx can reduce the cognitive workload in half compared to the SO edit system.
Such findings reject the null hypothesis (\(H_2\) ), which hypothesized that the EditEx tool is ineffective in helping users avoid rejected edits.

Discussions
In this section, we first explain the importance of the features used in our machine learning models (Section REF ). We then discuss the reasons behind the misclassifications of the machine learning models (Section REF ). Finally, we discuss the implications of our study findings and the developed EditEx tool in Section REF .
Ranking of Features in the Machine Learning Models
<FIGURE>In Section REF , we discussed several features used to develop our machine learning models in predicting whether an edit will be rejected or accepted. However, we do not know which features are more important in machine learning classifiers than others in differentiating rejected and accepted edits. To find the important features, we thus attempt to rank our features using two popular measures as follows.
Information Gain.
We attempt to determine which features are more robust than others for discriminating between rejected and accepted edits. We thus employ an information gain-based feature ranking technique because it can estimate the discrimination power of each of the given features. In information theory, the information gain of a random variable is the change in information entropy between an initial state and a state that takes some information [19]}. Therefore, the information gained from a particular attribute in classifying rejected edits is as follows.
\(Info\hspace{1.42262pt}Gain(C, a{_i}) = H(C) - H(C|a{_i})\) 
where \(C\)  represents a particular class (i.e., rejected/accepted), \(a{_i}\)  denotes the attribute, and \(H\)  denotes information entropy.
Fig. REF  shows the information gain of our selected features. We see that changes in text and reputation score have the highest information gain. It means that they might discriminate the edits more accurately than others. Changes in code and addition/removal of gratitude have the next highest information gain. However, the information gains of the remaining features are minimal. Therefore, those features contribute less to the machine learning models to classify the rejected edits from the accepted ones.
SHapley Additive exPlanations (SHAP) Feature Importance.
The SHAP value is the average marginal contribution of a feature towards the model’s prediction across all possible combinations of features [48]}. It shows whether a feature value can increase a model’s prediction over a random baseline [49]}. However, the idea behind SHAP feature importance is that features with larger absolute SHAP values are more important than others. SHAP values can be calculated for any tree-based model. We calculate the SHAP values to rank features from the Random Forest model.
Fig. REF  shows the SHAP values of our selected features. We see that the top four features are the same as when using the information gain-based feature ranking technique. According to SHAP values, code change has the second-highest capability of identifying rejected edits following the reputation score. However, the addition/removal of gratitude has more power to discriminate rejected edits from accepted ones over changes in texts.
We then attempt to see the actual effect of our selected features on the model's performance. We thus remove the low-ranked features (according to SHAP values) one by one and evaluate the performance of our model. Table REF  shows the experimental results. According to the experiment, the model's performance gradually degrades when we remove features one by one. For example, overall accuracy decreases 2% when we remove deprecation.
Interestingly, we see a slightly higher performance when we keep the top six features than the top seven. Then the performance again decreases after removing features. However, the overall accuracy drops from 69.8% to 61.3% when we remove all other features except the top four.
<TABLE><FIGURE>
Analysis of Trained Machine Learning Model & Its Misclassifications
Machine learning models can produce accurate/inaccurate predictions. However, their black box nature might prevent their easy adoption and enhancement by others. The SHAP [50]} is a popular model interpretation framework to interpret the classification/misclassification results of the model. In our experiment, we conduct a binary classification where the rejected edits was considered as the positive class and accepted edits as the negative class. Thus, our models attempt to predict the rejected edits by default. Therefore, a positive SHAP value indicates an increase in our models’ prediction of positive class and vice versa. Fig. REF  shows the importance of our selected features using a bee swarm plot from our random forest model. The bee swarm plot visualizes the SHAP value of a feature from each of the training instances on the x-axis. On the y-axis, it sorts all features in descending order according to their sum of SHAP values. The blue color indicates a low feature value, whereas the red indicates a high feature value in our plot. We see that reputation is the most important feature according to our random forest model. That is, community trust, which is estimated by reputation score, is an important predictor of the acceptability of suggested edits. We note that this feature with true response often leads to negative SHAP values, which indicates an increased prediction towards edit acceptance. That is, edits suggested by users with a high reputation score have a higher chance of being accepted and vice versa.
We also analyze our dataset for further insights on this. In particular, we conduct a comparative analysis of the reputation scores between users with rejected and accepted edits. We find the median reputation score of users whose suggested edits were rejected is 3428. On the contrary, such a score of users whose suggested edits were accepted is more than double (i.e., 7660). We then attempt to see whether the difference is statistically significant or not. We use the Mann-Whitney-Wilcoxon statistical significance test and find a statistical significance p-value (p-value \(\simeq \)  \(0.0\)  \(<\)  \(0.05\) ). We also use Cliff's delta test to determine the effect size and find a medium effect size (Cliff's d \(= -0.34\)  (medium)) with \(95\%\)  confidence. Given this evidence, low-reputed users are less trusted by the community, and thus their edits can be rejected.
The second most important feature is the code change. Code change with true response often leads to positive SHAP values, which indicates an increased prediction towards edit rejections. It refers that suggested edits get rejected when users change the code much. On the contrary, text change with true response leads to negative SHAP values, which means that significant changes in texts are acceptable. Similarly, we see that the true response of text & code format, reference modification, and complete change of posts lead to negative SHAP values, which indicates an increased prediction towards edit acceptance. On the other hand, the true response of gratitude, deface post, signature, status, greetings, and inactive links lead to positive SHAP values, which indicates an increased prediction towards edit rejections. However, according to the SHAP visualization, a few features such as reputation, code & text change, status, and reference modifications confuse our model. For example, reference modification's true response leads to positive and negative SHAP values. Therefore, those features might cause misclassifications of our model.

Implications of Study Findings
The findings from our study and the tool EditEx can guide the following major stakeholders in crowd-sourced knowledge-sharing platforms that use collaborative editing features:
forum designers to improve the edit system,

forum users to guide their edit behavior, and

researchers to study and improve collaborative editing support in crowd-shared platforms.
 We discuss the implications below.
Forum Designers. The quality assurance of shared content is paramount for the usefulness and popularity of a crowd-shared knowledge-sharing platform like SO. While the editing of content allows users to suggest improvements in quality, the lack of proper guidance to the user can lead to unnecessary rejections of the suggested edits. SO can use our developed machine learning model and the EditEx tool to offer on-demand and context-aware edit fix recommendations to the users. As we observed in Section REF , the SO users that used EditEx had significantly less number of rejections compared to the users that did not use EditEx tool. The SO edit assessment queue is set up to ensure novice SO users (with less than 2K reputation) can not make bad edits. Many users in SO fall under this novice category, but their inputs to edits are equally as important as the non-novice users. However, as we noted, the SO edit queue can be often congested with many suggested edits that it could take a disproportionate time for an edit to get reviewed. Even after the review, many suggestions can be rejected due to trivial edits like undesired text formatting. A tool like EditEx can help SO users by reducing such trivial edits, which then can ultimately help SO and its expert edit reviewers with less rejection, which in turn can reduce the workload on the edit queue.
Forum Users. Interactive browser plug-ins like EditEx can warn forum users of potential edits that could be rejected. Thus EditEx can improve the confidence of SO users during the editing. Indeed, as we reported in Section REF , the study participants were more confident while using EditEx than using SO editing system (average confidence of 4 while using EditEx vs 3.1 while using SO). The SO users had only 16% of their edits rejected while using EditEx, while the rejection rate was 65% while using SO. While a rejection ratio of 65% for SO could be biased due to our choice of editing task, the much lower rejection rate while using EditEx does indeed highlight that SO users can benefit from a simple tool like EditEx. EditEx is very easy to install. Given that EditEx is simply a browser plug-in that SO users can easily install, we hope that EditEx will be accepted by the wider SO community.
Indeed, our developed classifiers can be used to detect rejected edit reasons in SO automatically. It can extend current tools and techniques that predominantly use contents from suggested edits to recommend editing suggestions (e.g., see the works of Chen et al. [51]}, [52]}). The tool EditEx, with further modification (as suggested by our study participants in Section REF ), can be influential in the reduction of rejected edit reasons in SO and can improve the overall satisfaction of SO users. In the long term, the tool can promote better content because users will be more motivated. Such high quality contents then can offer better content and recommendation support for tools and techniques that focus on the quality of contents shared in SO [53]}, [54]}, [55]}, [56]}, [57]}, [58]}, the suite of tools and techniques developed to detect and recommend quality posts [59]}, [60]}, [61]}, [62]}, [63]}, [64]}.
Researchers. The quality of knowledge shared in SO is important because developers worldwide now rely on this shared knowledge. Indeed, knowledge shared in SO can support diverse activities like bug fixing, feature enhancement, API selection, and documentation [65]}, [66]}, [67]}, [68]}, [69]}, [70]}, [71]}, [72]}. This sharing of knowledge is important, because official software documentation can often be lacking [73]}, [74]}. However, the editing of content is a voluntary activity. SO users can be demotivated to produce quality edits if they become frustrated due to unnecessary/unwanted rejection of their suggested edits. Tools like EditEx or proactive policy assurance by Chen et al. [75]}, [51]} can help SO users with suggestions to improve their edits. The positive survey responses of our tool EditEx show the potential of deploying the tool in SO. Future research can contribute by including more features into EditEx and by conducting more studies to learn how SO users can further benefit from such tools. Such findings can promote quality contents, which then can support all development tasks that rely on SO, as noted above.
Threats to Validity
Threats to internal validity relate to experimental errors and biases [77]}. We asked participants to suggest edits using EditEx (treatment group) and SO's standard edit system (control group). The suggested edits were either accepted or rejected by the expert review. However, the accepted edits could be rejected later by a rollback that might affect the edit rejection ratio. We thus further analyzed how many accepted edits were rejected by rollbacks in our dataset. However, such a statistic was less than 1% in our dataset. Thus, it might not affect our results significantly.
Threats to external validity relate to the generalizability of a technique. We agree that there might be rejection reasons that we could not identify. However, we analyze statistically significant samples of rejected edits by rollbacks from questions and answers. We thus believe that our manual investigation exposes all the main rejection reasons. Still, there is scope to analyze more samples to explore additional rejection reasons. However, suggested edits can be rejected by either rollback or expert review. Unfortunately, we could not collect samples of those edits rejected by expert reviews. We could not find any convenient way to collect such samples since their information is not readily available in the SO data dump. Thus, similar to existing literature, we consider the rejected edits by rollbacks [3]}. However, suggested edits are reviewed in SO by users with at least a 2K reputation score. In our manually analyzed dataset, 90.6% (692 out of 764) of users who rolled back edits have a reputation score \(\ge \)  2K. The remaining were self-rollback (i.e., rolled back by post owner). Therefore, our intuition is that the main reasons for rejections by expert reviews would be similar to our identified reasons.
Our survey participants range from novice to experienced, mainly software developers and academicians (Table REF ). Such diversity in the survey participants offers validity and applicability to the survey findings. Furthermore, we ensure that control and treatment groups have participants with different professions and experience levels to mitigate individual bias.
We set a task list T1–T7 (Table REF ) and asked participants (control & treatment group) to suggest edits to posts based on that list. However, tasks T1–T6 were set based on our identified rejection reasons. Thus, EditEx alerts the treatment group while suggesting T1–T6, but the control group does not get such alerts. As a result, the treatment group receives favor from EditEx, which might reduce their rejections. To mitigate this bias, we asked participants to suggest free-form edits (T7). From T7, we attempt to see the effectiveness of EditEx in assisting users in suggesting regular edits besides preventing common rejections. However, EditEx can support the SO edit system to prevent 12% rejections. Such a finding confirms the effectiveness of EditEx in preventing not only common rejections but also rejections from regular edits.

Deviations From Registered Report
This section discusses the deviations of this study from our registered report [2]} and explains them.
Rollback Reasons & Predictors.
In our registered report, we wanted to see whether the addition/deletion of emotion influences edit rejection using EmoTxt [80]}. Our primary analysis found that emotion has almost no effect on edit rejection/acceptance. Furthermore, the overall accuracy of our rejected edit classifier improved only about 1% if we consider emotion as a predictor. However, integrating a complex model to capture emotion and its deployment is costly. Therefore, it can affect the performance of our online tool EditEx. We thus discard emotion from this study.
On the other hand, we added reputation as a predictor (Table REF ). The reputation score estimates how much the community trusts a user [81]}, [82]}. Therefore, we added Community Trust as a rollback reason (which was absent in the registered report) (Table REF ) that was estimated by reputation score. Moreover, users with lower reputation scores might violate the editing guidelines more than those with higher reputation scores. Note that violating edit guidelines is one of the causes of edit rejection. We thus consider reputation as a predictor that significantly improves the performance of our rejected edit classifier.
Another discrepancy to the registered report is that we included the Introducing Spam rollback reason under Other.
Manually Investigated Sample Size.
In the registered report, we manually investigated 777 rollback edits (382 questions + 395 answers). The statistically significant sample size of rollback edits for both question and answer is 382. However, due to a programming problem, we randomly selected 395 samples from rejected answer revisions. We kept 395 samples in the registered report because - (1) we completed our analysis using 395 samples, and (2) 395 is more than the statistically significant sample size. However, we later randomly selected 382 (among 395) samples (Section REF ) to equalize it with the statistically significant sample size and analyze them.
Recruitment of Participants.
We planned to recruit 30 participants (15 for treatment + 15 for the control group) who edited at least 100 posts. After deploying our tool EditEx, we realized that EditEx could be helpful to both expert and novice SO users. Therefore, we relaxed our constraints. We recruit participants who edited any SO post to ensure familiarity with SO editing. However, we struggled to recruit 30 participants due to COVID-19 and the extensive nature of this user study. The study was extensive because each user had to do multiple edits of SO posts. Primarily, we planned to recruit participants using a snowball approach. To recruit more participants, we then extend our approach. Besides snowball, we attempt to recruit participants using an open circular. However, finally, we recruited 20 participants (10 for treatment + 10 for the control group) with different experience levels and diverse professions (Section REF ).
Number of Edits Per User.
We planned to ask each participant to suggest ten edits. However, several participants (especially the control group) could not edit ten posts for three main challenges, as we discussed in Section REF .
EditEx's Functionality of Highlighting Texts.
We planned to include EditEx's functionality to highlight texts that may cause rejection. However, the current version of EditEx cannot highlight texts. EditEx predicts the edit decisions (accepted/rejected) and alerts users with the potential rejection reasons if rejected. While highlighting texts could be helpful, we found that the EditEx tool with the basic features was usable and effective. Therefore, we leave the highlighting of texts in EditEx as a future extension.
NASA TLX Workload.
We planned to estimate TLX effort as a task load by combining all the ratings provided by a participant in five dimensions in the TLX metrics. However, most existing studies estimate subjective workload by combining all the ratings in six dimensions [37]}, [38]}, [39]}, [40]}, [87]}. Therefore, we also take ratings from each participant on six dimensions (mental demand, physical demand, temporal demand, effort, performance, and frustration). Finally, we estimate the cognitive workload (Fig. REF ). However, we consider a scale of ten with one step size [41]} to take ratings conveniently from participants for each dimension. It slightly sacrifices granularity in comparison to scale 100 with step size five. However, the results should not be affected much.
Unidentified Reasons.
Our rejection reason classifier cannot identify a couple of rejection reasons, such as partial acceptance and incorrect text/code change. We can extract added or removed text/code using appropriate HTML tags. However, partly accepted text/code cannot be separated from added text/code. Furthermore, it requires analyzing the future revisions to check partial acceptance, which is impractical. We did not find any patterns that can identify incorrect changes in code/texts. Identification of such reasons demands manual efforts.

Related Work
We developed our tool EditEx to recommend fixes to suggested edits in SO so that SO users can avoid committing undesired edits that may lead to the rejection of the edits. As such, our research in this paper belongs to a broader area called `collaborative editing in social forums'. Major related work can broadly be divided into Studies of collaborative editing systems in crowd-sourced forums (see Section REF ) and Techniques to suggest improvements to the editing system (see Section REF ). In addition, SO data are used extensively in SE research for various tasks (see Section REF ), all of which could be potentially impacted by having low-quality data due to erroneous/inefficient edits.
Studies of Collaborative Editing Systems
Editing of content can improve the content quality. As such, it is intuitive that the social Q&A forums offer to edit the post contents. Since social forums can be accessed by many users simultaneously, it is a cost-effective measure for the forums to support collaborative editing by allowing their users to do the editing. Indeed, studies
show that collaborating editing in social forums and online collaborative knowledge-sharing portals (e.g., Wikipedia) can positively impact towards the improvement of shared contents [89]}, [90]}. The nature of the collaborative editing can be similar across the social forums (e.g., Q&A site) and knowledge portals (e.g., Wikipedia). The research of Li et al. [89]} looked at the adoption of Wikipedia-style collaborative editing into a Q&A site like SO. They found that users with good edits are rewarded with positive votes by other users. They analyzed five years of historical editing data from SO and found that substantive edits from other users can increase the number of positive votes by 18% for the questions and 119% for answers. This reward can be beneficial for a user who does the edit because the edit may only offer at most 5% improvement over the original post (i.e., the user can be rewarded with mindful but low-cost editing efforts).
Indeed, the SO reward system can serve as an added influence to the users to suggest edits. A recent study by Wang et al. [3]} in SO found that users are motivated to edit more when they are closer to getting
a badge.
Overall, both Wang et al. [3]} and Li et al. [89]} conclude that offering incentives as reputation scores is useful to improve post quality within a collaborative editing platform like SO.
This finding was also observed in other collaborative editing platforms like webcasts [95]} and Wikipedia [90]}.
Munteanu et al. [95]} tested the effectiveness of engaged users in collaborating in a wiki-like webcast platform to edit/correct transcripts that are produced from webcasts through
an automated speech recognition system. Collaborative editing can be a cost-effective but useful means to improve the quality of the ASR (Automated Speech Recognition) system in webcasts because ASR systems can have an average error rate of 45% - above the accepted threshold of 25%. The field study carried out by the authors in a real lecture environment found that using students to edit the webcast transcript was useful in reducing the error rate. The editing was supported via a webcast extension that engages users to collaborate in a wiki-like manner.
Kittur et al. [90]} find that the increase in the number of editors does not guarantee the quality of the articles on Wikipedia.
The quality of the question is important to get an answer: lack of clarity, relatedness, and reproducibility of the problem, as well as the too short question, could dissuade developers from answering the question [99]}, [58]}. The reputation
and past activity of an asker could also factor into the likelihood of a question getting resolved [56]}. As such
factors of good questions are investigated, e.g., code to text ratio, etc. [64]}, [103]}.
However, depending on the platforms and user characteristics, these factors can vary [55]}.
As such, it is important to detect content quality automatically [59]}, [106]}, [61]}. Wang et al. [3]} found that users who make more edits in a short time are likely to get more edits rejected. Thus bad edits can harm the content quality.
Our research on SO rollback edits initially started in 2019 to better understand the edit rejection reasons as reported by Wang et al. [3]}. Through our qualitative analysis of SO posts, we also found all the edit rejection reasons reported by Wang et al. [3]}. In addition, we found four more edit rejection reasons. We report the edit rejection reasons in Section  of this paper. While the above papers, including Wang et al. [3]}, focus on analyzing editing mechanisms in collaborative platforms based on empirical studies, our paper focuses on developing techniques to automatically suggest fixes to suggested edits so that the edits will not be rejected upon submission. As such, our developed tool EditEx can further contribute to supporting the content quality in social forums by assisting users with guidance on improving the quality of their suggested contents. Thus, our paper offers complementary viewpoints to the above studies by offering tools and techniques that can facilitate improved edit content in a social Q&A site like SO.

Techniques to Develop to Improve Collaborative Editing Systems
Collaborative editing systems are common in Wikipedia  [89]}, [90]},
GitHub code editing [114]}, webcasts [95]},
scientific contents [116]}, [117]}, and so on. Compared to substantial research on conducting studies on existing collaborative editing systems, we are not aware of much research that focuses on developing tools and techniques to improve the systems. This is perhaps due to the fact that currently available collaborative platforms like Wikipedia seem to work well and are hugely popular. In all these platforms, the focus of collaborative editing is to improve the quality of the shared content based on
user engagement [57]}.
Chen et al. [51]} observed that most of the edits in SO are small sentence edits. While developing their SOTorrent database, Baltes et al. [120]} also observed that majority of edits in SO are relatively small. In a follow-up study,
Chen et al. [75]} predicted whether a post needs to be edited. Their approach is based on the concept of `proactive policy assurance', which assures that a modification to a suggested edit will satisfy the current `reactive policy assurance' in SO, which accepts/rejects based on the matching of exiting editing policy after an edit is submitted (i.e., reactive).
They developed a deep-learning-based policy assurance tool to recommend post owners or other users' potential mid-level edits to given post content. The deep learning model is a CNN (Convolutional Neural Network). In a large-scale experiment, they find that the tool offers good precision, recall, and F1-score (at least 0.7) while suggesting mid-level edits.
As we noted in Section REF , our research of this paper started in 2019 to gain hands-on experience on the edit rejection reasons observed by Wang et al. [3]}. Our initial exploration led to an expansion of the edit rejection reasons and to the submission of a registered protocol report in 2020 [2]}. In the registered protocol report, we outlined our vision of this paper by offering to develop machine learning model to automatically detect the edit rejection reasons and to build our EditEx tool that can offer proactive guidance to fix suggested edits. While working on this paper, we observed that some edit rejection reasons could be present both in accepted and rejected edits, resulting in inconsistencies in the editing acceptance/rejection process. We reported a catalog of such inconsistencies in our MSR 2021 paper [13]}. In our MSR 2021 paper, we also report several rule-based tools that we developed to detect inconsistencies in SO edits automatically. While developing our EditEx tool in this paper, we purposefully did not consider those inconsistencies, given those were not outlined in our 2020 registered protocol report [2]}. We note that an immediate extension of EditEx could investigate whether and how the inclusion of the inconsistencies into the rejection prediction models and the EditEx tool could make the overall editing process more effective for the SO users. We leave it as our immediate future work.

Other SE Research using SO Data
Several studies have been conducted to study developer discussions on different crowd-shared developer platforms, including SO. Seaman et al. studied developer discussion on inspection meetings [126]}. Reiner et al. used content analysis to study developer discussions on software processes [127]}. Gottipati et al. study relevant answers in 3 software forums, Dzone, Tips, and Oracle forums [128]}. Several studies has focused on discussions on microblogs, such as, Twitter [129]}, [130]}, [131]}, and chat communities, such as, HipChat [132]}, IRC messages [133]}, [134]}, and Slack [135]}. Recently, SO Q&A forums have been subject to a number of papers to study various aspects of software development, such as what developers are discussing in general [136]},
or about a particular aspect, e.g., concurrency [137]}, big data [138]}, chatbot development [139]}.
Several studies has been conducted to study developer sentiments on online discussions (e.g., SO data) [140]}, [141]}, [142]}, [143]}, [69]}, [65]}, [66]}, [68]}, [72]}, [71]}, [70]}, [151]}. Guzman et. al. applied sentiment analysis on code comments [140]}. Islam and Zibran study emotional variations in commit messages [153]}. Garcia et al.  [154]} study the emotions of developers in the Gentoo community. Guzman and Bruegge studied developer sentiments on mailing lists [155]}. Novielli et al. conduct sentiment analysis on SO and Github discussions [156]}. Many of these studies use automated sentiment analysis tools, which are found to provide contradictory results in software engineering research [157]}.
All the above research works using SO data could be benefited from improved data quality offered by collaborative editing in SO. As such, our tool EditEx, once adopted by the SO users, can help the SO users as well as the SO-based research community with better quality data.

Conclusion
SO has become an essential online resource with millions of programming-related problems and solutions. However, the quality of the shared knowledge is vital for the growth and success of SO. To promote quality, SO introduces an edit system so that users can suggest an improvement to posts. Unfortunately, numerous suggested edits are rejected due to either undesired changes of posts or violating edit guidelines. Such a scenario not only hurts the quality of content but also frustrates and demotivates users. We conducted a qualitative analysis of 764 (382 questions + 382 answers) rejected edits by rollbacks and identified 19 rejection reasons. We then extract 15 texts and user-based features to automatically capture those reasons and develop four machine learning models using them. Our best-performing model can predict rejected edits with about 70% accuracy, and the rejection reason classifier can identify the potential rejection reasons with 67% accuracy.
We also introduced an online tool named EditEx that can be integrated with the SO edit system. It analyzes the edits, predicts whether they will be rejected, and suggests users with the potential rejection reasons.
We conduct a survey to assess EditEx and SO edit system. According to survey results, the participants find reasons for rejection identified by EditEx influential. Moreover, EditEx can support the SO edit system to prevent 49% rejections, including the commonly rejected reasons. Such a statistic is 12% when users suggest regular free-form edits. Moreover, our tool significantly decreases the subjective workload and increases participants' confidence in suggesting edits.
This research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery grants, and by an NSERC Collaborative Research and Training Experience (CREATE) grant, and by two Canada First Research Excellence Fund (CFREF) grants coordinated by the Global Institute for Food Security (GIFS) and the Global Institute for Water Security (GIWS).
