To assess the rankings produced by ST-Rank, we defined a test set by randomly removing 20% of the training rankings (along with their training instances). Specifically, we shuffled the order of each training ranking. Then, we presented the technologies of each ranking to two senior JS developers and asked them to produce reference rankings by sorting the technologies in descending order of relevanceThe reference rankings can be found in the Supplementary Material zip file at https://bit.ly/2w9sOzV.. The remaining 80% of the training instances were divided to perform a k-fold cross-validation [1]} with \(k=5\) , in order to train the ML model and find the best configuration of hyper-parameters for GBRank. Specifically, we used random search [2]} to explore the possible values of the hyper-parameters, and the values close to the best random configuration achieved by a grid search [3]} were applied. In the end, GBRank was run with parameters \(learning\_rate=0.004\) , \(max\_depth=50\) , \(min\_samples\_split=50\) , and \(min\_samples\_leaf=10\) .
