To get a good representation of many ACQIs across the training and test sets, we used the multi-label Scikit Learn python package and methodology of
[1]}, which supports balanced multi-label train/test splits which we adapted for nested cross-validation [2]}. Utilizing this package we implemented nested cross-validation using 5 cross-validated folds for the inner and outer loop. For predicting ACQI we tested logistic regression, random forest, and xgboost. For IQ we tested the above and a linear regressor.
We found that the best performing text-based model for predicting both ACQIs and IQ was logistic regression with C (inverse regularization strength) set to
0.01 and with the `balanced' class weight setting in Scikit Learn.
