Despite the fact that the MIO approach directly aims to solve the best subset selection problem and is faster
than the exact method for larger values of \(p\) , its implementation via Gurobi is significantly slow from a practical point of view. As a result, some of the indirect methods, such as Forward Stepwise selection and the least absolute shrinkage and selection operator (Lasso) and their variants, are still very common in practice. Forward Stepwise selection follows a greedy approach which, starting with an empty model (or intercept only model), iteratively adds the variable that is best suited [1]}, [2]}. On the other hand, instead of solving the highly non-convex best subset selection problem, Lasso introduced in [3]} solves a convex relaxation of it by replacing the (discrete) \(\mathcal {L}_0\)  norm \(\Vert \beta \Vert _0\)  in (REF ) with the \(\mathcal {L}_1\)  norm \(\Vert \beta \Vert _1\) .
Due to this clever relaxation, Lasso is a convex optimization problem. Thus, it is fast (and much faster than the MIO solvers). However, it is known that Lasso solutions are lacking of crucial statistical quality [4]} and do not normally lead to a best subset [5]}, [6]} solution.
