We want to highlight that the discrepancy of the performances among the CKL methods between encoder-decoder LM (T5) and decoder-only LM (GPT-2) may not solely be on the LM architecture, but also on the learning rate and the evaluation method (light-tuning was used to evaluate GPT-2 while we evaluated T5 in a zero-shot manner). We leave further exploration of training ever-changing decoder-only LMs such as GPT-2 as future work.