At the time Harper was in power, other countries were working quietly to remove their citizens in Guantanamo when it became apparent that it was not useful in continuing the imprisonment and torture, and that the prisoners were being tortured and denied basic human rights.  Khadr was left to rot as the last foreign citizen from a Western country remaining in Guantanamo, primarily because Harper was looking to squeeze out political points.  
The US Supreme Court ruled in June 2006 that the prisoners in Guantanamo had Geneva Convention rights to be protected.  It wasn't until Jan 2009 that the US govt conceded that torture happened at Guantanamo (right before Obama took office).   Harper was in power starting Feb 2006, and he put ideology ahead of citizen rights.