He certainly was not. He was widely praised by all parties in the House. The Reformer's came to not agree with him when they tried their usual undemocratic tricks. The three big ones were:

-rejecting their challenge of an opposition bill to commit the government to implement Kyoto (he was correct to rule so).

-all three parties asked him to make a ruling on matter of privilege regarding documents around Afghan detainees and he overruled the government (which then ran away from parliament making it only the 2nd time in our countries history where a PM with a minority govn't prorogued parliament - both done by Harper) 

-in 2011 he found the Reformer's in a contempt of parliament case.

Yet despite these rulings he still received praise from all parties, including the Reformer's. John Baird even predicted that "Milliken would “go down in history as, if not one of the best Speakers, the best Speaker the House of Commons has ever had."