It`s not to Walker to hold these people accountable. They are protected by the separation of powers, It is only for the voters of his district probably with a recall. 
But realistically?... It`s kind of like the illegal republican gerrymandering done here in Alaska that led to the SB21 vote in the legislature passing by one oily vote.