Everyone gets tired of not finding anything wrong and they quickly realize they are chasing the wind and wasting their time because it becomes apparent they are actually investigating a good person. 
Everyone gets tired of the ransacking of private lives, the cynical search for ulterior motives, the weighing of imperfect evidence; after a while, people are likely to say, Just let it go.