Lord Feldman spoke to journalists who wrote 'swivel-eyed loons' story but denies words were his The Conservative Party Chairman, Lord Feldman, has privately admitted speaking to journalists who reported a senior Tory figure describing local party activists as "swivel-eyed loons," it emerged on Sunday. The Independent understands that Lord Feldman does not deny having an informal conversation with journalists at a restaurant in London last Wednesday. But he insists that he never described Tory grassroots members as "all mad, swivel-eyed loons" - words attributed to a close ally of David Cameron over the weekend. Amid increasing anger from some MPs and members of the Conservative volunteer party about the alleged comments on Sunday night, colleagues of Lord Feldman rallied around him and insisted that he should not resign. But he is expected to be questioned on Monday about exactly what he did say to journalists at a Conservative Party board meeting. Brian Binley MP, who is a member of the Tory ruling body, said he would "be wanting to know more about it" when the group met. "If this was said, by whoever, it would not surprise me," he said. Certainly the gap between the party leadership and the party's voluntary sector in the country is sizeable. I have made that complaint again and again. There has been a disdainful view of the voluntary party. But one senior Tory warned that without Lord Feldman's fundraising skills, local Conservative Associations would not have the money to effectively fight the next election - regardless of what he might or might not have said. They said the party's ability to raise money to fight a national campaign depended on his contacts in the business world and that the Conservatives would suffer organisationally as well as financially if he had to resign. They added that Lord Feldman was "not political" and if he had made the comments widely attributed to him, then he was likely to be "reflecting the views of others" rather this own. "He might have made some sort of joke but I honestly don't believe he would have meant it in the way in which it has been interpreted," said one senior Tory who knows him well. He is not political and not someone who is used to talking to journalists. I know he does not have that disdain for the voluntary party and if he did say anything he is likely to be repeating somebody else has said. They continued: "Frankly the party would not exist financially if it wasn't for him. He is the man who has developed the contacts and has kept our donors with us, even when most don't like the publicity. It would be deeply damaging if he is forced out. Another Tory source pointed out that since being appointed, Lord Feldman had cleared the party's debts and balanced the books every year for eight years. The peer, who has been friends with Mr Cameron since they studied together at Oxford, was one of his first backers - giving £10,000 to his leadership campaign and securing the support of other businessmen. After the leadership election, Mr Feldman became one of the Conservatives" deputy treasurers and was promoted to be the party's chief executive in 2008, a key role preparing the Tories for the 2010 election. Once in Government, Mr Cameron made him a member of the House of Lords and gave him an office in Downing Street - a privilege never before bestowed on a party chairman.