Is it the end for Toronto Mayor Rob Ford? Mayor's brother is encouraging him to take a leave of absence, a source says "When you attack my integrity as a father and as a husband, I see red," Mayor Rob Ford says Ford apologizes for lewd remark about oral sex on live Canadian television Mayor and his brother will begin their "Ford Nation" TV show on Monday Toronto (CNN) -- Is Friday the beginning of the end for Toronto's troubled Mayor Rob Ford? Not so, if you ask Ford. But a growing contingent in Toronto's City Council seem to think it's time for him to go. The council is scheduled to meet Friday to begin mapping out a plan to usurp much of Ford's power. Despite admitting last week that he had smoked crack cocaine in a "drunken stupor" about a year ago, Ford has defiantly said he will not leave his job. But while fighting for his job Thursday, Ford added to his growing list of missteps. Early in the day, a scrum of reporters approached Ford to ask him about new allegations of drunkenness, drug use and the verbal and physical abuse of aides. In the course of answering, he denied a female staffer's allegation that he sought to perform oral sex with graphic language of his own, stunning reporters. Later in the day, he backtracked. With his wife at his side, he went back before reporters to say he had been under "tremendous, tremendous stress" and was getting unspecified support from "a team of health care professionals." But he called the latest allegations "100% lies." When you attack my integrity as a father and as a husband, I see red. Today I acted on complete impulse in my remarks," Ford said. He took no questions from reporters, who shouted sharp inquiries at him as he entered the office. Mayor Ford, why should we believe you? Why would you subject your family to this?" one asked. "What's the matter with you, Mr. Mayor?" another said. Plan to take power The Toronto City Council voted 37-5 on Wednesday to ask Ford to take a leave of absence, and most members turned their backs on him when he addressed their Thursday meeting. His brother, Doug Ford, a city council member, is urging him to take a leave of absence, a council source told CNN on condition of anonymity. The source did not want to be identified. The meeting Friday could lead to more Monday where the council says it expects to consider a motion to delegate duties not assigned by statute to the mayor. If approved, the motion would strip Ford of most of his duties. Among other things, the motion calls for reallocating the operating budget of the office of the mayor to the city clerk, according the council's agenda published online. Plans for a TV show But in a new twist, Canada's Sun News Network announced that Ford and his brother Doug will begin hosting their own TV show on Monday. The show, "Ford Nation," picks up where the two brothers left off on a talk radio show that ended its run last week, the network said on its website. A headline on that site posed the question, "Canada's Ultimate Reality Show?" And it quoted Doug Ford saying, "Rob is like Howard Stern or Rush Limbaugh" and "You just never know what he is going to say." "Mayor Ford seems to be in a no-holds-barred mode," the network said. There is no filter. There is no careful treading. Critics of Sun News Network have described its conservatism as "Fox News North," according to a story about its launch in 2011 by the Toronto Globe and Mail. Plans for legal action The mayor recently confessed to purchasing illegal drugs in the past two years, while he was in office. But he has steadfastly refused to step down, saying the voters of Canada's largest city should decide his fate in 2014. The latest allegations surfaced Wednesday in more than 500 pages of court documents that police used to get a search warrant for Alexander Lisi, Ford's friend and occasional driver, whom police accuse of marijuana possession and trafficking. Ford responded Thursday morning by threatening legal action against former staffers who claimed the mayor brought a woman appearing to be an escort to his office and drank alcohol while driving. The documents include police interviews with former staff members, information obtained from surveillance crews and cameras, and even an examination of the mayor's garbage. Several staffers said they were asked to buy alcohol for the mayor of Canada's largest city. One incident described by a former staffer alleged that Ford, while driving, stopped the vehicle, guzzled some vodka, and drove on. The allegations have not been substantiated by investigators, and Ford faces no criminal charges. He called the allegations "outright lies." On Thursday, Ford also said the so-called "escort" wasn't a prostitute but a family friend. CNN's Paula Newton and Laura Dolan reported from Toronto, Lateef Mungin and Matt Smith reported and wrote from Atlanta. CNN's Ashleigh Banfield and Michael Martinez contributed to this report.