Obama details broader housing plan, allowing homeowners to refinance mortgages not owned by Fannie Mae or Freddie Mac (AP) - President Barack Obama called on Congress Wednesday to make it easier for millions of additional homeowners to refinance their mortgages at lower interest rates even if they owe more than their homes are worth. He conceded that his administration's housing plans so far have not lived up to their promise. Calling the housing problem "massive in size and in scope," Obama detailed a proposal he outlined in his State of the Union speech last week, tackling an issue of vital concern in states key to his re-election. "This housing crisis struck right at the heart of what it means to be middle class in America: our homes," Obama said, speaking at a northern Virginia community center. Obama's proposal would give homeowners with privately held mortgages a shot at record low rates though a new government program, for an annual savings of about $3,000 for the average borrower. The program is the latest administration effort to help homeowners in the face of a massive number of foreclosures and plunging house values that have left millions of borrowers owing more than their homes are worth. The administration plan aims to ease the way toward refinancing for borrowers, who despite good credit have been unable to take advantage of lower rates because they are underwater on their loans or because banks fear they will be left taking losses. The administration has rolled out housing programs before with the hope of helping millions of struggling homeowners. But those initiatives have fallen short. "I'll be honest, the programs we've put forward didn't work at the scale we'd hoped," Obama said. Not as many people have taken advantage of it as we wanted. The housing issue, while national in scope, resonates particularly in election battlegrounds like Nevada and Florida which have faced record foreclosures. Obama himself drew attention to the politics surrounding the issue with an indirect jab at former Massachusetts Gov. Mitt Romney, now the front-runner in the Republican presidential contests. Romney in October suggested the foreclosure process should "run its course and hit the bottom." MORE: Does Obama Finally Have a Plan to Fix the Housing Mess? Without naming Romney, Obama said: "It is wrong for anyone to suggest that the only option for struggling, responsible homeowners is to sit and wait for the housing market to hit bottom. I refuse to accept that, and so do the American people. The administration proposal faces a major hurdle in Congress. The program would cost between $5 billion and $10 billion, depending on participation, and the administration proposes to pay for it with a fee on large banks. The administration has tried unsuccessfully before to win support for such a tax on large banks. Administration officials, however, said Obama would consider other ways to pay for the program. The plan would expand the administration's Home Affordable Refinance Program, which allows borrowers with loans backed by government-affiliated mortgage giants Fannie Mae and Freddie Mac to refinance at lower rates. About 1 million homeowners have used it, well short of the 4 million to 5 million the Obama administration had expected. Moreover, many "underwater" borrowers - those who owe more than their homes are worth - couldn't qualify. The administration estimates that 3.5 million borrowers with privately-held mortgages have high enough interest rates that they would have incentive to refinance under the new plan. That's in addition to 11 million borrowers who have Fannie- or Freddie-guaranteed loans who could be eligible for refinancing under the administration's proposed changes. MORE: The Neverending Bailout: The U.S. Is Still Owed $133 Billion from Crisis Fund About 11 million Americans - roughly 1 in 4 with a mortgage - are underwater, according to CoreLogic, a real estate data firm. Half of all U.S. mortgages - about 30 million home loans - are owned by nongovernment lenders. The new administration plan would permit homeowners to refinance their mortgages into loans backed by the Federal Housing Administration. To qualify, borrowers with privately held mortgages would have to have no more than one delinquency in the six months preceding refinancing. Their loans would have to fall within the mortgage limits set by the FHA in their home counties. The administration would encourage borrowers to apply their savings directly toward lowering the principle of their loans instead of reducing their monthly payments. As an incentive, borrowers who choose to rebuild equity would not have to pay closing costs and would have to agree to refinance into a loan with a 20 year term or less with monthly payments roughly equal to those they make under their current loan. Obama also announced new industry standards for mortgage servicers, a sort of "bill of rights" for borrowers that would protect them in their transactions. During his remarks, Obama brandished a simpler, three-page loan document to illustrate the new standards. The administration will also undertake a program that would allow the sale of foreclosed homes by Fannie Mae to investors who would then offer the properties for rental. Administration officials say there is a high demand for rental housing and such a program would also sustain neighborhoods by keeping foreclosed homes from falling into disrepair.