Attorney General Eric Holder gestures as he testifies on Capitol Hill in Washington, Wednesday, May 15, 2013, before the House Judiciary Committee oversight hearing on the Justice Department. President Barack Obama and Attorney General Eric Holder attend the 32nd annual the National Peace Officers Memorial Service, Wednesday, May 15, 2013, on Capitol Hill in Washington, honoring law enforcement officers who died in the line of duty. WASHINGTON (AP) - While President Barack Obama quickly added his voice to the bipartisan outcry over the Internal Revenue Service's targeting of conservative political groups for extra scrutiny, it's hardly the first time a White House and the tax agency have been targets for claims of political meddling and bias. Nor is it the first time that political and social advocacy groups have searched for and exploited loopholes and fine points in the federal tax code. President Franklin D. Roosevelt's frequent use of the IRS as a weapon of political retribution is well-documented. He reportedly had the IRS scrutinize tax returns of his harshest critics, including Sen. Huey Long of Louisiana and Hoover Treasury Secretary Andrew Mellon. And President Richard Nixon famously subjected those on his "enemies list" to tax audits and was even caught on Watergate tapes boasting of it to aides in the Oval Office. When Treasury Secretary George Schultz questioned Nixon's directive to dig into financial dealings of businessman Henry Kimmelman, a major backer of Nixon's 1972 opponent, Sen. George McGovern, Nixon complained to aides: "What's he trying to do, say that we can't play politics with the IRS? ..." President Dwight Eisenhower used federal tax collectors to go after members of the Communist Party. President John F. Kennedy set up an IRS program to "explore the political activities" of conservative nonprofit organizations. President Jimmy Carter's IRS director, Jerome Kurtz, ended the tax-exempt status of private Christian schools out of compliance with federal policy against racial discrimination. During Bill Clinton's presidency, the IRS audited the conservative Heritage Foundation and the National Rifle Association. And during President George W. Bush's tenure, the agency audited the NAACP for remarks its leaders made during the 2004 election urging Bush's defeat. The IRS alleged it amounted to improper political activity. The civil-rights organization argued it was the victim of political bias. The case dragged on for two years. In the latest flare up over political interference with the tax system, the IRS acknowledged that organizations applying for tax-exempt status during the 2012 election season were singled out for extra scrutiny if they had "tea party" or "patriot" in their titles. In some cases, groups were asked for names of donors. The agency apologized and insisted the practice was not politically motivated. But it touched off anger on Capitol Hill, coming at the same time as controversies over the Justice Department's seizure of phone records of The Associated Press and lingering questions over the administration's handling of September's deadly terrorist attack on a U.S. diplomatic outpost in Benghazi, Libya - inquiries that in combination threatened the president's credibility and, thus, his second-term agenda. "They finally messed with an agency everybody fully understands," said Senate Minority Leader Mitch McConnell, R-Ky., among Republicans complaining the loudest about the IRS transgressions and seeking to tie them to the president. Several days after IRS acknowledged its improper actions, Attorney General Eric Holder announced that his agency would investigate whether laws were broken as a Justice Department inspector general's report blamed "ineffective management" at the IRS. And, trying to control the fallout, Obama forced the resignation of Acting IRS Commissioner Steven Miller. "Americans are right to be angry about it and I'm angry about it," the president said of the IRS targeting. The IRS has to operate with absolute integrity. Thus far, little - if anything - has surfaced to suggest White House officials had advance knowledge of the targeting of conservative groups. It is true that, in the past, presidents going back as far as at least Franklin Roosevelt used - or attempted to use - the IRS to intimidate or go after their political opponents. So it's a bipartisan thing," said economist Bruce Bartlett, a tax-policy official in the administrations of Republican Presidents Ronald Reagan and George H.W. Bush. "But this one seems to have been driven more from the bottom up rather than the top down, promoted by lower-level IRS officials," Bartlett said. The IRS must weed out the social-welfare and educational organizations that qualify as fully tax exempt - designated in the federal tax code as 501 (c) (4) groups - from ones that are primarily political and generally not tax-exempt. The difference is that the tax-exempt groups cannot advocate on behalf of individual candidates or parties, even though many groups come close to crossing the line. The workload became even more intensive for the IRS as the number of groups registering for tax-exempt status roughly doubled in the aftermath of a series of campaign finance rulings ahead of the 2012 elections. Those decisions further obscured the already fuzzy line between political organizations and social welfare organizations and effectively spelled the end to many Watergate-era and later laws restricting political contributions. The rulings also coincided with a rise in political activism - most of it occurring on the right. The Supreme Court's 2010 Citizens United decision in particular helped move the debate over distinguishing between political and social welfare groups "into the realm of regulation and bureaucrats protecting themselves and tax lawyers paid to find loopholes," said presidential historian Stephen Hess. The dispute, along with the controversy over government seizure of AP phone logs and Benghazi, may not torpedo Obama's presidency "but reflects the political level of skill of this administration, which has not been as high as many of us thought it would be based on his skill in winning the election," said Hess. "There's a difference between campaigning and governance," he added. Follow Tom Raum on Twitter: http://www.twitter.com/tomraum