Over 20 million U.S. voters classified as 'inactive' Millions of voters may get a surprise when they show up at the polls on Election Day. They could be told, they are "inactive." An "inactive voter" is one who has not cast a ballot in two straight federal elections and failed to return repeated post cards from election officials seeking to verify voters' addresses. The law was intended to help clean up the nation's voter rolls. "An inactive voter is an individual who has not voted, typically, in literally almost half a decade," Louisiana Secretary of State Tom Schedler said. This is a personal responsibility to vote, it is your right. But it is also your personal responsibility. Schedler notes that his office continually tries to maintain a correct, current list of active voters and makes multiple attempts to reach out to those who have not voted, in order to ensure that they are included. During this election cycle, Louisiana officials have sent the required notices to 62,000 people and have established an online system for people to check their status and rectify it before Election Day. I put voting at the upmost, paramount of our legal rights. But you can see in business, everyone does some type of file maintenance or list maintenance, I don't see any difference in voter registration. Critics charge that the "inactive voter" lists across the country have led to the unfair removal of voters from the voting rolls. They have branded the process vote "purging." "Inactive voter lists really place the onus on voters, to figure out what their voter registration status is," laments Ryan Haygood, director of political participation at the NAACP Legal Defense Fund. Voters tend to believe, rightfully so, that after having registered to vote, they're always registered to vote. ... What we should be asking ourselves and what we should be embracing in this election cycle, is a democracy inclusion principle that invites all Americans to be included in the political process, because it is through participating in our democracy that we legitimize it. The number of "inactive" voters is staggering. In its 2011 report, the U.S. Election Assistance Commission put the number nationwide at nearly 21 million. The state with the largest number of "inactive" voters is California, with 6,367,117, or 36.8 percent of all state votes. Texas has 1,887,762; Florida, 1,323,288; New York, 1,126,491, and Illinois, 1,087,321. "Inactive" voters can still vote, but only after confirming their personal information or registering to vote all over again. In the states that require photo ID to vote, they will need to show it. "You're put back on the active list," Schedler said. A secretary of state doesn't go in and say, 'I think I'll take this lady off and I think I'll take this gentleman off, just because maybe they're not the party affiliation or the way I may want to vote.' I think that's ridiculous," he said. It's set up in federal law, it's set up in state law and secretaries of state are doing their job by doing this. He also notes that when it comes to "inactive" voters who fail to show up to vote for many years, officials can only help so much. How much can you do? You can't drag someone down to the poll, effectively. Others caution that the massive number of "inactive" voter names could lead to voter fraud. "You look at what happened in Florida in 2000, my biggest fear would be that you would have that times 30," says Lionel Rainey III, spokesman for the new voter activist group Voteguards.org. "What I am worried about is people being able to have that list and to send people into those polls to vote for a candidate, especially with states that don't have voter ID laws," he said. If you look at how close some of these elections are, in some of these states, it doesn't take that many votes to be able to swing an election, especially when you look at an election like this one that is turning out to be a lot closer than anyone anticipated. ... If that data got into the wrong hands, it is simple to understand how anyone could be able to use that to manipulate an election. "If the integrity of the elections are ever questioned, this country could turn into a third world country overnight," says Schedler. I know here in Louisiana, we have certainly learned our lesson. Haygood is worried that legitimate voters could be hampered, and the result would be confusion Nov. 6. "With the election less than a month away, it is important for voters to understand today what their voter registration status is, and not wait until Election Day to figure those things out," he advises. The NAACP Legal Defense Fund has set up a hotline for voters to help them determine their status, and even polling location, by simply calling 866-OUR-VOTE. "It is really important for voters not to wait until Election Day," he said. A smooth registration process benefits voters, and those who run elections. "At the end of the day," Schedler said, "the best thing I can be told by a losing candidate is, "I never had a question about the fairness of the election. If that happens, democracy wins and I have done my job. Fox News' Kathleen Foster contributed to this report