(CNN) -- Removing non-U.S. citizens from lists of registered voters is a legal necessity, the Republican governor of Florida asserted Tuesday after his state was threatened with legal action by the U.S. Department of Justice over the so-called voter "purge."
Gov. Rick Scott said on CNN's "Starting Point" the state had identified "almost 100 individuals" who had registered to vote but weren't U.S. citizens.
"Over 50 of them have voted in our elections," Scott said. "I have an obligation to enforce the laws of our land. You don't get to vote in Florida if you're a non-U.S. citizen.
Speaking at a hearing on Capitol Hill Tuesday, U.S. Attorney General Eric Holder said maintaining voting rights for citizens was critical.
"The right to vote is the lifeblood of our democracy," Holder said. "It is what makes this nation exceptional. The work that I have been doing, the department has been doing, the speeches that I have been giving are not - I am not advocating for a party. I am advocating for a principle. The principle is the right to vote."
On Monday, the Justice Department sent a letter threatening legal action against the state, citing violations of voting rights laws.
The lawsuit comes after the department began questioning the legality of the state's so-called voter purge program, which would remove names from Florida's voter rolls months before the 2012 presidential election, when Florida will play a key role as a battleground state with a large chunk of electoral votes.
Using information from Florida's Department of Highway Safety and Motor Vehicles, the state identified more than 100,000 names of non-eligible voters that could potentially be on the lists illegally.
The Justice Department letter states that the Florida program improperly uses the information collected from old driver's license applications, saying the data could be outdated.
"A number of persons will subsequently have become citizens and lawfully registered to vote," the letter states.
Three of the state's largest counties -- Miami-Dade, Broward, and Palm Beach -- agreed last week to end the removal of the names. The legal counsel for Florida's county election officials recommended halting the purge of names until the state responded to the federal government's legal assessment.
Critics say the plan unfairly targets minorities, and paint it as an attempt by a Republican gubernatorial administration to dissuade typically Democratic voters from going to the polls.
Scott said Tuesday that claim was bogus.
"This is not a partisan issue," he said on CNN. "This is not Republican or Democrat or independent issue. This is an issue that I want, all of us want, everyone wants every U.S. citizen to go and register to vote. Participate in elections. But non-U.S. citizens shouldn't be doing that."
Florida, which itself is suing the Department of Homeland Security for access to a citizenship database, was within its legal rights in moving ahead with the purge, Scott said.
"We need the Homeland Security database. Look, think about it. I don't know anybody that wants to say we want non-U.S. citizens to vote in our races. We want to do the right way. That's what we tried to do. Our secretary of state's office has been working with Homeland Security for months asking for this database so we can do it right," the governor said.
In 2008, President Barack Obama won Florida with 51% of the vote, and the state is classified as a "toss up" on CNN's Electoral Map in 2012. There are 29 electoral votes at stake.