(CNN) -- Anti-tax activist Grover Norquist said Monday that his group, Americans for Tax Reform, would work to unseat Republicans who break their pledge to never vote for higher taxes.
His vow came after prominent GOP lawmakers said over the weekend they would consider breaking the Taxpayer Protection Pledge in order to reach a deal with Democrats and President Barack Obama to avoid tumbling over the fiscal cliff - the combination of sweeping spending cuts and tax increases that would go into effect at the end of the year if negotiators can't reach a deal on reducing the federal debt.
Norquist said his group would "certainly highlight who has kept their commitment and who hasn't" when it comes time for lawmakers like Sen. Lindsey Graham and Rep. Peter King to run for re-election, though Norquist claimed voters generally decide on their own to oust elected officials who vote to raise taxes.
"Historically the people who lose do so because the people in their state have figured that out," Norquist said on CNN's "Starting Point with Soledad O'Brien."
On Sunday, Graham and King joined other Republican lawmakers, including Georgia Republican Sen. Saxby Chambliss, in publically breaking from Norquist, whose pledge stipulates signers will "oppose any and all efforts to increase the marginal income tax rates for individuals and/or businesses" and will "oppose any net reduction or elimination of deductions and credits, unless matched dollar for dollar by further reducing tax rates."
Both Graham and Chambliss will be up for re-election in 2014.
Graham said on ABC he'd be willing to break the pledge if Democrats are willing to help pass spending reforms on government programs like Social Security and Medicare. That combination of revenue and spending cuts is in line with what a majority (52 percent) of Republicans prefer, according to a CNN/ORC poll out Monday.
"I'm willing to generate revenue," Graham said on ABC. "It's fair to ask my party to put revenue on the table. We're below historic averages."
Norquist shot down Graham's suggestion Monday, saying it was impossible to imagine a situation where Democrats would agree to Graham's conditions.
"He lists this incredible list of reforms and entitlements that the Democrats would never give him," Norquist said, describing the demand for entitlement reforms akin to a "pink unicorn that doesn't exist."
Rep. Peter King, R-New York, sided with Graham Sunday, saying on NBC a pledge signed decades ago shouldn't be considered relevant today, and that "everything should be on the table."
"A pledge you signed 20 years ago, 18 years ago, is for that Congress," King said. "For instance, if I were in Congress in 1941, I would have signed the declaration of war against Japan. I'm not going to attack Japan today. The world has changed. The economic situation is different."
Norquist challenged that assertion, saying the pledge was good for the length of a lawmaker's time in office.
"Peter King knows full well the pledge he signed, and others have signed, it's for while you're in Congress, it's not for a two-year period," he said.
Ultimately, Norquist downplayed the talk from Graham and King on breaking the pledge as just that - talk.
"No pledge taker has voted for a tax increase. You've had some people discussing impure thoughts on national television," he said.
™ & © 2012 Cable News Network, Inc., a Time Warner Company. All rights reserved.