The Office of the Director of National Intelligence has released approximately 1,800 pages of documents that shed more light on the Foreign Intelligence Surveillance Court.
The documents indicate that the National Security Agency violated its own internal guidelines relating to phone numbers it can "query" from among records the agency collects.
Moreover, the documents indicate that the NSA presented false information to the surveillance court about the violation.
"The people responsible for authoring the report did not fully understand how the operation was working," a senior intelligence official said. "That misrepresentation resulted in a factually inaccurate report."
The documents satisfy a judge's order pertaining to public records requests from the American Civil Liberties Union and the Electronic Frontier Foundation, a privacy advocacy group, about FISA Court interpretations of the section of the Patriot Act dealing with collecting metadata, the so-called business records provision.
The metadata program started in 2006 and allowed the NSA to seek to obtain more information about a number if there was "reasonable articulable suspicion" that the number was linked to terrorism.
The NSA also kept a separate "alert list" that was used to compare the new numbers that were coming in daily and consider whether new numbers should be added to the category of those with "reasonable articulable suspicion."
The alert list started with about 4,000 numbers and ended up with 17,835, most of which did not have the required suspicion, officials say.
The court ruled that the NSA was allowed to have the alert list, but the agency could not run it against the larger database because it did not have the reasonable suspicion.
Every day, phone companies sent metadata, which went into an archive. But each day, the NSA ran the alert list against the new information to see if it could establish reasonable suspicion. This went on from May 2006 until January 2009.
"To further complicate matters," an official said, "reporting to the court, we described the alert list but did not describe (it) accurately."
Senior intelligence officials attempted to assure reporters that the news was not so much the compliance violation, but the fact that the NSA uncovered the problem, reported it to the Justice Department and the FISA Court, "took steps thereafter to do a thorough scrub of operations," and reported back to the FISA Court after the changes had been made.
In one declassified order from March 2009, Judge Reggie Walton said the court would "not permit the government to access the data collected until such time as the government is able to restore the court's confidence that the government can and will comply with previously approved procedures for accessing such data."
A senior intelligence official noted "fairly strong language" by the court, but stressed that it did not find any "intentional attempt" to violate the law or abuse the program.
Because there was such confusion about the program, the NSA instituted new steps to guard against future violations, including adding a compliance director, the officials said.
One official said this proved that there was "effective oversight by the executive branch and the court. NSA is not perfect and screws up from time to time." But there never has been any indication that these programs have been abused by spying for improper purposes or exceeding guidelines with improper authority, he said.
The officials said they did not know of any NSA employee who was punished or fired as a result of the problem.
Director of National Intelligence James Clapper said in a statement that the incidents were promptly reported to the court, which ordered NSA to seek its approval to query metadata on a case-by-case basis, except when lives were under imminent threat.
"The documents released today are a testament to the government's strong commitment to detecting, correcting, and reporting mistakes that occur in implementing technologically complex intelligence collection activities, and to continually improving its oversight and compliance processes," he said.
The American Civil Liberties Union said the documents confirm that the agency "cannot be trusted" with such sweeping powers and that the "secret and one-sided" judicial review is not an adequate check.
"The abuses revealed in these documents are alarming but also predictable. These violations are the inevitable result of allowing the NSA to assemble a vast database of sensitive information about every American," Alex Abdo, staff attorney with the ACLU National Security Project, said in a statement.
The civil liberties group has challenged the constitutionality of the NSA phone records collection program in court.