(CNN) -- Army, Navy, Marines and Air Force chiefs expressed opposition Tuesday to removing the chain of command from sexual assault investigations, as Senate Armed Services Committee Chairman Carl Levin said there may be public confusion about the military's reporting process.

Legislation has been introduced to give responsibility to military prosecutors, instead of commanders, in these probes.

Referring to media reports that there is only one way to report sexual assault, Levin, D-Michigan, asked each of the military heads at a hearing if there currently are multiple options in addition to notifying a unit commander.

The three generals and admiral all replied yes. They also told the committee that instances of commanders ignoring their judge advocate generals' advice in sexual assault cases are extremely rare.

Sen. James Inhofe, the top Republican member of the Armed Service panel, earlier called sexual assault in the military "an enemy to morale and readiness," and urged his colleagues to tread carefully in tackling the issue.

Inhofe said he is opposed to any legislation "removing commanders from their indispensable roles" in the military justice system, and noted that military and civilian courts are different animals because members of the military do not enjoy the same rights as civilians.

"There's a risk of unintended consequences if we act with haste without thorough and thoughtful review," the Oklahoma lawmaker said.

The congressional committee called the unprecedented hearing, which includes testimony from the Joint Chiefs of Staff and top military lawyers, after Sen. Kirsten Gillibrand, D-New York, introduced legislation that would remove the chain of command from the process victims go through to get their claims heard

Gen. Martin Dempsey, chairman of the Joint Chiefs of Staff, Gen. Raymond Odierno, chief of staff of the Army, Adm. Jonathan Greenert, chief of naval operations, Gen. James Amos, commandant of the Marine Corps, and Gen. Mark Welsh, chief of staff of the Air Force, each acknowledged that sexual assault is a serious problem, but one that commanders are equipped to handle.

They all used their opening statements to the committee to express opposition to Gillibrand's proposal.

"These crimes cut to the heart of the army's readiness for war. They destroy the very fabric of our force -- soldier and unit morale," Odierno said.

But while there may be derelict commanders, he said, those are anomalous, and the chain of command must be "fully engaged and at the center of any solution" to the issue.

Gillibrand was undaunted, emphasizing that commanders would be removed from the process only for the most serious crimes, such as rape and murder.

She said she agreed with the military chiefs that "the chain of command is essential for setting the climate," but there is a difference between setting a tone and dealing with serious crime, especially when one of the compounding factors in reporting sexual assault is a lack of trust.

"They fear retaliation. They fear being blamed," Gillibrand said of the victims. "That is our biggest challenge right there."

She said there are commanders who are not objective, who don't want women in the military in the first place, who don't know the difference between a "slap on the ass and a rape."

Sens. Lindsey Graham, R-South Carolina, and Claire McCaskill, D-Missouri, also previously expressed concerns about the military lumping all sexual assault and sexual harassment cases under the category of unwanted sexual contact, rather than breaking out the more severe crimes of rape, sodomy and assault.

Gillibrand added that the United States should follow the leads of allies like Israel, Australia and the United Kingdom, all of which, she said, have taken the chain of command out of investigations into serious crimes.

Dempsey replied that he'd prefer a "constellation of checks and balances" to help empower commanders and hold them accountable. Amos said he'd be open to removing the chain of command if he thought it would work, but he had no proof it would.

Earlier in the hearing, Dempsey said he has seen numerous proposals that have merit. Among them: prohibiting people convicted of sexual assault from joining the military, administrative discharges for those convicted while serving, requiring commanders to report all claims up to the next commander, and increasing the transparency of commanders' actions.

"Our goal should be to make commanders more accountable," he said.

Pressed later by Sen. John McCain about whether there are sufficient regulations to prevent convicted sexual predators from joining the military, Dempsey said no.

Presently, there are "inadequate protections for precluding that from happening, so a sex offender could, in fact, find their way into the armed forces of the United States," Dempsey said.

Amos said that the Marines have tackled scourges in the past -- racism after World War II and drug use after the Vietnam War -- and prevailed. Discipline and behavior problems must be handled from the top down, he said.

"A unit will rise or fall as a direct result of the leadership of its commanding officer," Amos said. "They should never be forced to delegate their authority."

He further said that in 43 years, he couldn't think of an incident in which he opposed a judge advocate general's recommendation to prosecute, but he recalled many incidents in which he had ordered prosecution when the JAG advised against it.

Levin appeared to agree with the military chiefs, saying, "The chain of command has achieved cultural change before. For example, two generations ago when we faced problems with racial dissension in the military, and more recently, with the change to the 'don't ask don't tell' policy, and the chain of command can do it again."

Gillibrand wants to give military prosecutors -- rather than commanders -- the power to decide whether cases are investigated because, she asserts, the current system opens the victim up to retaliation. Gillibrand and others feel commanders cannot be impartial figures, especially if both the victims and perpetrators are under their command.

"When we just talk (to victims) informally, they tell us they don't report because they are afraid of retaliation, being marginalized, having their careers end or being blamed," Gillibrand has said.

The military has been hit hard over the issue of sexual assault among its ranks, with the Defense Department reporting an estimated 26,000 cases of unwanted sexual contact, ranging from rape to groping, in 2012. That was a 35% jump from 2010, the Defense Department said.

The report prompted President Barack Obama, during May 24 commencement exercises at the Naval Academy, to tell graduates, "Those who commit sexual assault are not only committing a crime, they threaten the trust and discipline that makes our military strong."

A handful of recent high-profile incidents have brought this issue to the forefront:

-- An Army sergeant first class assigned to the sexual assault prevention unit at Fort Hood, Texas, is being investigated for alleged sexual assault, pandering, abusive sexual contact and maltreatment of subordinates.

-- In early May, an Air Force officer who worked with an assault prevention unit was charged with sexual battery after being accused of grabbing a woman and groping her buttocks and breasts in a parking lot not far from his Washington office.

-- Three U.S. Naval Academy football players are under investigation in an alleged sex assault involving a female midshipman at an off-campus "football house" party in April 2012, according to a Defense Department official. The victim says she learned from friends and social media that the players claimed to have had sexual intercourse with her while she was intoxicated, her lawyer said.