The allegations arose after Tuesday's publication of two damning reports into the management and conduct of the swimming team, which produced Australia's worst results in two decades at last year's Games.
"To establish the right culture, we have to investigate these allegations and deal with them appropriately by putting in place the right framework," Swimming Australia president Barclay Nettlefold said in a news release on Thursday.
"I will be encouraging the panel to look at each allegation and we want to stop talking about rumour and act on the facts of what did or did not actually occur.
"We will be decisive, we will be firm and we will discipline athletes, coaches and staff accordingly, where such action is deemed appropriate and necessary, but we must get the facts first and follow the right processes to do that."
The panel, comprising former Australian Rugby Union chairman Peter McGrath and three members of the SA board, will start work on their investigations immediately.
Nettlefold said swimmers, coaches and staff would be encouraged to come forth with information to aid the inquiry.
SA later said that six members of the men's 4x100m freestyle relay team had come forward to discuss a team bonding session at a training camp in Manchester before the Games.
The squad, comprising James Magnussen, Matt Targett, Eamon Sullivan, James Roberts, Cameron McEvoy and Tommaso D'Orsogna, arrived in London confident of winning gold but ended up fourth in the final.
Australia's swimmers are usually expected to lead the country's charge for medals in the first week of the Games but they failed to win an individual title and ended up with just one relay gold, six silver and three bronze medals.
The Bluestone review of team culture at the London Games alleged that slack management had allowed a "culturally toxic" environment to develop among Australia's swimmers.
An Independent Swimming Review into the high performance programme at Swimming Australia commissioned by the Australian Sports Commission also reported on Tuesday and made 35 recommendations for improvements.
One of the more serious allegations to emerge after London was that some team members had been subjected to initiation rituals involving Stilnox - a sedative banned by the Australian Olympic Committee just before the Games.