The 38-year-old backed the controversial decision to rest Mitchell Starc from the Boxing Day Test, insisting Cricket Australia was doing the right thing by managing and monitoring bowlers' outputs. Ponting was heavily consulted on the pros and cons of a rotation system when in charge of the team, and supports the continued use of the policy.
"It actually annoys me a little bit," he said of the criticism. "There are a lot of people who are talking about it that probably don't need to be talking about it, to tell you the truth."
He added: "A lot of the people that are talking about it - past bowlers, in particular - have got no idea of the workload of some of the guys that are playing now.
"People from the outside can make opinions on it all they like. But until it starts coming from within, then people should just mind their own business I reckon."
Ponting explained the thinking behind the rotation system was simple - to ensure fast bowlers were at their peak each time they played.
"Everyone in and around the Australian team is trying to do the best thing for Australian cricket," he said.
"They're not doing anything to try to harm anybody or make the team weaker. It's all about trying to put the best and fittest bowlers on the park every time we can."
England wicketkeeper-batsman Matt Prior, who arrived in Sydney on Thursday to play in the Big Bash League, is another fan of the rotation policy.
"If you try to get those guys to play every day, I think you're going to probably shorten their careers," he said. "You've got to look after your assets like you would in any business."