Brazil was rocked by a wave of protests when it staged last month's Confederations Cup, considered a dry run for next year's World Cup.
Among the protesters' many grievances was the amount of public money being spent on new stadiums when local public services such as health and transport are in a dismal state.
Rio de Janeiro will stage the 2016 Olympic Games and has already suffered one major embarrassment with the athletics stadium, which was completed only in 2007, having to shut for 18 months for repairs to the roof.
"We have to explain very clearly to the public that the investments made in the Olympic Games are for a sustainable legacy for generations to come," Rogge said in a conference call.
"We must make it clear that, yes, the games are a force for the good, that they improve the society and this has to be explained, because people don't know what the investments are."
He said it was up to the host country to decide how to fund Olympic Games building.
"This is a sovereign decision of any government that organises the Olympic Games," said the Belgian, who leaves office in September.
"As you know, the investments are not for the short term, it is not for the two weeks of the Olympic Games or World Cup, they are for generations to come, whether it be bus lines, or improvements to the airport."
Rogge shrugged off concerns about the slow pace of preparations for the Games.
"There is a lot of good work being done, there are a lot of things to be achieved before 2016, we have urged them to accelerate but I remain optimistic and there is no concern whatsoever on the quality of the Games."
Rogge praised the recent reforms made in wrestling as the sport bids to keep its place at the Games.
The IOC surprisingly dropped wrestling from the list of core sports for the 2020 Games in February, forcing it to join seven other candidate sports battling for the one vacant spot.
Wrestling is now on a shortlist of three sports, along with squash and baseball/softball, and the winning sport will be chosen at the IOC Congress in Buenos Aires in September.
"I think that wrestling's leadership responded well to the challenge," said Rogge. "They improved the federation by giving athletes voting rights, rules have been changed.
"Whether this will be enough to give them selection for 2020 remains to be seen and I will be neutral among the three sports but they did a good job."
- Sports & Recreation
- 2016 Olympic Games