But the 25-year-old is not about to go out with a whimper and has his sights firmly set on next month's World Championships to get the ball rolling.
Fogg had a busy schedule at the London 2012 Olympics last summer, reaching the final of the 1500m freestyle before heading outside to finish fifth in the 10km open water event.
He is continuing to juggle both but his London 2012 open water result and a maiden World Cup win in Mexico in April has him believing his best chance of medalling at the Rio 2016 Games is outside.
Fogg was officially selected for the open water event at the Barcelona World Championships last month but admits he won't be turning his back on the 1500m freestyle anytime soon.
"My result at London 2012, finishing fifth in the open water, and then winning gold in Mexico just reaffirmed to me that my future lies not in the 1500m down the line," said Fogg.
"Don't get me wrong, I still love doing the 1500m and will continue to give it my all, but my best chance of a medal at Rio probably lies in the open water.
"This means that I will probably have to focus and concentrate fully on the open water nearer the time, but before that I will continue to give the 1500m my all.
"Getting to the 1500m final at London 2012 was a great result and that is what I will be looking to do again at the worlds in the summer.
"I can't say for sure but there is a chance this could be my last World Championships for the 1500m and I want to go out on a high if that is possible."
While his open water place is secured, Fogg still has to head to Sheffield for the British Gas Swimming Championships at the end of the month to earn his 1500m freestyle place.
The 10km open water and 1500m freestyle events are just over a week apart at the World Championships in Barcelona but Fogg insists nothing will stop him aiming for a final in the latter.
"I am used to and I did both at the worlds in 2011 so I should be fine," said Fogg, who you can go along and watch in Sheffield with tickets being available from the British Swimming website.
"Ten days or so is not a lot of time to recover because the open water swim does take a lot out of you and you need to rest up.
"But I am used to it and I really want to build on what I did at the Olympics so that is my aim."