Eva Risztov of Hungary held on to win the gold medal, ahead of the 24 other starters, after a desperate sprint finish.
Haley Anderson of the United States won the silver medal after making a late charge to catch Risztov while Italy's Martina Grimaldi won the bronze.
Payne won the world title in 2009 and 2011, won Olympic silver in Beijing and was considered one of the British swimming team's top medal shots in the 10km event.
"I just didn't go my way from the start," she admitted.
"They played me at my own game by leading out from the front, to be fourth in the world is not too bad but it's not what I wanted.
"I was hurting for most of the swim, I just was working harder than I would normally like to take the pace of the race. I just didn't have enough at the end unfortunately.
"It was pretty violent, and I'm a lover not a fighter, at the start with lots of kicking but that's no excuse.
"I need a bit of a break now and I've got a wedding to plan in two months, which is going to be my focus now. It's not the last of me for swimming."
Risztov, a former Olympic pool swimmer, who quit the sport in 2005 then made a comeback in open water in 2009, led for all but one of the six loops the competitors were required to swim.
She opened up a gap on the leading pack at the start of the final loop and was three body lengths clear approaching the finish when Anderson started to reel her in.
Anderson, whose older sister Alyssa won a gold medal at London as part of the American 4x200 metres freestyle relay team, drew level with Risztov as they approached the finish but the Hungarian kicked again and got her hands on the pads first in one hour, 57 minutes and 38.2 seconds, just four tenths of a second in front of Anderson.
Grimaldi, runner-up at last year's world championships in Shanghai, won the bronze medal by the same tiny margin ahead of Payne.
Payne usually leads races from the front but clearly found the furious pace set by Risztov too tough to handle.
Five swimmers, including Payne, started to make a break from their rivals as they came to the end of the penultimate of six laps around the Serpentine in central London
Risztov took them across the line with Grimaldi, Anderson, Germany's Angela Maurer following, just ahead of Payne.
Tens of thousands of people lined the banks of the lake on a glorious summer's day for one of the few events at the Games that were free for the public to watch.
They let out a mighty roar when the starter's gun was fired and the swimmers plunged from the pontoon into the murky water that they had to share with geese, ducks and swans, thrashing through thick weeds and poisonous algae that can cause severe skin rashes, eye irritation, vomiting and diarrhoea.
The exhausting race took its toll on two competitors, South Africa's Jessica Roux and Brazil's Poliana Okimoto, who were both hauled from the water before the end.
- Martina Grimaldi