After dropping the first set, the 17-time Grand Slam title winner was on the brink of going out in straight sets at 3-5 down in the second, and later 3-5 down in the tie-break before winning four straight points.
Monfils then broke the Swiss former world number one in the fourth game of the decider.
"[It was] one of those matches that could have gone either way," Federer told the ATP tour website. "I think I was a little lucky in the second set to get back, to be quite honest. I played a solid tie-break.
"I had chances early on in the third set I think to make something happen. He kind of recovered, then started to serve a bit better again towards the end of the third set, which made it difficult for me. It's a bit of an up and down performance for me."
"I knew that I choked on this forehand at 5-4 (in the tie-break)," Frenchman Monfils - a former top-10 player whose career has been derailed by injury - said.
"Then I changed my ideas because I was maybe nervous, too. Even (though) I was tired, I tried to play more aggressively sometimes, going more for my shots and it worked.
"It's a good win for me. I feel a bit sorry for him because I know he's running for London. But it's tennis. He's going to have more opportunity (in) those weeks coming up."
It was Monfils's second victory over Federer in eight meetings, and leaves the Swiss battling to reach the end-of-season ATP World Tour Finals for the 12th consecutive year.
"I'm just going to focus on trying to play well. I'm just going to try to have a good tournament in Basel and Paris. In the end, if I get the invite (to London), I'll be there," Federer added.
Monfils will play top seed and defending champion Novak Djokovic in the quarter-finals after the Serb brushed aside Fabio Fognini.
Other matches saw wins for Rafael Nadal and Jo-Wilfried Tsonga go through, while David Ferrer was shocked by Florian Mayer, and Tomas Berdych lost out to Nicolas Almagro.
Shanghai Masters Men's Singles Round 3 matches on Thursday
1-Novak Djokovic (Serbia) beat Fabio Fognini (Italy) 6-3 6-3
2-Rafael Nadal (Spain) beat Carlos Berlocq (Argentina) 6-1 7-6(5)
Florian Mayer (Germany) beat 3-David Ferrer (Spain) 6-4 6-3
15-Nicolas Almagro (Spain) beat 4-Tomas Berdych (Czech Republic) 6-7(6) 6-3 7-6(4)
Gael Monfils (France) beat 5-Roger Federer (Switzerland) 6-4 6-7(5) 6-3
6-Juan Martin Del Potro (Argentina) beat 11-Tommy Haas (Germany)WO
7-Jo-Wilfried Tsonga (France) beat 12-Kei Nishikori (Japan) 7-6(5) 6-0
8-Stanislas Wawrinka (Switzerland) beat 10-Milos Raonic (Canada) 7-6(2) 6-4