Button was running fifth and suspects he could have outfoxed Mercedes' Lewis Hamilton and Nico Rosberg by the finish had he made a three-stop strategy work.
The Mercedes ultimately pitted four times and cruised home under team orders nursing tyre wear and fuel shortages.
Button lost nearly two minutes when he was released early from his third pitstop and had to be dragged back for a wheel to be fitted correctly. He was running 12th when he eventually retired with a severe vibration.
"It's easy to say I could have been on the podium," said Button. "I'm sure people will disagree with me, but I think we had a chance to fight with Mercedes.
"We had one less pitstop to do. I don't know what might have happened, but at worst it would've been fifth and we would've fought against Mercedes."
The 2009 world champion also believes McLaren made substantial progress in Malaysia, having only finished ninth in Australia seven days earlier.
"It's a good improvement on the last race," Button said. "Only a week ago we never would've thought we'd be racing for fifth.
"There are positives, but it's always very tough when you have issues like this.
"It's a lot of points to throw away, but we are improving and I can see we can be even more competitive in China."
Team-mate Sergio Perez managed ninth despite a last-minute fourth tyre stop.
"I think we managed to get the best out of the car," he said.
"Eighth was possible. We stopped very late as the tyres gave up."