Mahendra Singh Dhoni’s side posted a target of 152 for six, knowing that only a significant win would do after Pakistan beat Australia in the day’s first match in Group 2.
India had to restrict South Africa to 121 in order to sneak through to the semi-finals, but AB de Villiers’s men passed that total and were eventually bowled out for 151 to fall agonisingly short with just one delivery remaining.
The result sees Australia and Pakistan progress from Group 2 of the Super Eights with the two sides set to take on West Indies and Sri Lanka, respectively, in the semi-finals.
South Africa, who ended pointless after the second group stage, were left crushed by their three defeats.
"Disappointed isn't the word to describe it, we've had a poor tournament," de Villiers said. "We've let the country down and that's what hurts most. We did all the preparation possible, we thought we were ready, but the guys didn't click."
Dhoni, meanwhile, appeared surprisingly upbeat despite his side's shock exit: "If you take the Australia game out, where the rain intervened, we performed brilliantly and can't have control over how other teams played.
"Overall I'm quite happy with how the team performed. There were instances where we should have had 10-15 runs more on the board. We lost too many wickets early."
Gautam Gambhir set the tone with a faltering 12-ball eight as he was bowled by Morne Morkel, and Virender Sehwag followed for 17 as he played all around a straight ball from Robin Peterson.
It was a very sluggish start from India, and the dangerous Virat Kohli was dismissed for just two as Jacques Kallis had the number three caught behind by de Villiers.
Rohit Sharma helped to rebuild alongside the belligerent and thrilling Yuvraj Singh, who launched two huge blows into the stands for maximums in a resurgent middle-over spell.
Sharma was trapped lbw by Peterson for 25, however, and Yuvraj saw his woodwork scattered as Morkel struck for his second wicket to see his side back into the ascendancy.
Suresh Raina compiled a breezy 45 off 34 balls before he was run-out by Johan Botha, and captain Dhoni was left unbeaten on 23 with his side six down after their 20 overs.
India knew that they would have to restrict South Africa to 121 in order to secure a semi-final berth, and it all looked to be going their way as the prolific Hashim Amla was dismissed for a second-ball duck by Zaheer Khan with Virender Sehwag taking a sharp catch at slip.
Kallis departed for just six as Irfan Pathan claimed a valuable scalp in the fourth over, and captain de Villiers could only muster a run-a-ball 13 as Yuvraj breached his defences to break his stumps.
It was all India, until Faf du Plessis looked to have taken the game away from Dhoni’s men with an inspired 65 off 38 balls with two sixes and six fours.
Du Plessis saw wickets tumble at the other end, but he had made a serious dent in the required runs, while there were fleeting contributions from JP Duminy (16) and Farhaan Behardien (13).
Yuvraj eventually accounted for du Plessis, while Lakshmipathy Balaji and Zaheer dismissed Duminy and Behardien, respectively as India forced their way back into the game.
Peterson (10) was bowled by Khan, who finished with three for 22 from his four overs, and Albie Morkel was left to attempt big shots at the death with only a faint hope of victory.
But by this time, South Africa had moved comfortably past 121, and India’s stay in the competition was ended as Pakistan supporters celebrated wildly in the stadium – their team had progressed through to the semi-finals.
Both Morkel brothers were dismissed in a dramatic final over with 14 runs required off it, and two towering sixes were not enough for South Africa, who were bowled out with a ball to spare just two runs shy of their target.
Australia and Pakistan progress through to the semi-finals, while for South Africa and India, the inquests into their hugely disappointing campaigns will now begin in earnest.