In a close fight at the end of the session, Hamilton produced a 1:29.398 to beat Vettel's Red Bull to the top spot by 0.103 seconds.
Hamilton had held provisional pole on a 1:29.540s after the first Q3 runs, with Vettel going 0.039 quicker before the Mercedes responded and beat him.
Mercedes appeared to make a simple miscalculation with Rosberg, believing his early Q2 time - good enough for second when it was set - would be sufficient.
But the second half of the field was shaken right up when nearly everyone improved in the final seconds, and as the likes of Jenson Button, Daniel Ricciardo and Nico Hulkenberg squeezed into the top 10, Rosberg was pushed out.
Mark Webber put the second Red Bull third, ahead of Lotus pair Kimi Raikkonen and Romain Grosjean.
Ricciardo - whose Toro Rosso team-mate Jean-Eric Vergne was only 16th - continued his excellent qualifying form with sixth.
Ferrari opted out of the pole fight. Felipe Massa and Fernando Alonso ran medium tyres in Q3 rather than the faster softs and ended up on row four.
McLaren's Button followed the same tactic in ninth and did not do a Q3 flying lap. Hulkenberg reached Q3 for Sauber and followed suit.
Neither Force India reached Q2, with Paul di Resta 12th and Adrian Sutil a very disappointing 15th at home.
Sergio Perez looked in danger of getting knocked out in Q1 at one stage, so his eventual 13th place was not as bad as it might have been. His McLaren will share row seven with compatriot Esteban Gutierrez's Sauber.
Williams joined Caterham and Marussia in the bottom six.