The French qualifier forced a tiebreak after saving two match points on his serve in the 12th game of the third set.
In the tiebreak Roger-Vasselin recovered from 6-4 down to win the last four points, sealing victory when Dolgopolov's forehand went long.
"(Dolgopolov) just didn't play well on those match points. He looked a bit tense out there," the 28-year-old, who reached his third quarter-final of the year, said.
Roger-Vasselin now takes on giant Croatian Ivo Karlovic, who edged out Russian Alex Bogomolov Jr 7-6 6-4 after serving 16 aces.
Seventh seed Carlos Berlocq of Argentina was also knocked out while Italy's second seed Andreas Seppi and fourth-seeded Brazilian Thomaz Bellucci advanced.
Seppi beat Dutchman Igor Sijsling 7-5 6-2 and Bellucci eliminated Italian Flavio Cipolla 6-4 6-1.
In the ATP Stockholm event, top seed Jo-Wilfried Tsonga disposed of Japan's Go Soeda in straight sets, the Frenchman winning 6-2 7-6 to move into the quarter-finals.
The result boosted Tsonga's hopes of clinching a spot at the ATP Tour finals in London. He currently occupies the eighth and final place for next month's season-ending spectacular.
Second seed Tomas Berdych also eased through, getting past Juergen Zopp of Estonia 6-1 6-4. The Czech served 11 aces and saved the only break point he faced as he won in an hour and 11 minutes.
Second seed in the ATP Vienna event Janko Tipsarevic moved into the third round with a 6-3 6-4 win over Ernests Gulbis.
The Latvian, a wildcard, was no match for the Serbian's rallying despite sending down four aces to Tipsarevic's none. While both players got in 62 per cent of their first serves, Gulbis was only able to win 59 per cent of those points while his opponent took 73 per cent of his.
Third seed Tommy Haas of Germany also had no problem in disposing of American Jesse Levine 6-4 6-2.
ATP Moscow results
Edouard Roger-Vasselin (France) beat 1-Alexandr Dolgopolov (Ukraine) 6-3 5-7 7-6(6)
4-Thomaz Bellucci (Brazil) beat Flavio Cipolla (Italy) 6-4 6-1
2-Andreas Seppi (Italy) beat Igor Sijsling (Netherlands) 7-5 6-2
Jerzy Janowicz (Poland) beat 7-Carlos Berlocq (Argentina) 6-3 6-4
ATP Vienna results
3-Tommy Haas (Germany) beat Jesse Levine (U.S.) 6-4 6-2
2-Janko Tipsarevic (Serbia) beat Ernests Gulbis (Latvia) 6-3 6-4
Marinko Matosevic (Australia) beat Dominic Thiem (Austria) 6-3 7-6(5)
Paolo Lorenzi (Italy) beat Vasek Pospisil (Canada) 6-4 6-4
Aljaz Bedene (Slovenia) beat Donald Young (U.S.) 7-6(4) 6-7(2) 6-3
Grega Zemlja (Slovenia) beat Matthew Ebden (Australia) 7-5 6-4
ATP Stockholm results
1-Jo-Wilfried Tsonga (France) beat Go Soeda (Japan) 6-2 7-6
2-Tomas Berdych (Czech Republic) beat Juergen Zopp (Estonia) 6-1 6-4
Sergiy Stakhovsky (Ukraine) beat 5-Feliciano Lopez (Spain) 7-6(5) 7-5
7-Marcos Baghdatis (Cyprus) beat Alejandro Falla (Colombia) 6-0 3-0 (Falla retired)