Devils center Ryan Carter lit the lamp from a feed by Stephen Gionta with 4:24 remaining to push New Jersey ahead, and Zach Parise removed any suspense by scoring an empty net goal with just over 30 seconds left.
The victory lifted New Jersey to a 3-2 lead over their cross-river rivals in the best-of-seven Eastern Conference finals and they will have the chance to seal a place in the Stanley Cup when the series returns to New Jersey on Friday.
"I think giving up a three-goal lead and coming back and winning the game late shows a lot," said veteran Devils goaltender Martin Brodeur. "I thought we stayed composed."
The Devils are aiming to return to the finals for the first time in nine years. The Los Angeles Kings have already booked their place in the Stanley Cup finals after wrapping up the Western Conference title in five games over the Phoenix Coyotes.
Gionta, who joined New Jersey late in the season from their American Hockey League affiliate, got the Devils going, flipping a rebound past Rangers netminder Henrik Lundqvist less than three minutes into the game.
Ninety seconds later, Patrik Elias deflected in another goal before New York had registered a shot on Brodeur.
The Garden crowd were stunned into silence when Travis Zajac's blistering slap shot on a three-on-two break got past Lundqvist low on the stick side for a 3-0 Devils lead with 10:11 remaining in the opening period.
That rocket seemed to wake up the Rangers players, who upped the tempo and intensity.
The East's top seeds got a goal back from Brandon Prust, returning after a one-game suspension for elbowing, at 15:41 of the first period, and 32 seconds into the second Ryan Callahan cut the Devils' lead to 3-2.
In complete control, the Rangers scored the equalizer through Marian Gaborik 17 seconds into the third, sparking derisive chants of "Marty, Marty" at netminder Brodeur, who left the puck out wide of the crease for his defenseman only to see Gaborik scoop it up.
The Devils would have the last laugh however as they regained their resolve and turned the tables on the Rangers.
Gionta, whose older brother Brian won a Stanley Cup with the Devils in 2003, raced to the corner to win the puck along the boards and centred to Carter, who rushed in just ahead of Carl Hagelin and put the game-winner past goalie Lundqvist.
Parise then wrapped up the scoring in the final minute.
"It was an adventure," Devils coach Peter DeBoer said. "Kind of a tale of two or three games out there."
The 40-year-old Brodeur, who owns three Stanley Cup rings, was eyeing a tantalizing return to the NHL title series 17 years after his first trip to the Stanley Cup championship.
"It's a good feeling," he said. "I think we worked really hard to get in that position. I think we're really proud for the guys."
Rangers coach John Tortorella bemoaned his team's sluggish start but credited his players for their fierce fightback in outshooting New Jersey 28-17 for the game.
"We're always looking to try to start the game strong," he said. "We had a tough start. Tough luck for them a little bit on a couple of them. But I thought we probably played our best game of the series."