The Germans, who replaced Anna Friesinger-Postma after her prone sliding finish in the semi-finals with Katrin Mattscherodt in the final, clocked three minutes 2.82 seconds to defend their title.
Japan, using skaters Masako Hozumi, Nao Kodaira and Maki Tabata, crossed the line of the six-lap, 2,400m race in 3:02.84.
Friesinger-Postma celebrated her fifth Olympic medal dating back to 1998 in Nagano, thrusting her arms in the air and joining her team mates for a victory lap around the Richmond Olympic Oval ice with two German flags stretched between them.
The 33-year-old German nearly scuppered her team's chances when she lost her balance around the last turn and fell to the ice when her legs gave out in the semi-final against the US.
After frantically paddling her way through the last 10 metres, she pounded the ice in disgust thinking she had caused a defeat that eliminated their golden hopes, before looking up in astonishment at a 0.23 second victory.
"In the second last corner I lost my groove and had to catch up and then in the last metres I slipped again," the 33-year-old said.
"I thought I destroyed everything. I kicked my leg because I know the transponder is on my leg for the timing.
"I thought I had messed it up but you see, impossible things can happen," she said. "We were just so happy after this."
Friesinger, who had a tough year plagued by ankle and knee injuries and contracting swine flu, said her second consecutive Olympic gold medal in the team pursuit was special.
"This medal is so valuable for me," she said, stroking it and looking at it. "Everyone sort of expects you to win it but winning one of these is so hard."
The German, who has now won at least one medal in the past four Olympics, had already said this was her last Games.
"I have problems with my knees so no, no, no," she said when asked whether the Sochi 2014 Games was a possibility for her.
The bronze medal went to Poland, with Katarzyna Bachleda-Curus, Katarzyna Wozniak and Luiza Zlotkowska, who beat the United States team by 1.57 seconds when American Catherine Raney-Norman faded badly at the finish.
Poland's skaters carried their flag as well and in a good-natured tribute to the German veteran, they all dropped to the ice and slid on their bellies in a humourous salute.
The Japanese showed their joy at winning silver by holding hands and leaping on to the podium together.