The 17-time grand slam winner and top seed, seeking his first singles gold medal, had three match points in the second set but then suddenly crumbled, hitting shot after shot into the net.
The tense battle revived memories of the pair's first round clash at Wimbledon in 2010, when Federer recovered from two sets down to beat the gritty Colombian.
"I've struggled against him in the past at times," Federer told reporters after the match.
"All of a sudden I missed the match points, things got difficult and he played a great match to come back, so I'm relieved of course. Falla is a great player, counter-punches well, plays really well and made things really difficult for me."
Returning to Centre Court less than three weeks after winning his seventh Wimbledon title there, Federer made a solid start, breaking his opponent to secure the first set 6-3.
The second set began in a similar vein, and at 5-3 Federer looked on track to wrap up the win. But a string of errors saw Falla seize the opportunity to fight back, saving three match points and breaking the 30-year-old twice to take the set and force a decider.
Normal service appeared to be resumed on the sun-drenched court at the start of the third set, as the Swiss broke his opponent in the opening game. But the world number 51 swiftly broke back to level at 2-2.
Federer, who won doubles gold in Beijing with Stanislas Wawrinka, broke again to go 4-3 up, then held serve to force Falla to serve to stay in the match.
Buoyed by chants of "Let's go Roger, let's go", Federer didn't let the chance slip away from him this time, looking decidedly relieved to secure his victory.
In the next round, Federer will meet Julien Benneteau who he narrowly beat in the third round of Wimbledon this year after losing the first two sets to his French opponent.
However, tennis fans did not have to wait long for the first upset of the Olympics with sixth seed Tomas Berdych losing 6-4 6-4 to Belgian world number 75 Steve Darcis at the opening match on Centre Court.
As the Czech approached the net to shake the winner's hand he pulled a slightly bemused look as if to say "What happened there?" before congratulating his grinning opponent, who had never before played on the 15,000 capacity court.
"It has always been a dream to play (on Centre Court). When you're young you watch Pete Sampras playing there and you think maybe one day, and today it was my day," Darcis said.
"It was a great atmosphere. To play those matches is why you play tennis and today it was an amazing feeling."
Having won the first set and gone 5-3 up in the second to serve for the match, Darcis wobbled to hand his opponent a break.
Berdych's change in fortunes looked set to continue as Darcis, serving for the match a second time, made a string of sloppy errors to go 15-40 down, hanging his head in frustration.
But the crowd, half-empty as many took the opportunity to soak up the atmosphere around the venue before watching Roger Federer and Serena Williams later, got behind the underdog and with a lot of clapping and some cries of encouragement the 28-year-old clawed his way back to deuce.
The pair battled it out in a long rally, which Darcis ended with a classy cross court backhand to gain match point before Berdych powered a shot long to gift the Belgian the match.
Darcis will face Colombia's Santiago Giraldo in the second round.
"I played him in Munich on clay this year and I lost already, but you know I just beat Tomas," said Darcis.
"I'm playing good, I'm confident, so why not Santiago, he's a great player and it will be very tough but I will do my best and see how it goes."
In the doubles, the Murray brothers lost their first round match to Austrian duo Jurgen Melzer and Alexander Peya, eventually falling 5-7 7-6(8-6) 7-5.
