Stand-in Parkhead skipper Joe Ledley opened the scoring in the fifth minute with a header before a mistake by Celtic left-back Emilio Izaguirre in the 13th minute allowed Esmael Goncalves an easy chance to level.
Anthony Stokes restored the visitors' lead in the 21st minute with a close-range header which, ultimately, was enough to put the Parkhead men into Monday's semi-final draw at Hampden.
After beating Celtic in the Scottish Communities League Cup semi-final at Hampden in January, Saints boss Lennon was looking for lightning to strike twice.
"It didn't but we came close to it," he said.
"I am very proud of the group of players.
"We got off to a poor start but to get ourselves back into the game was pleasing and then we lose another poor goal from another set-play.
"Graham Carey had a terrific free-kick and there was a save to match it and then Sam Parkin should have done slightly better and the ball fell to Lee Mair who hit the side-netting."