Six-times French Cup winners St Etienne, who already beat Paris St Germain on penalties after a goalless draw in the quarter-finals, had not featured in a final since 1982.
They will face Stade Rennes or Montpellier at Stade de France on April 20.
St Etienne, who had the best chances throughout, were on the brink of being eliminated but Lille's midfielder Florent Balmont missed his penalty to keep them alive. They eventually won 7-6 after Idrissa Gueye's attempt flew well above the crossbar.
"We've been waiting for it for a long time," St Etienne captain Loic Perrin told French channel France 3.
"We could have won this thriller before but it's a very happy end. It's wonderful. Now, we have to lift the trophy."
St Etienne could have secured their place in the final much earlier if Lille's goalkeeper Steeve Elana had not produced a string of impressive saves.
On a freezing cold evening the hosts kept Elana busy, the goalkeeper saving Fabien Lemoine's powerful effort from just inside the box in the 26th minute.
He was then fortunate to see striker Brandao, who has won the last three League Cups with his former side Marseille, head a Yohan Mollo cross just wide three minutes later.
St Etienne pushed harder, Renaud Cohade's half-volley cleared by the goalkeeper, who was again at his best in extra-time when he parried a second Cohade effort in the 105th minute.
However, he could not carry on the run in the shoot-out as he failed to save a penalty with St Etienne deservedly going through.