India had been banned since December 2012 after elections at its Olympic association (IOA) saw a corruption-tainted official voted in as secretary general.
The IOC lifted the ban on Feb. 11 following fresh elections last week, allowing it to be represented at the Olympics and bringing the total number of nations competing at Russia's first Winter Games to 88.
"Very, very emotional," new IOA President N. Ramachandran, who is also heading the World Squash Federation, told Reuters Television.
"For us after having been out of the IOC and the Olympic movement for 15 months to come back to it and to come back to it within two days of our elections, I just couldn't believe it."
The Indian delegation arrived at the ceremony at the mountain athletes' village amid loud cheers.
"When the IOC lifted our ban in the midst of an Olympic event, it's the first time it's happened and I am truly grateful to the IOC President Thomas Bach ... for having lifted the ban within two days," Ramachandran said.
India's three athletes had marched under the Olympic flag at the opening ceremony but can now carry their own standard on Feb. 23 when the Games come to an end.
"Well it's like coming to the Olympics all over again so this is the real one," said luger Shiva Keshavan, who has already competed.
"Unfortunately my event is over but you know it is important that India is back and all of us are very happy about that. I hope something like this doesn't happen again in the future," Keshavan told Reuters after the small ceremony that also included a few visiting Indian fans.
Two of the three athletes, including Keshavan, are currently Olympic scholarship holders.
The world's second most populous nation became the first to win readmission to the Olympic family during an ongoing Games.
"All the Indians are very disappointed because you know we have had so much support from our own country, some people have come from so far ... and not to see the flag there, just because some people put their personal interest before was very disappointing."
"But now it's time for a fresh start and I hope there is better days to come for winter sports," he said.
The final athlete to compete in Sochi is Alpine skier Himanshu Thakur, who will take part in the giant slalom.