Two dropped out because of hypothermia, according to official reports, as the intense cold and rain took its toll.
"It was the coldest day I've ever had on the bike," Spain's Juanma Garate, who is 37 years old and has been a professional since 2000, told Spanish television.
"We'd been racing in temperatures 30 degrees higher yesterday and it takes time to adapt to the cold.
"People think we've got some sort of race clothing that means you don't get wet and cold - they're wrong."
Alejandro Valverde, who is third overall, said conditions were "inhuman". The Spaniard, who lost time, said he "almost fainted" from the cold on one descent and was dropped by the other favourites.
Among the top names who abandoned were Italy's Ivan Basso, twice the winner of the Giro d'Italia, and Czech Roman Kreuziger, who was fifth in this year's Tour de France.
American team Garmin-Sharp, who lost two of their lineup, now have just five riders left in the race, while Lotto-Belisol of Belgium had three dropouts and are down to four.
More bad weather is expected on Sunday's second Pyrenean stage, with heavy rain and thunderstorms forecast as the peloton tackles four major mountain climbs before finishing at the Peyragudes ski station.
The Tour of Spain finishes on Sept. 15. Italy's Vincenzo Nibali is the overall leader.