The Istanbul club said they had terminated Terim's contract after he declined to accept a proposed two-year extension to his term.
In a statement on its website, Galatasaray said "recent developments" had started to damage the club's "values", without elaborating. Turkish media have been reporting growing discord between Terim and the club's management.
"First and foremost, our board of directors are responsible for protecting these values and passing these on to future generations," the club said in the statement.
"As part of this responsibility, in order to prevent further deterioration of our community, our board of directors decided unanimously to notify Mr. Terim that his contract has been terminated," it said.
Terim had only recently begun his third stint at the helm of the Turkish national team after they departed with Abdullah Avci, but he was only expected to job-share until the end of their current World Cup campaign when his country would appoint a permanent successor.
The legendary Turkish manager leaves Galatasaray sitting in 11th place in the Super Lig, albeit with a game in hand after their clash with leaders Besiktas was abandoned after the pitch was invaded by hundreds of fans.
Terim led Galatasaray to the Champions League quarter-finals last season, with major summer signings Didier Drogba and Wesley Sneijder playing an influential role, but last week they were thrashed 6-1 at home by Real Madrid in the same competition.
Roberto Mancini has emerged as the early front runner to fill the vacant managerial position with several Italian and Turkish sources claiming he's already in Istanbul for negotiations, according to Football Italia.