Schaaf, 52, will be replaced by assistants Wolfgang Rolff and Matthias Honerbach for the last two weeks of the season.
"I had an outstanding time with Werder," said Schaaf, "connected to many positive experiences and great successes. I want to say thank you to all, who joined and supported me on my way. I wish Werder Bremen all the best for the future."
Schaaf was a one-club man, playing for the club as a youth before joining the professional ranks in 1978. The defender was assistant manager while still a player, retiring in 1995.
He took over as the manager of the second team that same year, before being given the permanent job in 1999.
Werder won the double under his tenure in 2003-4, and won the German Cup three times and League Cup once under his tenure.
But their fortunes have declined after a run of five straight appearances in the Champions League last decade, and this season the club lie a disappointing 14th with one match remaining in the Bundesliga.
Sporting Director Thomas Eichin paid tribute to his service to the club, saying he defined Werder Bremen:
"As announced, we’ve analysed our sporting development and came to the conclusion, that we’d like to start off new," Eichin said.
"Thanks to Thomas, for everything he did in more than 40 years as member of Werder Bremen as player and coach. The club celebrated its biggest victories with him. He defined Werder.”