The club confirmed the news in a statement on their website, meaning O'Neill's tenure ends after just 15 months in charge.
The Black Cats lie in 16th place in the Premier League following the latest loss, with 31 points from as many games. Wigan and Aston Villa lie a point behind with a game in hand.
"The club would like to place on record its thanks to Martin and wishes him well for the future," Sunderland said.
"An announcement will be made in the coming days regarding a successor."
O'Neill took charge of Sunderland, the club he supported as a boy, in December 2011. The club enjoyed a bright start under his tenure, but a poor run of form towards the end of the season saw the team finish down in 13th.
The Northern Irishman was backed with money in the summer, signing Adam Johnson (£12 million) and Steven Fletcher (£12m rising to a potential £14m) amongst others. But after a season of underachievement and seven matches of a disappointing campaign remaining, the club has chosen to axe their manager.
O'Neill, after the match against United, had spoken of his optimism to negotiate their way through the relegation battle in the weeks ahead.
"I am as buoyant now as - I was nearly going to say as I have always been," O'Neill told reporters.
"What is it - the half-full or the half-empty glass? I'm even going to the three-quarters full. There's a real determination in the dressing room. The players are obviously disappointed, but they can take a lot from the second half. Yes, there is a determination about the players in there that we can do it."