Brown described Muir's decision to award Aberdeen a penalty in their crucial encounter on Sunday a "disgrace" and argued it had sent his team down. Niall McGinn netted the equaliser to consign Dundee to the Clydesdale Bank Premier League drop after Muir had pointed to the spot when Peter Pawlett fell theatrically as he ran past Lewis Toshney.
Aberdeen manager Derek McInnes has defended Pawlett, who has previously been banned for diving, and insisted there was contact, but Brown claims he would substitute any player who won a penalty in such a way for his team.
Brown, who recently signed a two-year contract, told BBC Radio Scotland's Sportsound: "I'd take him off right away. And if I feel as though the SFA would punish me for any outburst about any referee then I'd walk away from football.
"It's our livelihoods. It's the players' livelihoods. Players are going to lose their jobs, staff around Dens Park are possibly going to lose their jobs. It was a high-profile game, a massive game. Surely they could change the officials and get the top officials?"
Brown is unconcerned whether Pawlett faces retrospective action but he feels the Scottish Football Association should look at lessons from their own perspective.
"He is a young player and he's got previous but you have officials out there to deal with that," said Brown, who has lost just one of his nine SPL games in charge. "It was a blatant dive. He was diving with two feet before he has even come near the player.
"I'm not too concerned about the player, I'm concerned about the officials. The officials are the ones that have cost us the game. I have never seen a blatant a dive in a long time. The referee is 20 yards away, the assistant referee is just behind that. Surely he can see that?
"Dundee had an opportunity to take this down to another weekend and it was taken out of our hands by a diabolical decision. I'm still very bitter about it. Maybe the guy's okay, a decent referee, but he made a massive mistake.
"I've been happy with the referees. We had some dodgy decisions, in the cup game against Dundee United the ball was out the park and they got a free-kick and got a goal from it. I've not complained about too many referees, in general they have been spot-on. I had a gripe about yesterday because it was a major decision."