The Daggers went in front in the 26th minute thanks to Zavon Hines' header but the Shrimpers hit back at the start of the second half with substitute Cauley Woodrow scoring twice in three minutes.
Yet the visitors duly netted four goals in just eight minutes at Roots Hall to wrap up the tie - two of which coming from errors by goalkeeper Paul Smith.
Southend, who remain under a transfer embargo and are unable to bring in new players, have now triumphed just once in eight games.
"When we needed character and resilience we just didn't have any," Brown said.
"I thought we actually edged the first half despite being behind and it all looked good when Cauley came on and got a couple of goals.
"After that we made some individual errors which I'm not going to moan about because they happen.
"However, I will be studying the recording of the game long and hard to get to the bottom of what went wrong."
But Brown refused to criticise 'keeper Smith, who was barracked by his own fans.
"I'll take the blame for Paul's display," Brown said.
"It was his first game of the season and he did show signs of being rusty but I'll take the blame for that with my selection of him."