Pompey's Andy Barcham fired in a 20-yard thunderbolt after just 90 seconds but after the hosts missed a series of chances, the Shrimpers took advantage with late goals from Will Atkinson and Barry Corr to clinch all three points and move Southend up to third.
It was the fourth league win on the spin for Brown's men, but the manager was disappointed with the first-half display.
"I did not enjoy the first half as the last thing you want to give this big crowd is something to cheer about and they scored an absolute world-class goal," said Brown.
"To be honest we could have been two or three down at half-time and the turning point was when (David) Connolly went through one-on-one and he missed the target."
After Cauley Woodrow saw red for an elbow on Jed Wallace the visitors sprung into life and turned the game on its head.
Brown added: "Our finish to the game was phenomenal though and we played better football when we went down to 10 men. That is the only downside to the performance that we had to wait for that.
"I thought the decision for the red card was harsh and the referee was impetuous to get his red card out so quick.
"These lads are never down and it is a success story but I won't start thinking about being in the top three just yet as there is a long way to go."