The Shrimpers secured a fourth successive win thanks to an injury-time header from centre-back Luke Prosser which came after both teams had seen their earlier goalscorers sent off.
But Brown felt his players were way below their best in front of a bumper crowd of more than 9,000.
"It's a fantastic sign to win when you're not playing well but I'm disappointed because I wanted to show these fans the type of football we're capable of playing," said Brown.
"With prices being reduced, we had more than 9,000 here, which was great to see, but we just couldn't seem to get going.
"We rode our luck at times and were fortunate to be 1-0 up at half-time because York were the better team in the first 45 minutes.
"But we dug in to win the game, which was great, and I just hope that's sent the fans home happy."
The home fans were on their feet after just six minutes when Kevan Hurst fired the Shrimpers in front as his angled 16-yard shot flew in the back of the net via the inside of the left post.
York reacted well to going behind and Southend goalkeeper Daniel Bentley made a smart double save to keep out both Keith Lowe and substitute David McGurk in quick succession.
Ryan Bowman then sent a scuffed shot against the right post before he drew the Minstermen level in the 56th minute, firing home a low shot from 16 yards after Bentley had done well to parry out Josh Carson's close-range header.
Two minutes later, things got even worse for Southend when Hurst, who had earlier been booked for celebrating his goal with the fans, was shown a second yellow card for pulling Ben Davies' shirt.
After the winger's dismissal, Bentley soon had to be at his best to superbly block a well struck Elliott Whitehouse volley.
But York were then also reduced to 10 men when Bowman was dismissed in the 86th minute for a late lunge on John White.
And Southend made the most of the striker's dismissal by regaining the lead right at the death when Prosser headed home Ben Coker's right-wing free-kick for his second goal of the season.
- Sports & Recreation
- Luke Prosser