Following a goalless and fairly even opening 45 minutes, Inverness ran away with it in the second half, Andrew Shinnie opening the scoring with a sublime effort, before Billy McKay struck twice to take his tally for the season to 20 goals.
The loss was a sore one to take for Brown, who felt his side had just as many chances as Caley Thistle, but failed to make them count.
He said: "It was a poor performance from us, but all the credit should go to Inverness, who took their chances very well.
"Jonny Hayes had great chances for us in each half, but couldn't take them, and that was the difference.
"If you don't take your chances and you defend in the slipshod manner we did then that's what happens."