Much of the pre-match talk was about the hosts moving up to second with three points, although it was City that looked more like the promotion challengers in south London.
Nathan Redmond's excellent solo effort put the visitors ahead after 24 minutes with a Damien Delaney own goal doubling their advantage eight minutes later.
Palace looked resurgent at the start of the second half but Birmingham weathered the storm, with Ravel Morrison and Shane Ferguson then netting to secure a heavy win.
It was an awful day for the Eagles as their 18-match unbeaten home league sequence came to an abrupt end.
"No one was expecting that, I certainly wasn't," said Holloway, who was heckled by a fan in the post-match press conference. "Unfortunately I have got to deal with it right now. I think we started the game brightly, got behind them a couple of times.
"We didn't pick out the right ball, didn't get people on the end of it and then their goal seemed to affect everybody out there, which is completely unusual.
"Then they got a second one from a set piece and I was expecting a response like we did away at Watford.
"I can't remember my team missing as many balls flying across the box or making as many wrong decisions as they did out there and they've gone and scored four. That is the last thing we need but we've got to get on with it.
"We have to bounce back and deal with things better. It is simple as that."