Bradford skipper Gary Jones scored direct from a corner to extend the Gills' desperate start to the season.
Allen is yet to see his side win in either league or cup so far this season and did not appear until almost 7pm after a meeting with his players.
When he did emerge the 48-year-old refused to pull any punches, saying: "I wasn't happy. I made two changes from what was an excellent performance away at Coventry and I think I've got it wrong.
"I've sat here from day one and backed the players. I always stood up and given them full support, but I thought there were far too many player well below par. The attitude wasn't right.
"Hopefully over the next few weeks there will be some changes. We've got to make some changes. It's been far too many games. The paying public deserve better than that.
"We need to revisit the principles and values that have got us into this division. We got promoted as champions with a hard-working, dedicated, honest group of blokes.
"It's been too many games I've sat here and defended, looked after, supported [the players] but I need to take some action no doubt about that.
"The players have let themselves down, the club down, the shirt down."
James Hanson should have extended Bradford's lead only to miss with a header from close-range, while Cody McDonald squandered Gillingham's best chance in the second-half as they slumped to the bottom of the table.