Jake Hyde's 20-yard strike was enough to give the visitors only their second away win of the season, and left the Gills with two wins out of their last eight league games.
However, Allen said: "I think we've lost three in 14 or something, so I suppose that's pretty good.
"It was only a couple of weeks ago that we won at Rotherham. There's room for improvement, but the results don't worry me.
"It was a disappointing first-half performance. I've got to ask myself and perhaps ask the players why we didn't get going.
"Barnet do play some good football. It's difficult chasing a team when they've got an extra man in midfield but it's our job to counter that and put our own mark on the game.
"The performances are what you want to be good. Some of them have been, and then there have been spells that haven't been as good and that's what needs to be looked at.
"You have got to try to help the players. We've got to be firm, assertive but we need to encourage them too.
"No-one likes coming off when they've just been booed. That hurts all of us. People want to go away from football happy and that's our job."