Danny Kedwell netted the only goal of the game, and his sixth in as many matches, from the spot just past the hour-mark after Danny Livesey had clumsily fouled on-loan Gills midfielder Danny Hollands.
"We should have had a penalty before half-time. I don't know how the referee didn't give it," Kavanagh said.
"I spoke to him at half-time and he said it wasn't a goal scoring opportunity.
"The ball was in the middle of the goal, six yards out, and their player handles it.
"It is disappointing and it is an opportunity wasted. We weren't anywhere near the level I was expecting us to be at.
"It wasn't good enough. We needed to show more quality, more penetration, and we needed to pass the ball better.
"We were the more dominant team, but we weren't as effective as we would've liked.
"It was a very disappointing afternoon after such a long journey to get to the game.
"The fans were magnificent once again. I am disappointed that we didn't get a victory for them.
"We have got to go again on Tuesday and put it right."