The Cumbrians wasted an early chance through David Amoo but the game's decisive moment came when Adam Rooney burst into the 18-yard box and was brought down by Conor Townsend. Referee Mark Heywood awarded a penalty.
Goalkeeper Mark Gillespie got a hand to Rooney's spot-kick but it had enough power to give the Irishman his fourth goal of the season.
Kavanagh was sent from the touchline after 74 minutes for sarcastically applauding a decision by Heywood, but his side then carved out their best chance as Oldham keeper Mark Oxley did brilliantly to deny Lee Miller.
Kavanagh was unhappy with the penalty decision, insisting: "It shouldn't have been given.
"I haven't seen a replay, but in real time it looked as though their player tripped himself up and made a meal of it.
"Some of the referee's decisions were poor throughout the game. He gave us nothing all day and obviously he decided to send me off.
"It was very frustrating, but we also need to look at ourselves because we didn't have enough performances either with or without the ball.
"The first half was as bad as we've played since I took over. We knew we had to improve on that and we did create some chances.
"In the last few minutes we got balls into the box and Oldham's keeper made a very good save, but fair play to Oldham for doing what was needed to win the game."