The result ended the Latcis' six-match winless run and Kavanagh's frustration saw him sent to the stands after 74 minutes for sarcastically applauding a refereeing decision.
The visitors had a good early chance but David Amoo's near-post shot was saved by Mark Oxley.
Oldham went ahead after 26 minutes after Rooney burst into the area and was felled by Townsend. Rooney blasted a low penalty just beyond the dive of goalkeeper Mark Gillespie.
With the hosts dominant, Gillespie had to make smart saves from Kirk Millar, James Wesolowski and James Dayton.
Carlisle came to life in the closing stages but Lee Miller's point-blank header was brilliantly kept out by Oxley.