Kavanagh blasted Bratt for not awarding his side an 83rd-minute penalty with the score at 2-1 when David Amoo appeared to be tripped by Yaser Kasim.
He said: "The referee was absolutely useless and (Swindon manager) Mark Cooper agreed the same.
"I think the incident involving Amoo should have been a penalty and I think the referee has just made an excuse.
"Amoo was clipped, he was stopped from taking a touch of the ball. If he did take a touch of the ball it would have taken him clean through on the goalkeeper, but he's been wiped out.
"Some of the decisions he made for and against us were absolutely abysmal.
"But the referee tells me he (Amoo) slipped over and made a meal out of it so it's very frustrating. We made a silly mistake which allowed them to get back in front but it's a great free-kick from (Alex) Pritchard."
Massimo Luongo fired home from Pritchard's whipped cross to open the scoring. Lee Miller nodded against a post before Carlisle did equalise through Sean O'Hanlon's header. Pritchard restored the Robins' lead with a lovely curling free-kick before substitute Dany N'Guessan tapped in Nile Rangers' cross late on.
Kavanagh added: "We didn't threaten their goal enough so I am disappointed, but the better team won."