Cas recovered from 18-12 down early in the second half to bag a second successive home victory ahead of the expected unveiling of Featherstone's Daryl Powell as their new permanent head coach on Tuesday morning.
"I thought the lads played really well, " said Orr, who will stay on at the club to assist new coach Powell. "We were pretty good in the first half and the second half was probably the best we have played this year and we thoroughly deserved our win."
He added: "The lads were in a really confident mood at half-time and although Hull KR scored after 10 seconds into the half, we still had 39 and a half minutes to put it right.
"The lads dealt with it and got with the game. We played some pretty good stuff in the second half. We held the ball really well and we turned the ball over in the right areas of the field.
"I thought Kirk Dixon and Michael Shenton had outstanding games on our right. They both defended strongly and attacked really well. I also thought Jamie Ellis had one of his better games. He was dangerous with the ball and put in some really good kicks."
Rovers crashed to their third successive league defeat. Liam Salter's two tries and Michael Dobson's four goals were in vain for as they suffered another blow to their top-eight hopes.
Coach Craig Sandercock said: "Nothing went particularly wrong. I'm pretty proud of the boys.
"I've never been involved in a rugby league game where one side has defended their own line for the entire 80 minutes.
"CastIeford had enough ball to win 50 games, let alone one."