But inside two minutes the Millers were back on level terms. Joe Scarz's cross from the left deceived central defender Cian Bolger and Matt Tubbs controlled on his chest before firing past Sam Walker.
And 12 minutes later the home side went in front. Kari Arnason's long throw-in caused panic in the Colchester rear guard, Lee Frecklington's shot was blocked but Kieran Agard fired home through the crowd.
But just as Rotherham thought they had won, the U's grabbed an equaliser in the final seconds of added time with an Elliot Lee header from Garbutt's corner which the linesman ruled had cross the line.
"A couple of weeks ago we talked about the fact that we don't get breaks, the fact that we don't get last-minute equalisers," said Dunne after the 2-2 draw.
"But here we kept going. We didn't play well and we couldn't deal with the long ball threat and Rotherham had players who played really well, but I am not going to fault my players.
"We had a bit of a discussion at the break and heated words were exchanged and we got a response second half. We have a young team and they kept going and kept going.
"We were talking about subbing Elliot Lee but we always felt he could get us a goal and thankfully it happened.
"Rotherham will be very disappointed but we have been on the end of a lot of those kind of things and my players have got through a really sticky, sticky game and come out with another point."
On the late equaliser, Dunne added: "The word from the dressing room was that the ball definitely crossed the line."