The in-form Bees were in cruise control early on after goals from Marcello Trotta and Adam Forshaw had given them a two-goal lead and U's defender Josh Thompson was sent off.
But teenager Drey Wright gave Colchester hope and Brentford had to rely on Harlee Dean's late goal to extend their unbeaten league run to 10 matches.
Rosler said: "I think we almost made too good a start to the game, in a way. The first two goals that we scored almost came too early on in the game and we thought that the game was won.
"We got a bit careless and a bit sloppy with the way that we were playing and that cost us a bit. But we're in an absolutely brilliant position at the moment and we have to keep believing in ourselves.
"It's entirely up to us now, but we did a very good, professional job."
Trotta gave Brentford a fourth-minute lead when he fired in from Clayton Donaldson's far-post nod down.
After Thompson was dismissed for denying a goalscoring opportunity after bringing down Donaldson on the edge of the penalty area, Forshaw netted with a well-placed free-kick.
But 17-year-old Wright fired in a low shot from Kem Izzet's pass to give Colchester hope before Brentford midfielder Tom Adeyemi was dismissed after picking up his second yellow card in the space of 10 minutes.
However, with five minutes remaining Dean tapped in from two yards to wrap up Brentford's win.