Tiny forward Brandy burst on to a pass from Sam Mantom in the 62nd minute, rounded goalkeeper Mark Cousins and slotted in his second goal for the club to condemn Colchester to their fifth successive league defeat.
Brandy could have had a couple of goals in the opening 27 minutes. His first shot in the 12th minute deflected off Colchester defender Tom Eastman for a corner, then, sent clear by Craig Westcarr, he dragged a low drive wide.
The Saddlers almost paid for that in the 29th minute when their former striker Jabo Ibehre cleverly controlled a throughball and chipped Walsall goalkeeper Aaron McCarey, only to see the ball come back off he crossbar.
Colchester striker Freddie Sears then shot just wide before home defender Ben Purkiss drove wide from inside the box and in a desperate finish visiting defender Josh Thompson saw his header strike the crossbar.