Danny Kedwell put the Gills ahead from the penalty spot before Alexander received a straight red card for kicking Leon Legge, and then Lee was adjudged to have fouled Nicky Adams and shown a second yellow card.
Kyle McFadzean equalised for Crawley and Clarke put them ahead from close range only for Legge to toe poke the visitors level once again.
It was Clarke, though, who had the final say by bursting through to hit the winner with an unstoppable shot into the top corner nine minutes from time.
Clarke already boasts four goals this term and Barker says he is vital to the team after having the "season of his career" last term.
Allen described referee Steven Rushton's decision to send Lee off as "unbelievable" and claimed his player was merely trying to allow the ball to go out of play and did not foul Adams.
Allen said: "It wasn't even a foul, an unbelievable decision. We're going to miss Lee but, as it was a yellow card, an appeal is unlikely."
The Gillingham boss insists his men do not look like a side who have yet to win this season and is backing them to turn the corner quickly.
He added: "My players are frustrated, disappointed and upset. We have no big time Charlies. They are upset and so am I. But this is where you do your stuff.
"I give my players credit every week; they keep going with their effort and commitment. They want to turn these good performances into victories.
"The players put in a fantastic effort and it's disappointing to lose on two set plays.The decision to send off Lee and the injury to Legge which caused him to go off were crucial to the outcome.
"I have a good rapport with the referees' association, although I've criticized referees more this season than last already. But I'm sure our luck will change, some of it has got to go in our favour."
- Sports & Recreation
- Charlie Lee
- Leon Legge
- Nicky Adams
- Gary Alexander