The game burst into life after 22 minutes when Elliott Hewitt handled Francois Zoko's shot on the goal-line and duly saw red, with the latter casually stroking in the resulting penalty.
Gillingham briefly thought they had a route back into the game after Cody McDonald's composed finish only for Charles to pop up with his second three minutes later following a Freeman free-kick.
Gills boss Taylor said: "I'm very proud of the players for the way they kept going because it was a difficult evening.
"Obviously it was a disappointing first goal for us in respect that it wasn't good defending and then we got a man sent off as early as that.
"It was the correct decision by the referee and the wrong decision from Elliott.
"I feel he should have let it go in because at that early stage, if it goes in you've still got 11 men and you've still got time to recover.
"It made it a difficult night but I thought the players kept going so I was really impressed with that and we've had enough chances.
"Cody could have had four goals so in that respect I'm very proud of them.
"There were a few little frustrating things that we've got to improve on but there were a lot of good performances."
- Sports & Recreation
- Luke Freeman
- Darius Charles
- Elliott Hewitt
- Francois Zoko