Troy Deeney scored the spot-kick after Matej Vydra was adjudged to have been fouled by Peter Clarke.
But once Vydra turned in Marco Cassetti's cross 11 minutes after the restart there was never much doubt about the outcome.
The in-form Czech international scored his eighth in his last four Watford league starts and 16th of the season after excellent hold-up play by Deeney.
The hosts saved their best for last though, when a superb one-touch passing move ended with Cristian Battocchio scoring his first for the club.
The 4-0 defeat stretched the Terriers' poor league run to 12 without a win, and Grayson said: "I thought, first half, we were well truly in the game. We kept them quiet, you could sense the atmosphere had gone quiet as well and the decision just before half time changed the game.
"We let it affect us and it obviously gave them a big confidence boost. The decision was wrong, it's a great tackle by our right-back and it's outside the box, so it's a poor decision.
"But in the second half we felt sorry for ourselves and allowed Watford to dictate the game and when you do that against a team like Watford then you're under pressure because they're a good side.
"You can see how technically good they are but we certainly felt sorry for ourselves and that was reflected in us getting done four, rather than maybe one or two."
Grayson though, was full of praise for the quality of the Hornets' fourth goal, adding "Watford certainly got into their stride and I suppose if anybody watches the fourth goal they'll see what a great goal it was. If that was in the Premiership, it would be applauded by a lot of people."