The Canaries had stunned their high-flying hosts with goals from Steven Whittaker, Sebastien Bassong and Grant Holt giving them a 3-0 half-time lead.
Swansea rallied through strikes from Michu and Jonathan De Guzman but Robert Snodgrass made sure of the win with his excellent free-kick before Michu netted again in stoppage time.
Hughton said: "The first half could not have gone better but it was always going to be difficult to sustain that.
"We did ourselves no favours by conceding so early in the second half, which was always going to give them a lift.
"But at 3-2 a lot of people who were not here would have expected it to go to 3-3 and become a Swansea win.
"We had to dig deep and the character we showed at 3-2 with 30 minutes to go was excellent as it was really tough."