Notts had started the game well but an 11th-minute corner saw Nile Ranger left unmarked and the Robins striker made no mistake with a header into the top corner of the net.
Swindon then doubled their lead from the penalty spot in the 28th minute through Dany N'Guessan and went on to dominate the majority of the game.
Kiwomya stated that his team needs to cut out the mistakes - particularly from set-pieces - if they are to climb the Sky Bet League One table.
"We cannot come away to teams like this, who are playing with confidence and give them two goals like that," said Kiwomya.
"I'm disappointed for the boys because we've been playing so well and in the first 10 or 15 minutes we looked as though we were going to be a team that could have got some points.
"Then with their first corner they score, then we give them a penalty and then in the second half they're playing with confidence and we're chasing the game.
"We had four corners in the first 10 minutes but you don't see anyone going in and scoring at their end.
"These are the things that turn us into a top half of the table side rather than sitting in the bottom half.
"We might be going in to double figures now with goals that have cost us from set-pieces and we have to be able to defend much better as a team and that's for me to look at."