Carlos Edwards and Frank Nouble netted for the hosts after the break following a first half which ended all square as Jordan Rhodes scored against his former club, cancelling out Luke Hyam's early opener.
"We didn't start bad and DJ Campbell could have scored within a minute," said Bowyer after the 3-1 defeat in Suffolk.
"We didn't defend their first goal well enough although I thought we were the better side before half-time and deserved our equaliser.
"But we gave them their second goal and their third and you can't do that. It is not as if we were cut open and they scored breathtaking goals.
"It is frustrating as we have kept five clean sheets at home. We have to find the mentality to do that away from home - and stop being so nice."