And James Baxendale popped up with a late winner but Adams felt his team should have been awarded a penalty when Robertson appeared to be fouled just seconds before Walsall broke away to score.
"I don't want to sound like a sour and bitter man because we've lost a game of football, but we've not had the rub of the green," he said.
"I thought it was a penalty but the important thing is the referee, who is in charge of the game, didn't.
"I'm disappointed to be out of the cup, but I'm learning all the time about the new players we've brought in and I'm seeing a lot of positives.
"We looked dangerous and played well for much of the game against a very good Walsall side. We created chances and were unfortunate to be behind at half-time, but we got a response in the second half.
"I thought it was going to go to extra-time when I would have put on fresh legs, but unfortunately we've conceded a poor goal.
"Our players have followed the ball instead of the runner and that was disappointing. We need to keep clean sheets but we are not looking like we are going to keep a clean sheet at the moment."