Harriers' 3-2 victory at London Road was only the second time a non-league side had beaten Posh at home in the competition in the club's Football League history - Sutton United in 1987 is the only other time it has happened.
It denied Peterborough a trip to Barclays Premier League club Sunderland in the fourth round later this month.
Kidderminster, from the Skrill Premier and 47 places below Posh in the football pyramid, deservedly won the third round replay having held Ferguson's side to a goalless draw in the first game.
Tommy Rowe's goal gave the hosts the half-time lead but quick-fire strikes from Michael Gash and Jack Byrne at the start of the second half turned the tie on its head.
Britt Assombalonga levelled things up again from the penalty spot after 74 minutes but Joe Lolley, who Peterborough tried to sign this month, scored what proved to be the winning goal just two minutes later.
"It is a very bad result on a very bad night for us. We have failed to take a great opportunity to reach the fourth round and face a Premier League team," said Ferguson, whose team have lost their last four league matches and are now out of the cup.
"We conceded three very avoidable goals and that is what cost us dear. One of them was a free header from six yards and my goalkeeper shouldn't have been beaten at his near post like that for the winner.
"You should go on to win a cup tie when 1-0 up at home and we didn't do that. The second goal in the game was always going to be the important one. If we had got it we would have gone on to win it comfortably, but we conceded and then found ourselves behind soon after. It all comes down to the manner of goals we lost.
"Things don't seem to be happening for us at the moment, but you have to give Kidderminster the credit for coming here and scoring three goals.
"We were on top in the first-half, but only scored one goal. At half-time I asked them to keep the tempo up and do the same things.
"At 2-2 I thought Kidderminster's heads dropped a little. We are a fitter team than them so I expected us to go on and win, but they scored straight away which gave them something to hold on to and believe me that makes a massive difference."
Ferguson revealed that skipper Rowe was forced off in the first-half with a thigh injury.
To complete a bad night for Posh they are now out of the League One play-off places after Walsall beat Oldham.