The result meant the Saddlers winless run was extended to a worrying 16 matches.
Smith said: "We have done everything but win that game. The only way I could see them (Yeovil) scoring was from a mistake or set piec and that's what happened.
"Ben Purkiss went down when he should have cleared for their first and the second was from a free-kick when we should have won the first header but the lads are a bit nervous having gone so long without a win."
Walsall took the lead in the 13th minute when former Manchester United striker Febian Brandy delivered a perfect low cross to the far corner of the box and Jamie Paterson met it first time to score his fourth of the season.
It looked all over when Brandy found Will Grigg in the 72nd minute and her cut inside a defender before driving in his fifth of the season with a low left foot shot.
But after Grigg almost notched a third for Walsall, Yeovil poured forward and Sam Foley crossed for substitute Reuben Reid to cut the lead from close range after 78 minutes.
Just as the game entered stoppage time a McAllister free-kick unsettled the home defence and Foley smashed a low shot into the back of the net to claim a rather lucky point for Yeovil.
Smith added: "We played some really good stuff and Yeovil are fortunate to come away with a point."
- Sports & Recreation