Villa at least ended a run of four straight defeats with a 1-1 draw at home to Swansea, but there were more boos from the Villa Park faithful as the visitors dominated.
Of the seven teams below Villa in the Premier League table, five have already parted company with a manager this season.
He said: "I've got a good relationship with Randy, but that doesn't mean I'm immune to it. I always have that fear. I always had that fear as a footballer and I always do when I'm the manager. I think you've got to have that.
"I don't think it will get any tougher than what happened last year and he was great with me then. I spoke to him last night and he's been good now as well."
In the same week last season Villa lost all three games, conceding 15 goals and scoring none, and when Crystal Palace's Dwight Gayle netted the winner in injury time on Thursday it appeared festive cheer would again be entirely absent from Villa Park.
Things began well on Saturday with Gabriel Agbonlahor netting after only seven minutes - the first time Villa have scored in the first half of any home league game this season.
Agbonlahor rushed to Lambert and the squad celebrated exuberantly in a huddle in front of the Villa bench.
"I didn't know that was going to happen," said the boss. "They're a group that are really close together, which you need."
But Swansea quickly took control of the game and there was no doubt they deserved their equaliser, which was headed in by Roland Lamah in the 37th minute.
The visitors dominated the second half completely, but could not turn possession into chances, with Brad Guzan having only one real save to make from Jonathan De Guzman's free-kick.
Lambert cited the absence of injured trio Christian Benteke, Ron Vlaar and Jores Okore, saying: "It's a game that you'd love to win, but a point stops that sequence of losing, which was vital for us.
"We're without big players, and it's a lot for us to carry. We're a young squad and when you lose people like Benteke, Vlaar and Okore, it's a big chunk of our spine.
"We (his management team) have only been here 17 months and I knew the first few years was going to be really tough, but we're halfway there when you look at the table. It's a big point.
"Swansea are a good side, they pass the ball really well, but Guzan hasn't really had many saves.
"Some games you play really well and end up losing, some games you have to fight tooth and nail to get something, and we certainly did that."
Swansea are also in a sticky run having not won any of their last six games, and it was confirmed on Saturday that top scorer Michu has had an ankle operation that will keep him out for six weeks.
Boss Michael Laudrup had no doubt they deserved more from Villa Park.
He said: "Normally when you get a point from away games in the Premier League, especially if you are a mid-table or bottom team, you should be pleased, but I think after our first half we should have won the game.
"Even conceding an early goal, we came back, dominated completely, created chances, scored the equaliser and could have scored one or two more.
"Second half we still dominated in possession but without creating a lot of chances. Villa did not have any chances at all.
"Even taking off my Swansea glasses, I still think as a neutral you would say that we should have won this game. I think they're quite happy with a point."
- Sports & Recreation
- Randy Lerner