The 44-year-old Irishman resigned as Easter Road boss on Friday - just two days after fans protested outside the stadium following Hibernian's second derby defeat to the administration-hit Jambos this season.
But Williams believes Fenlon had been living on borrowed time with the club's followers ever since the dark day of May 19, 2012, when the club suffered their Hampden humiliation against Paulo Sergio's Hearts.
The 31-year-old Englishman told Press Association Sport: "The fans are entitled to their opinions. They pay good money to come and watch us, and without them the club is nothing.
"But it has been difficult for the manager with the fans ever since the cup final defeat to Hearts.
"This season, a couple of results against our local rivals - who are struggling - have magnified things.
"He recovered as a manger and turned things around but that 5-1 defeat has stuck long in the memory of the fans and their frustration was clear to see after Wednesday's loss."
Fenlon saved the club from relegation in his first season in Edinburgh but still ended up clinging on to his job following the cup final performance.
Last season was better as the former Shelbourne boss guided the Leith men to seventh in the Scottish Premiership as well as a second successive Scottish Cup final.
But they lost that game by three goals to Celtic before starting this campaign with a mortifying 9-0 aggregate capitulation to Malmo in the Europe League qualifiers.
Worse was to come as Hearts - docked 15 points for breaching insolvency rules and banned from signing new players - still managed to achieve a pair of derby victories, with the second coming in Wednesday night's League Cup quarter-final.
Fenlon walked after deciding he had taken the club as far as he could, leaving assistant boss Jimmy Nicholl in charge for Sunday's Premiership trip to Motherwell.
But Williams says the players must accept some of the blame.
Admitting he was stunned to see the boss quit, Williams said: "It came totally out of the blue.
"There was a lot of negativity after Wednesday's game but no-one expected the gaffer to be gone before Sunday's match.
"I was at home with my wife and son and pretty much saw it on TV. I couldn't believe it because while I've only been here about 15 months, I think the squad has improved massively.
"Since the manager came in, we've gone from a team battling against relegation to seventh place in the SPL and two cup final appearances. We're only five points off second place now.
"But football is an immediate, results-based business and people want fast results quickly no matter how realistic their expectations are.
"All the players are responsible because we are the ones on the pitch. The manager can only control so much. We should have beaten Hearts on Wednesday and perhaps it might have been a different story for the manager.
"There is an element of guilt definitely. If we had done better the manager might not have been under the negative scrutiny he has faced, which forced him into this decision."
A trip to face fourth-placed Motherwell at Fir Park is the last thing the troubled Edinburgh side need but Williams is hopeful of a win to restore confidence before Fenlon's replacement is named.
He said: "It's important we try to focus on Motherwell now because each and every player is now playing for their future, depending on which manager comes in.
"With Jimmy in charge, there is continuity there and he's a great coach. If we can get a positive result perhaps we can make the best out of this situation."