Disappointed with Serbia's poor run, which left their chances of snatching a playoff berth hanging by a thread, fans have often jeered the national team during home games. Coach Sinisa Mihajlovic and FA head Tomislav Karadzic have also come under pressure to step down.
"The fans are entitled to turn their backs on you if they feel that you are not playing within your limits," Ivanovic told reporters in Serbia's Stara Pazova training base near Belgrade on Monday.
"Football is such that they will always recognise a fully committed approach and reward it with their respect and support, even if you lose a game, provided that you've left it all out there on the pitch.
"We can't blame poor luck for our woes either because you have to earn your luck on the pitch," added the Chelsea defender.
Serbia are third in the group with seven points from as many games, nine behind Croatia and 12 adrift of leaders Belgium.
Only a win would keep Serbia in the hunt for a playoff berth, which they could snatch from Croatia's grasp on goal difference if they win all three of their remaining games and the Croatians suffer three defeats.
Ivanovic conceded it was an unlikely scenario but underlined that beating their neighbours could turn around Serbia's long-term fortunes as it would instill confidence into a talented but inconsistent outfit lacking experience and a win against strong opposition.
"It would be this generation's biggest win because Croatia are a top quality side and this is exactly the kind of match we need at this stage of our international careers to build a decent platform for the future," he said.
"Our performances have improved through the qualifiers although the results may suggest otherwise but that's because there is little or no room for error in a group as tight and balanced as ours.
"A win on Friday can't erase our slip-ups which left us with only a theoretical chance of progressing but big players are forged in big matches and this is certainly an opportunity for some of our youngsters to shine."
Teenage forward Lazar Markovic, one of five Benfica players called up by coach Sinisa Mihajlovic and described by the 44-year old as a key player whose pace and versatility could swing the match Serbia's way, was relishing the challenge.
"I am beaming with confidence and can't wait for the kickoff because this is going to be the biggest game of my career," said the 19-year old, who scored a stunning equaliser for his club in a 1-1 league draw at city rivals Sporting on Saturday.
"We will fight till the bitter end to qualify as long as the mathematical chance is there but be what may, the game against Croatia is a very special occasion and we are hoping that our fans will stand united as one."