Modric tried to quit Spurs for Chelsea a year ago under the management of Harry Redknapp, and this summer appears keen on a switch to Spanish champions Real Madrid.
Reports suggest the Croatia international has failed to travel with Spurs on their tour of America - something Villas-Boas appeared to touch upon while addressing the Spanish press.
Speaking to AS, he is reported as saying: "I think Modric is wrong. This will go against him.
"He has worsened the situation with what he's doing and now the chairman (Daniel Levy) is very angry. This is also not a good time for Daniel Levy, who is in the USA at the moment, because his wife is ill."
Villas-Boas also admitted Modric would be a major loss if he leaves Spurs.
"He's good, very good. I tried to sign him last summer for Chelsea. He will be a big loss.
"He's a player who can make the difference, who gives a lot of clarity to the play when the team needs it and who can play in several positions," he said.