"You've hosted the Games before, withdraw your bid so we can host them once," Erdogan told a joint press conference he held with Abe to sign a $22 billion dollar deal to build a nuclear facility in Turkey.
Erdogan's comments appeared to be a calculated attempt to defuse tension after Tokyo's governor Naoki Inose made allegedly Islamophobic remarks in relation to Turkey's bid.
Istanbul, Turkey's most populated city and business capital, is competing with Madrid and Tokyo to host the 2020 Olympics. Istanbul is bidding for a fifth time after its previous campaigns were unsuccessful, while Tokyo held the event in 1964.
Tokyo's outspoken governor, who heads the city's bid for the 2020 Olympics, apologised earlier this week for "inappropriate" comments he made about rival candidate Istanbul and Islamic countries.
Inose's remarks, made in a recent New York Times interview, prompted the International Olympic Committee (IOC) to investigate, while Turkey's sports minister said they were "unfair and disheartening" and "did not comply with the spirit of the values of the Olympic Movement".
"I believe that the prime minister will appreciate our request, and it would be right if he gives related instructions to the Tokyo governor," Erdogan said, causing laughter in a room of Japanese and Turkish government officials and businessmen.
In his interview, Inose, who was elected Tokyo governor last year, said: "Islamic countries, the only thing they share in common is Allah and they are fighting with each other, and they have classes.
"For the athletes, where will be the best place to be? Well, compare the two countries where they have yet to build infrastructure, very sophisticated facilities," he said.
Inose later apologised for his remarks.
"If my remarks caused any misunderstanding I would like to apologise for them," said Inose, adding that it was a "good experience" and he now understood where the "lines are drawn".
"I want to keep campaigning strictly in accordance with the IOC rules that one should not criticise other cities," he said.
Abe politely declined Erdogan's request, but said he would be the first to congratulate Turkey if it won the right to host the Games.
The hosts for the 2020 Games will be decided at the next IOC Session in Argentina in September.