Nabil Elalem was in his car with a colleague when two cars carrying armed men in military-style clothing blocked the road, told him he had to go with them and sped away, colleagues said.
"There are rumours going round about his whereabouts but we do not know where he is," his brother Khaled Elalem told Reuters.
"The state should be doing more. We are calling on the International Olympic Committee and all Olympic committees around the world to get involved in this matter so that he can return to his family."
Khaled said his family had demonstrated in front of the interior ministry on Monday and that he had met the prosecutor general to ask for his help in finding his brother.
Since the end of the uprising that toppled Muammar Gaddafi last year, the interim government has struggled to impose its authority on numerous armed groups who refuse to lay down their weapons and often take the law into their own hands.
Mustafa El-Huni, deputy chairman of the ruling National Transitional Council, said on Monday the authorities were working on securing Elalem's release.
"We are doing our best to track him down," said Omar Rahal of the Tripoli military council.
Elalem, a former Libyan judo champion, took charge of Libya's Olympic body after its president Mohammed Gaddafi, one of Muammar Gaddafi's sons, fled to Algeria last August.
His deputy said late on Monday he hoped Elalem would be released "in the next few hours".
Libya's representatives at the London Olympic Games, which begin on July 27, are due to head to the British capital in a few days. The team of about 10 athletes is set to compete in judo, swimming, athletics and weight-lifting.
Jacques Rogge, president of the International Olympic Committee, said on Monday the IOC had been in contact with the Libyan Olympic Committee.