Competing at the Charlotte Grand Prix, one of the final lead-up events before next month's United States Olympic trials, Phelps came second to China's Wu Peng in the 200 metres butterfly.
His time of one minutes 56.87 seconds was more than five seconds outside the world record he set at the 2009 world championships in Rome.
The defeat came just a day after he finished second in the 200m freestyle, raising doubts about his fitness and form leading into this year's London Olympics, which the American has said will be his final event before he retires from swimming.
But Phelps, who has won 14 Olympic gold medals, including eight from Beijing four years ago, said he was unconcerned about his results because he was focussing on bigger things.
"I know this is not the Olympic trials, this is not the Olympic Games. It's a stepping stone," he said.
"I feel like these are little quizzes. They are like little building blocks."
The US Olympic trials will be held in Omaha, Nebraska, starting in the last week of June.
Under the tough qualifying rules, only the top two swimmers from that meet can represent the US in individual events at London.