American swimmers Jack Conger and Gunnar Bentz were removed from a flight to the US by the Brazilian authorities, the United States Olympic Committee said.
The drama happened hours after a judge ordered gold medallist Ryan Lochte and team-mate Jimmy Feigen to stay in the country as police investigate their claim they were robbed during the Olympics.
American swimmers Gunnar Bentz and Jack Conger were taken off a plane at Rio airport as local police investigate Sunday's alleged robbery involving them and two team-mates.
Bentz and Conger, who both won gold medals in the 4x200 metres freestyle relay, claimed their taxi returning to the athletes' village, which also had James Feigen and Ryan Lochte in it, was stopped by men posing as armed police who robbed them.
USOC spokesman Patrick Sandusky confirmed the swimmers were detained but had no other details.
Sandusky said: "We can confirm that Jack Conger and Gunnar Bentz were removed from their flight to the United States by Brazilian authorities. We are gathering further information."
Lochte and his three team-mates said they were held up at gunpoint in a taxi as they returned to the athletes' village from a party, several hours after the last Olympic swimming events were held.
But that story appears to be starting to unravel, as CCTV footage has emerged in the Brazilian media of Feigen and Lochte returning to the village much later than they claimed, and Lochte's wallet, which was supposed to have been stolen, passing through the security check. Rio police have also been unable to find the driver of their taxi.
Police have found little evidence so far to support their accounts, and say the swimmers were unable to provide key details in interviews.
Sandusky said local police came to the athletes' village on Wednesday morning to meet the pair and seize their passports but said "we were not able to make the athletes available" as the swimming team moved out of the village when their competition ended last weekend.
The office of Judge Keyla Blank gave the order earlier on Wednesday to seize the passports of only Lochte and Feigen.
Lochte's father, Steve Lochte, told The Associated Press by phone from his Florida home that his son called him on Tuesday after arriving in the US.
The 32-year-old swimmer was going to pick up his car and buy a new wallet to replace the one that he said was stolen in the robbery.
"I'm just happy he's safe," the elder Lochte said. "It was an unfortunate experience for him and the other three. I don't know what all the controversy is. They were basically taken out of the taxi and robbed. The main thing is he's very lucky that he's safe and that all they got was his cash and wallet."
He said his son's Olympic credentials and mobile phone were not taken during the incident early on Sunday morning. He said he was sure Ryan had his passport or he would not have been allowed to board a plane.
Associated Press reporters waiting outside Lochte's home in North Carolina did not see the 12-time medallist.
Feigen's whereabouts could not immediately be confirmed, though he told the San Antonio Express-News he was still in Brazil.
The 26-year-old Feigen won a gold medal in the men's 4x100 metres freestyle relay, while Lochte, 32, won his 12th Olympic medal, and sixth gold, in the 4x200m relay.