Secret Service scandal and alleged prostitution inquiry began with night of partying The incident that prompted the Secret Service to remove up to a dozen agents from Cartagena, Colombia, before President Obama's arrival took place during a night of partying late Wednesday and early Thursday at the Hotel Caribe where members of the president's advance team were staying, people familiar with the matter said. At least one agent had a sexual encounter in his room with a local woman, who demanded to be paid afterward, according to the account. It is unclear whether the agent knew the woman was a prostitute and, when he declined to give her money, the woman reacted angrily and created a disturbance at the hotel, said those familiar with the matter, who have close ties to the Secret Service. A dozen Secret Service agents sent to Colombia to provide security for President Obama at an international summit have been relieved of duty because of allegations of misconduct. President Obama says legalizing drug use is not the answer to trafficking in illegal narcotics in the Americas, countering a growing chorus in Latin America to discuss decriminalization as a way to ease deadly cartel violence. How will Secret Service recall affect Obama's Colombia visit? View here The woman eventually complained to hotel staff and local police authorities, who notified the U.S. Embassy, said the people, who spoke on the condition of anonymity to speak freely about an ongoing investigation. Two of them learned of details directly from agency employees and the other was briefed on the matter. The U.S. diplomats informed the Secret Service, and the Colombian authorities provided a list of all U.S. officials in the hotel, including at least five military personnel. Col. Scott Malcom, a spokesman for the Department of Defense, confirmed that five military employees were in violation of a curfew that night. They remain in Cartagena and have been confined to quarters in the hotel. "When they return, there will be a full investigation," Malcom said. The Hotel Caribe is in Bocagrande, a seaside district of Cartagena, a few blocks from the Hilton, where Obama is staying. The Secret Service has opened an investigation into the matter after recalling the agents and replacing them with another team shortly before Obama's arrival Friday afternoon for the Summit of the Americas, which runs through Sunday. Obama and 32 other heads of state have gathered to discuss economic ties and trade. Secret Service officials have stressed that Obama's safety was not compromised by the upheaval, but the sensational investigation has threatened to overshadow his trip. Briefing reporters in Cartagena, press secretary Jay Carney said the White House learned of the matter on Thursday and Obama was informed Friday. Carney directed most questions to the Secret Service in Washington, saying it would be inappropriate for the White House to weigh in while the matter is under investigation. "This has not been a distraction," Carney said. It has been much more so for the press than for the president, who is going on with his work here. Staff writers Joe Davidson, Ed O'Keefe and Scott Wilson and special correspondent Juan Forero contributed to this report. Wilson and Forero reported from Cartagena.