Government sources say preliminary findings show prostitutes were brought back to a hotel None of the agents or officers involved are part of the president's protective detail, sources say A law enforcement association chief calls it an "isolated incident" A former reporter says it is "clearly the biggest scandal in Secret Service history" Cartagena, Colombia (CNN) -- A group of Secret Service agents and officers sent to Colombia ahead of President Barack Obama were relieved of duty and returned home amid allegations of misconduct that involved prostitution, according to two U.S. government sources familiar the investigation. Roughly a dozen Secret Service members are being investigated over early findings that they allegedly brought back several prostitutes to a hotel in Cartagena, the sources told CNN Saturday. None of the agents or officers were part of the president's personal protective detail. The incident overshadowed the start of the sixth Summit of the Americas, where the president was to focus on trade, energy and regional security. Obama: 'Hemisphere is very well positioned in the global economy' There was a dispute between at least one Secret Service member and a woman brought back to his hotel over a request to be paid, the U.S. government sources said. At least one of the women brought back to the hotel complained to police, and complaints were filed with the U.S. Embassy, the sources said. A spokesman for Colombia's National Police declined to comment, referring questions to the Secret Service. Secret Service spokesman Edwin Donovon said that the matter was being turned over to the agency's internal affairs unit, but declined to provide details about the accusations or the number of agents involved. Before the president's arrival, he said, a group of Secret Service agents in Cartagena were relieved of duty and replaced after "allegations of misconduct," he said. The Secret Service takes all allegations of misconduct seriously," he said. Ronald Kessler, a former Washington Post reporter who has written a book about the Secret Service, said the incident was "clearly the biggest scandal in Secret Service history." Jon Adler, president of the Federal Law Enforcement Officers Association, cautioned against characterizing the "isolated incident" as a scandal. "That's just sort of an overdramatic interpretation of an isolated incident," he said. We have to trust the process of the internal review. While soliciting prostitution is legal in Colombia, it is considered a breach of the agency's conduct code, the government sources said. High-level officials in the Secret Service and the Department of Homeland Security were outraged over the incident, the sources said, noting that the investigation indicated the prostitutes were brought back to a hotel that had been secured for Obama's visit. The U.S. government sources stressed that there was no threat to Obama. Because the trip already had a large security detail, only some of the recalled agents and officers were replaced, the sources said. More than 7,600 police officers and thousands more troops have been deployed in the walled colonial city of Cartagena as part of stepped up security for the summit. Submarines are patrolling in the coastal waters near the city, armed helicopters are hovering at the ready and snipers in strategic locations are watching for suspicious activity, officials said before the summit's start. Anti-explosive robots and radiation detectors are also part of the security detail. CNN's John King, Dan Lothian, Randi Kaye, Chelsea J. Carter, Mike Ahlers and Catherine E. Shoichet and journalists Jorge Baron and Fernando Ramos contributed to this report.