More details emerge in Secret Service scandal Source: One agent brought a woman back to a different hotel days earlier Defense Secretary Panetta says U.S. Military members involved lost security clearance The scandal involving prostitution in Colombia has embarrassed the Secret Service The White House says there is no indication any of its staff members were involved Washington (CNN) -- One of the Secret Service agents linked to the Colombia prostitution scandal brought a woman back to the Hilton Cartagena, the same hotel where President Barack Obama later stayed, five days before the president arrived in Colombia, a source familiar with the investigation told CNN on Monday. The source was not certain if money exchanged hands or if the Secret Service member simply brought a woman he met -- a foreign national -- to the Hilton. According to the source, the incident appeared to be separate from the one a few nights later that caused 11 Secret Service members to be sent home for alleged heavy drinking and consorting with prostitutes. It also raised questions about a possible security breach immediately preceding Obama's visit. Six Secret Service members have left their jobs in the wake of the incident in Cartagena, Colombia, which came while they were on a security detail in advance of Obama's trip there for the Summit of the Americas. The agency had announced Friday that a 12th Secret Service employee" had been implicated. In addition, the U.S. military is investigating 11 of its own service members for possible heavy drinking and consorting with prostitutes. Defense Secretary Leon Panetta said Monday that security clearances have been suspended for all all U.S. military personnel involved in the incident in Cartagena. Based on the investigation so far, there is no evidence the women involved realized the Secret Service personnel were with the agency, the source said Monday. Investigators have interviewed the prostitutes and other women involved. "It doesn't appear these women knew who the heck they were," the source said. Meanwhile, House Homeland Security Chairman Peter King told CNN he expects more Secret Service members to be forced out soon. "I am very certain that within the next day or so you're going to see a number of people leaving the Secret Service," King, R-New York, said in a telephone interview. King, as well as the other source familiar with the investigation, told CNN that polygraph tests administered to the Secret Service members helped get information to force out six of them so far. According to King, the 11 Secret Service members originally under investigation were all given drug tests, which came back negative. From the start, Secret Service Director Mark Sullivan told legislators how outraged he was about the scandal, promising a broad and comprehensive investigation. So far, most legislators seemed comfortable with the way the Secret Service has reacted to the scandal. However, both the House and Senate Homeland Security Committee chairmen are now conducting their own investigations and will likely hold hearings in coming weeks. In a letter Friday, the ranking Republican on the Senate Judiciary Committee asked the Secret Service about the potential involvement of White House staff in the scandal. Sen. Chuck Grassley, R-Iowa, questioned Sullivan and Acting Inspector General Charles Edwards if they were investigating the possibility that the White House Communications Agency and the White House Office of Advance may be involved. Grassley asked specifically if the agency reserved rooms or shared them with White House staff members for operational matters. White House Press Secretary Jay Carney told reporters Monday there was no indication of any involvement by White House staff members. He said the White House general counsel's office looked into the possibility, as a matter of due diligence, and found no evidence that members of the White House staff or advance team took part. Last week, Carney took exception to criticism of the Obama administration by some Republicans, calling it an attempt to politicize the issue. "It is preposterous to politicize the Secret Service," Carney said then when asked about comments by former Alaska Gov. Sarah Palin and other Republicans that the scandal indicated poor leadership by Obama. Sullivan -- who briefed Obama on the investigation Friday, according to White House officials -- has ordered a "comprehensive" investigation of everything that happened during the trip. That includes interviews with every Secret Service member on site, hotel staff and alleged prostitutes, a source said Friday. The 11 Secret Service employees accused of consorting with prostitutes arrived earlier the same day as a part of the "jump team" that flies in on military transport planes with vehicles in the president's motorcade. According to sources, the alleged prostitutes -- the youngest of whom were in their early 20s -- signed in at the Cartagena hotel where Secret Service members apparently stayed, flashing their local ID cards.