3 Secret Service agents refuse prostitution scandal polygraph Rep. Peter King says 3 of the 12 Secret Service agents refused a polygraph test King sent 50 questions to the Secret Service chief Twenty-four people are linked to the prostitution scandal Washington (CNN) -- Three of the 12 Secret Service agents involved in the Colombia prostitution scandal refused to cooperate with authorities and submit to a polygraph test, according to House Homeland Security Committee Chairman Peter King, R-New York. The three agents were among the first forced out of the service when news of the scandal in Cartagena broke, King told CNN late Tuesday night. The nine remaining agents took the polygraph. And while none of them failed the test, some responses led to the loss of several jobs. King received the information as a part of a response to 50 questions he sent to Secret Service Director Mark Sullivan on April 20. Sullivan submitted his answers to King on Tuesday. While King did not send CNN copies of the responses -- which he said are marked "law enforcement sensitive" -- he highlighted several details. Among other things, one agent said in the polygraph test that he was "actively engaged" with one of the prostitutes when said she wanted to get paid, King said. In response, the agent threw her out of his room. The agent told U.S. officials he didn't realize the woman was a prostitute, and has not been fired. U.S. officials have now interviewed 10 of the 12 women involved in the scandal, King noted. The Secret Service and Colombian authorities are currently trying to track down the remaining two. King stressed what he called a "pleasant surprise" -- Sullivan's decision to called the Homeland Security Department's inspector general before bringing the agents back from Colombia. King said there weren't many surprises in the responses to his questionnaire. "Sullivan was giving us good information all along," he told CNN. House Oversight Committee Chairman Darrell Issa and Rep. Elijah Cummings, the panel's ranking Democrat, have submitted another 10 questions to Sullivan, including a precise time line of exactly what happened in Cartagena. The pair also sent a letter to Defense Secretary Leon Panetta requesting details of a military investigation by May 8. The Homeland Security Department's inspector general is currently investigating the scandal, in addition to four congressional committees as well as internal reviews by the agency, the military and the White House. The top legislators on the Senate Homeland Security and Governmental Affairs Committee said earlier Tuesday they've also sent a letter to Sullivan asking for information on the incident. A total of nine agents have resigned or are in the process of being forced out. Three other Secret Service agents were cleared of serious misconduct, and the military is investigating the alleged involvement of 12 of its service members. On Monday, the Homeland Security official announced his separate investigation of the incident, which embarrassed the government and raised questions of a possible security breach before President Barack Obama arrived for the Summit of the Americas. The "field work is beginning immediately," acting Inspector General Charles Edwards said in a statement. The Senate Judiciary Committee questioned Homeland Security Secretary Janet Napolitano about the controversy at a hearing last week. On Tuesday, Sen. Joe Lieberman, an independent from Connecticut who chairs the Senate Homeland Security Committee, and ranking Republican Sen. Susan Collins said they sent Sullivan a letter Monday that also sought answers about what happened. "We wish to determine whether those events were indicative of a pattern of behavior by agents or officers of the Secret Service, and need to be addressed systemically, or if they instead constituted an isolated incident warranting action only with respect to the individuals involved," said the letter from Lieberman and Collins. The U.S. Southern Command expects to finish questioning the 12 military personnel implicated in possible wrongdoing this week before forwarding its findings to military lawyers for review, and then to Gen. Douglas Fraser, commanding general of the U.S. Southern Command, a Defense Department official said Monday. Last week, the Secret Service distributed new rules for its agents on assignment intended to prevent a repeat of such alleged misconduct, according to two government sources familiar with the resulting investigation. Also in the new standards, foreigners are banned from Secret Service hotel rooms at all times, except for hotel staff and host nation law enforcement and government officials on official business, according to the officials, and all Secret Service personnel are prohibited from going to a "nonreputable establishment." Allegations of further transgressions by agents have emerged after the initial reports of heavy drinking and consorting with prostitutes last month before Obama arrived in Cartagena. Recent claims include an account from El Salvador described by CNN affiliate Seatte TV station KIRO as very similar to the Colombia scandal, involving members of the Secret Service and other government agencies. However, Panetta said last week that his department is not investigating any of its troops over the reported incident in El Salvador, while State Department spokesman Victoria Nuland said embassy staff in El Salvador were being questioned about the allegations But he added that, while the DEA had seen news reports, "we are unaware of any allegations of misconduct." CNN's Tom Cohen, Carol Cratty, Ed Payne, Ted Barrett, and John King contributed to this report.