Panetta: U.S. running out of patience with Pakistan on militant havens June 7, 2012 -- Updated 1137 GMT (1937 HKT) U.S. Defense Secretary Leon Panetta, left, speaks during a joint press conference in Kabul on June 7, 2012. U.S. Defense Secretary Leon Panetta makes an unannounced stop in Afghanistan He says U.S. troops have a right to defend themselves against Pakistan-based militants Pakistan must do more to take on the al Qaeda-linked Haqqani network, he says Kabul, Afghanistan (CNN) -- U.S. Defense Secretary Leon Panetta warned Thursday that the United States was "reaching the limits of our patience" with Pakistan over militants that attack U.S. forces in neighboring Afghanistan from havens within its borders. "It's extremely important that Pakistan take action to prevent this kind of safe haven," he said during an unannounced visit to Afghanistan. Panetta also stressed that troops on the Afghan border have every right to defend themselves against the al Qaeda-linked Haqqani network and other militant groups when they launch attacks from Pakistan. Anybody who attacks U.S. soldiers is our enemy. We are not going to take it," he said. We have every responsibility to defend ourselves, and we are going to make very clear that we are prepared to take them on -- and we have to put pressure on Pakistan to take them on as well. Panetta's comments come three days after a CIA drone strike in Pakistan's tribal region killed Abu Yahya al-Libi, the No. 2 man in al Qaeda and a longtime public face of the terrorist network. Monday's strike was the third such deadly attack in as many days and the 21st suspected U.S. drone strike in Pakistan this year. At least six missiles were fired at a militant compound near the town of Mir Ali in the North Waziristan region, near the Afghan border. Pakistani lawmakers have called for an immediate end to the drone strikes, which have been denounced for killing civilians. But speaking on a stop in India on Wednesday, Panetta said Washington had made clear to Pakistan that it would defend itself against those who sought to attack the United States. And we have done just that. We have gone after their leadership, and we have done it effectively ... targeting al Qaeda leadership and terrorists. And very frankly, the terrorists who threaten the United States threaten Pakistan as well. This is not just about protecting the United States; it's also about protecting Pakistan. And we have made very clear that we are going to continue to defend ourselves. President Barack Obama's administration recently defended its use of unmanned drones to target suspected terrorists overseas in a rare public statement, with John Brennan, the president's top counterterrorism adviser, saying the strikes are conducted "in full accordance with the law." The program uses unmanned aerial vehicles, often equipped with Hellfire missiles, to target suspected terrorists in remote locations overseas, with many such strikes occurring in Yemen and Pakistan, despite internal opposition to the practice within the latter country. The Pakistani border area is widely believed to be the operating base for the Haqqani network and other militant groups that have attacked international troops in neighboring Afghanistan.