Washington (CNN)In 2011, al Qaeda took Warren Weinstein hostage. Then, about a year later, his family paid money to his captors, said a Pakistani source who was in regular contact with the kidnappers. It did not lead to the American aid worker's release, and he was inadvertently killed in an anti-terror strike in January, the White House announced Thursday. After the transfer of funds in 2012, the captors, who never referred to themselves as "al Qaeda," but instead as "Afghans," began demanding prisoners be released in exchange for Weinstein, most prominently Dr. Aafia Siddiqui of Pakistan, the source said. She has been described as the "poster girl" for Islamic jihad and is serving an 86-year sentence in the United States. The militants also wanted the release of local militants who hailed from the Pakistani province of Waziristan, along the Afghan border, the source said. The men on the other end of the phone spoke Pashto with an accent typical for the border region. And they were professional in their dealings, said the source, who noticed a marked change after the money was paid. The people who had originally talked to the source about Weinstein vanished and were replaced by new voices on the phone. The militants also connected themselves with other terror events. After ISIS beheaded American James Foley, Weinstein's Afghan captors told the source that "the Iraqis" were asking for the American and that they were preparing an "orange suit" for him -- a reference to the suits that victims have worn when ISIS militants murdered them. When the Taliban released U.S. Army Sgt. Bowe Bergdahl, a militant on the phone bragged to the source that he had been one of Bergdahl's kidnappers. They had said, at one point, that if they did kill Weinstein, they'd announce it in a big way, because he was too big a catch not to publicly celebrate. The captors had called the source daily since sometime in 2012, and the last time the source spoke with them, in early April, they said that Weinstein was still alive. In spite of a request for proof of life, the militants didn't give one. After that, the calls stopped. The source never had a number for the militants; they had always been the ones who called. The source declined to comment on the amount of money transferred to the captors, leaving the disclosure to Weinstein's family. Could Weinstein have been saved? Weinstein's family in Maryland was initially reluctant to take their case to Congress. Weinstein's wife, Elaine, and his two daughters worried that drawing attention to their loved one's plight -- especially media attention -- would make Weinstein, a government contractor working with USAID in Pakistan, a more valuable hostage to his al Qaeda captors. But the family switched course and went to their representatives in Congress in late 2013 after a video of Weinstein -- frail and apparently in declining health -- surfaced in which he said he felt "totally abandoned and forgotten" by his country. Will the drone debate return? More than a year later, that reluctance evolved into a close bond between the Weinsteins and the Maryland delegation of lawmakers and staff who pressed Weinstein's case with the Obama administration as well as Pakistan. The congressional offices helped the family navigate the maze of government agencies working to free their loved one, according to one of those lawmakers and a Senate aide. "We don't get choked up too often at work," Algene Sajery said as she held back tears. "But this is really hard." Sajery is a foreign policy adviser to Sen. Ben Cardin, D-Maryland, who worked closely on the case. After the White House announced that Weinstein had been accidentally killed in a January U.S. drone strike, the aide's first call was to Weinstein's daughters. "Myself and my colleague worked really closely with them, talked to them all the time ... they're just such good people," Sajery said of the family. "There's a personal connection there." Cardin and his staff had tracked the Weinstein case since news of his abduction surfaced in August 2011, but when the family reached out to get help pressing their case, Cardin's staff joined forces with Rep. John Delaney, the Weinsteins' congressman, and began setting up meetings for the family. Cardin's staff set up meetings for the senator and Weinstein's family with everyone from the U.S. special representative to Afghanistan and Pakistan to the Pakistani ambassador in Washington. Cardin and Sen. Barbara Mikulski, D-Maryland, wrote to President Barack Obama, calling on the U.S. to beef up efforts to secure Weinstein's release and dedicate more resources to the cause. And the staff and lawmakers helped the Weinsteins weave through the network of agencies -- from the FBI to the State Department and the White House -- working to bring Weinstein home. "We focused on making sure that the capabilities of the government was well coordinated," Delaney told CNN on Thursday. "The government is a bureaucracy, and you have to make sure that it's working." But for the Weinsteins and the families of other American hostages held by terror groups abroad, the government hasn't always worked well enough -- a frustration that was palpable to the staffers on Capitol Hill who worked to help them. Al Qaeda hostage Warren Weinstein killed . "Unfortunately, the assistance we received from other elements of the U.S. Government was inconsistent and disappointing over the course of three and a half years," Elaine Weinstein said in a statement Thursday. "We hope that my husband's death and the others who have faced similar tragedies in recent months will finally prompt the U.S. Government to take its responsibilities seriously and establish a coordinated and consistent approach to supporting hostages and their families." It's a criticism that has resurfaced as the families of American hostages were killed abroad spoke up and voiced their frustrations with what they characterized as insufficient U.S. government efforts to bring their loved ones home. Delaney was "saddened, disappointed and outraged that our government was not able to bring Warren home," he said in a statement Thursday. And in an interview with CNN later in the day, he strayed away from blaming the Obama administration or people in various U.S. agencies, instead pointing the finger at a disjointed system. The Weinstein family's frustration with that system boiled over last summer when Bergdahl, the U.S. Army sergeant, was released by a Taliban-affiliated group in exchange for five Guantanamo Bay prisoners. The U.S. policy of not negotiating with terrorists appeared to have an exception. "It's one of those things where it's both there's some hope, but at the same time it caused them some great frustration," Delaney recalled, saying the family asked: "Why not Warren?" But as the State Department called the Bergdahl exchange a "unique situation," the Weinsteins' hopes were dashed. "That's when they asked us to really step up our efforts," Sajery said. "That's when they decided to really go public." Delaney introduced a resolution calling on the Obama administration to use all tools necessary to bring Weinstein home and make his return -- and that of other U.S. hostages abroad -- a top priority. Mikulski, Cardin and Sens. Dick Durbin and Mark Kirk of Illinois pushed a similar resolution in the Senate. And the Weinsteins took to the airwaves, with Alisa Weinstein appearing on CNN's "AC360" to make a public appeal for her father's release -- and for the U.S. government to do more to secure his freedom. "My father is just as deserving of freedom as Sgt. Bergdahl, as are all of the Americans who are being held abroad," she said on CNN last June. "You cannot distinguish between these hostages. ... They can't just pick and choose, decide that it works to get one person out and then leave everybody else there." White House press secretary Josh Earnest on Thursday said that "significant resources" were dedicated to try to free Weinstein and that "as painful as it is," the U.S policy of not negotiating with terrorists would remain in place, arguing that removing that policy could promote kidnappings abroad and put more Americans at risk. And State Department acting spokeswoman Marie Harf said many officials at the department were in touch with the Weinsteins throughout the process. While the U.S. policy of not negotiating with terrorists would remain in place, the State Department is reviewing how it works to secure the release of American hostages held by terrorists abroad, she said. The White House announced the review last fall, which Obama ordered last summer after terrorists killed or kidnapped Americans abroad. Speaking at the Office of the Director of National Intelligence on Friday, Obama reiterated that the White House will review what happened. "We all bleed when we lose an American life," he said. "We all grieve when any innocent life is taken. We don't take this work lightly." Obama made to sure to praise the intelligence community overall, noting that much of their accomplishments remain classified, while only their failures become public. "The world doesn't always see your successes -- the threats you prevent, or the terrorist attacks you thwart, or the lives that you save," Obama said. "It can be frustrating sometimes, but that's part of the function of our democracy. But I know what you do." But Delaney, the Weinsteins' congressman, is revving up to push for more reforms that will make government agencies more effective at finding and freeing American hostages held abroad. Delaney on Thursday called on the government to streamline the efforts of various agencies and countries in the region that help the U.S. find American hostages -- an effort he's been pushing in recent months. The tragedy of Weinstein's death could be just the momentum needed to spur those reforms. "I think that every single American wants any American held hostage returned," Delaney said. "There's tremendous support to do more." On Friday, Elaine Weinstein said in a statement that the family has "been moved by the tremendous outpouring of support from around the world." "We appreciate the sympathy and condolences we have received from those who knew the Warren we loved so much as well as those who did not," the statement said. "Our thoughts and prayers are with the families of others who have been taken hostage around the world as they endure these terrible ordeals." CNN's Sophia Saifi reported from Islamabad, Pakistan; Jeremy Diamond reported and wrote from Washington; Saima Mohsin reported from Bangkok. CNN's Ben Brumfield contributed to this report.
About a year after al Qaeda took Warren Weinstein hostage, his family paid a ransom, a Pakistani source says .
The captors demanded that other prisoners be released, the source says .
Weinstein, an American aid worker, was killed in a drone strike in January, the U.S. says .