Albert Lin is hunting for Genghis Khan. 
 
 Legend has it that Khan, the ruthless conqueror who was the first emperor of the Mongol Empire, was buried in an unmarked tomb in northern Mongolia about 800 years ago. 
 
 But finding said tomb is a task that has eluded scientists for years. Mongolia encompasses more than 600,000 square miles of largely uncharted, rural territory, which makes Lin’s mission an extremely challenging one. 
 
 Luckily, the explorer and research scientist at the University of California at San Diego has more than 7,000 people around the world helping with his mission, called the Valley of the Khans Project. The idea is to find the tombs of Genghis Khan and his descendants, and other ancient Mongolian artifacts. 
 
 Lin’s army of helpers are amateurs, working from the comfort of their home computers. 
 
 Through a Web site called Field Expedition Mongolia, which Lin and his colleagues developed jointly with National Geographic, volunteers are helping sift through 85,000 high-resolution satellite images of Mongolia. 
 
 Every time volunteers log in to the site, they are shown some of these images. An online tutorial instructs them on how to look for particular objects and tag them as “roads,” “rivers,” “modern structures” or “ancient structures.” They can zoom in and out and scroll in all directions. 
 
 They are also told to simply tag places as “other” if they see something peculiar. This is the sort of vague judgment that humans can perform but that computers cannot, Lin said. 
 
 “What a computer can’t do is look for ‘weird things,’ but when you ask a human brain, you don’t have to tell it what ‘weird’ is; we know,” Lin said. 
 
 Those weird things could be important archaeological finds, he said. 
 
 Last summer, Lin and his colleagues were in Mongolia inspecting the places that had been tagged by the online volunteers. Anytime there was a cluster of tags marked as “ancient structure” or “other,” they would note the latitudinal and longitudinal coordinates, grab their GPS devices and scope it out. 
 
 “We’d literally jump on horses or get in a helicopter and go check it out,” said Lin. “Every tag was weighted on how many other people tagged the same thing.” 
 
 Projects like this one mark a new twist in “citizen science,” where new technology, when used effectively by large groups of people, can help speed up scientific developments, reduce costs and increase efficiency. 
 
 Sometimes online volunteers led the explorers to disappointing finds, such as a herd of sheep on a satellite photo that looked like an ancient structure. But there were also some remarkable ones, such as the discovery of 3,000-year-old Bronze Age tombs, remnants of large cities and ancient monoliths hidden in the region’s vast, grassy steppe. 
 
 “These are hard to find on horseback, but from space and in the images, you can make out these shapes,” Lin said. 
 
 Making it fun 
 
 Though professional scientists have collaborated with amateurs for decades, social networking and the Internet are making it more fruitful than ever. 
 
 “We found that we could make something that was engaging enough to inspire people to participate without having to pay them,” says Lin. “This is the part of citizen science that is most interesting to me: How can we motivate people to dedicate their time?” 
 
 How? By making it fun, Lin said. 
 
 Lin began thinking about creating an online expedition that tied into his real one about five years ago, when Amazon.com’s Mechanical Turk made its debut. 
 
 An online crowdsourcing marketplace, Mechanical Turk allows requesters with small tasks to pay people for their time. Anyone with an Amazon account can participate, and the tasks are usually quite simple, such as “pick out the images with tattoos from this set,” or “verify the existence of these business Web sites.” Some tasks pay just pennies, per task or verification, while others pay more. 
 
 Lin believed that he could get more traction by creating a site that offered a fun experience rather than a paid one. “People are so excited to learn about Mongolian archaeology,” he said. “They start to learn stuff about what they’re doing and feel more connected to what’s going on in that part of the world.” 
 
 Every volunteer who logs on to the Valley of the Khans project site, developed with a design company called Digitaria, gets to feel like an explorer, digging through images and playing what feels like a game but performing work that has much more significant ramifications. 
 
 “It connects you more on a personal level than going to a museum,” said Allison Shefcyk, a 24-year-old in Connecticut who tagged more than 50,000 images from her home computer. “I ended up picking up some books on Genghis Khan and Mongol culture, and even though I never set foot there, it all provided a deeply moving experience." 
 
 Biomedicine and galaxies 
 
 The Khan expedition is not the only research project engaging people on their home computers with gaming strategies. 
 
 Another is EteRNA, created by researchers at Carnegie Mellon and Stanford universities. It allows players to tinker with nucleotide bases and come up with synthetic RNA designs. 
 
 The creators of the site hope that by generating a large assortment of designs, they can speed up discoveries in biomedicine. Every week, the most promising designs are actually synthesized by scientists in a lab at Stanford. 
 
 An older project, Galaxy Zoo, allows volunteers to help classify images of galaxies taken by a robotic telescope in a project called the Sloan Digital Sky Survey. 
 
 There were thousands of images to classify, so researchers decided to create a crowdsourcing tool that uses the public’s help in making classifications. 
 
 To participate, volunteers flip through images and answer simple questions, such as whether a certain galaxy looks completely round, partially round or cigar-shaped. 
 
 A year after its July 2007 launch, Galaxy Zoo had more than 50 million classifications from more than 150,000 people. The project so far has generated more than 20 academic papers by researchers around the world in astrophysics and astronomy journals. 
 
 And birds, too 
 
 Another project, run by the U.S. Geological Survey, is using crowdsourcing and technology to digitize a project called the Bird Phenology Program. 
 
 The project was started by Wells Cooke, an American ornithologist who wanted to gather information on bird migration. Starting in 1881, amateur bird-watchers mailed in thousands of index cards detailing information on birds they had seen, first to Cooke and then to the American Ornithologists’ Union. The federal government maintained the program in its final years, but participation declined and it was closed in 1970. 
 
 But the cards, and the wealth of information they contain, remain in file cabinets. The U.S. Geological Survey is scanning the cards, and volunteers can log on and enter the information into a central database. 
 
 It’s secretarial work, not rocket science, said Jake Weltzin, executive director of the National Phenology Network, the USGS-funded organization that is coordinating the program, but “people really like to contribute to the scientific process and really feel they are contributing to a bigger project,” Weltzin said. 
 
 “These are all volunteers; one person has digitized 20,000 cards,” Weltzin said. “We couldn't afford to pay her.” 
 
 Eventually, researchers will be able to use the data to study population changes, and perhaps better understand the effect of climate change on birds and their habitats. 
 
 Shefcyk, the enthusiastic volunteer in the Valley of the Khans project, said that participating was particularly meaningful to her. 
 
 As a child, Shefcyk had trouble fitting in with other kids and was eventually identified as having an autism spectrum disorder. But she spent her time devouring books on archaeology and reading nonstop about dinosaurs, Egyptian pyramids and Mayan ruins. 
 
 “I didn’t have many friends, and archaeology was the whole world to me,” she said. “I would always start conversations and ask things like ‘Who’s your favorite Mayan king?’ ” 
 
 Flash forward 15 years and Shefcyk is still fascinated by archaeology and is a frequent visitor to the National Geographic Web site. When she found out that ordinary people could help Lin with his project, she was intrigued. So every day last summer, while Lin’s team was in Mongolia, she would tag images and look out for blog posts from the scientists. She was enthralled to read their chronicles and get quick feedback on the sites they visited. The explorers also offered tips to help volunteers tag objects more accurately. 
 
 “It’s one of those things where you’re adding your piece, and it’s about knowing that you’re something that’s much bigger than yourself, no matter how small the involvement,” she said. 
 
 As for whether any of the tagging has gotten him closer to finding the burial site of Genghis Khan or one of his successors, Lin is coy. “I can’t say yet what we found,” Lin said. “We’re in the midst of compiling the research.” 
 
 Bhanoo is a science writer based in the San Francisco Bay area. ||||| Abstract Massively parallel collaboration and emergent knowledge generation is described through a large scale survey for archaeological anomalies within ultra-high resolution earth-sensing satellite imagery. Over 10K online volunteers contributed 30K hours (3.4 years), examined 6,000 km2, and generated 2.3 million feature categorizations. Motivated by the search for Genghis Khan's tomb, participants were tasked with finding an archaeological enigma that lacks any historical description of its potential visual appearance. Without a pre-existing reference for validation we turn towards consensus, defined by kernel density estimation, to pool human perception for “out of the ordinary” features across a vast landscape. This consensus served as the training mechanism within a self-evolving feedback loop between a participant and the crowd, essential driving a collective reasoning engine for anomaly detection. The resulting map led a National Geographic expedition to confirm 55 archaeological sites across a vast landscape. A increased ground-truthed accuracy was observed in those participants exposed to the peer feedback loop over those whom worked in isolation, suggesting collective reasoning can emerge within networked groups to outperform the aggregate independent ability of individuals to define the unknown. 
 
 Citation: Lin AY-M, Huynh A, Lanckriet G, Barrington L (2014) Crowdsourcing the Unknown: The Satellite Search for Genghis Khan. PLoS ONE 9(12): e114046. doi:10.1371/journal.pone.0114046 Editor: Michael D. Petraglia, University of Oxford, United Kingdom Received: April 22, 2014; Accepted: September 16, 2014; Published: December 30, 2014 Copyright: © 2014 Lin et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. All relevant data have been uploaded to the Harvard Dataverse and can be accessed at https://thedata.harvard.edu/dvn/dv/votk. Funding: This work was funded by National Science Foundation (EAGER ISS-1145291 and HCC IIS-1219138). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. 
 
 Introduction Ultra-high resolution satellite imaging enables a new paradigm in global exploration. This study surveys sub-meter resolution satellite imagery of the Mongolian steppe to identify largely undocumented cultural heritage sites across a sparsely populated and undeveloped landscape. With continued advances in sensor technologies, the capabilities and limitations of remote sensing is being determined less by data resolution and more by the methods that analyze the increasingly massive datasets. Overwhelming data volumes have often led to automated analytical approaches. However, in visual analytics automated approaches lack the flexibility and sensitivity of human perception when seeking singular, undefined anomalies. This study therefore utilizes scalable, loosely guided, online volunteer participation to generate human identifications of unknown anomalies within massive volumes of geospatial remote sensing data. The emergence of statistical trends from a large sample of independent inputs highlights the collective human perception of the images' content. Similar to volunteer geographic information networks such as OpenStreetMaps [1], inputs from multiple contributors generate a collective map of local knowledge [2], only in this case the term “local” represents the global base of human visual perception. Turning to the crowd as the “partner of choice” for scalable problem solving is becoming increasingly attractive across broad domains both in science and industry [3], [4]. While many crowdsourcing approaches rely upon gameplay dynamics [5], [6], monetary incentives (e.g., Amazon's Mechanical Turk [7], [8] or [9]) to motivate participation in individual tasks, others utilize the social recognition of charismatic challenges such as space exploration [10] to encourage participation. It has been shown in several of these cases that complex problems, if strategically structured, can be solved by pools of volunteer participants with little to no pre-existing domain knowledge of their analytical challenge [11]. Here we leverage the charismatic challenge of directing a field expedition in search for the tomb of Genghis Khan (Chinggis Khaan) to engage tens of thousands of public volunteers and generate millions of micro-contributions towards an archaeological satellite imagery survey. These contributions aggregate into a collective geospatial map of analytical cross-verifications. In this particular case, since there exists no historical or archaeological record describing the physical appearance of this tomb, we turn towards the crowd not only to tackle the data size challenge of large scale satellite remote sensing, but more importantly to pool human perception and intuition when sifting through the data for anything that looks “out of the ordinary”. This loosely defined search criterium presents a challenge, where a participant's desire for validation or guidance cannot be met with an administrative answer. Thus, we introduce a framework to enable a crowd directed evolution of feature search criteria that mimics crowd reasoning processes in nature. The collective behavior of flocking animals, for example, is enabled through simple decision making and feedback criteria adhered to by the independent agents of that collective [12]. Similar to the vector and spatial referencing data required by individual birds within a flock, our study constructs multi-directional channels of visual reasoning feedback between the individual participant and the crowd. We hope that in this scheme, the collective will guide itself towards an agreement regarding what to look for, essentially facilitating a “collective intelligence” beyond that of individuals whom comprise the collective [13], [14]. We explore this in the case of perceptual reasoning. 
 
 Discussion We have described a crowd based solution to a satellite imagery remote sensing challenge of both data volume and search target ambiguity. Specifically, we charged an online crowd of volunteer participants with the challenge of finding the tomb of Genghis Khan, an archaeological enigma of unknown characteristics widely believed to be hidden somewhere within the range of our satellite imagery. This is a needle in a haystack problem where the appearance of the needle is unknown. To address this constraint we designed a system where participants actively evolve the collective training base of user feedback with their own inputs. Furthermore, the framework of the system created a resilient and self-validating data source through massively paralleled and constrained user inputs. Thus, we rely upon the emergence of agreement regions from independent tags to guide both the online volunteer community as well as the field archaeological expedition that surveyed anomalies on the ground. The entire data set is distributed in randomly chosen image tile subsets to participants, thus a “global” kernel density estimation approach is introduced to normalize saliency across all image tiles and to create an overall agreement region ranking. This concept could be applied as a distributed voting framework, where overlapping subsets allow for large data sets to be subdivided and parsed among many voters and then recombined into a single collective vote. Of the top 100 accessible locations identified by the crowd, 55 potential archaeological anomalies were verified by the field team, ranging from bronze age to Mongol period in origin. Yet, the question remains: could these results have been obtained just as effectively or more effectively without crowdsourcing? Or more specifically, could a small team of trained archaeologist have found the anomalies quickly by visually scanning the images on their own? After all we did expend 30K hours (3.4 years) of collective human survey effort. Looking first at the data size challenge, we have surveyed a historically significant area of roughly 6,000 km2. This is twice the size of Yosemite National Park, with equally diverse geologies and significantly greater in-accessibility. A ground survey of this detail for the entire range would have been prohibitive. Yet, at 0.5 meter/pixel resolution, a satellite imagery survey of the same area is in itself exhausting. A single archaeologist would have had to scroll through nearly 20,000 screens (assuming 1280×1024 screen resolution) before covering the whole area. But putting the data size challenge aside for a moment, we can observe the crowd's ability to be sensitive yet flexible. We continue to emphasize that very little is known about the likely visual appearance of the search target. Thus, we cannot limit our search criteria to what is traditionally “expected” from the known literature. Here is where the authors believe the power of crowdsourcing lies not only in harnessing parallel networks for scalable analytics, but in forming the collaborative frameworks necessary to cultivate collective reasoning. We depend on the crowd to process and identify the unexpected. Within our framework we observed that when participants were not provided the incremental peer based feedback loop they were statistically less likely to positively identify these anomalies, suggesting a form of collective reasoning has emerged within our participant pool that is variant and potentially more effective than the accumulated independent reasoning of individuals within that pool. Furthermore, while we acknowledge that there may be anomalies that remain undetected, this statistical variance suggests that largely parallel analytics (crowdsourcing) can provide better outcomes than an individual survey. While this study focused on an archaeological survey, there exists a broad range of challenges where scalable human perception networks could be effectively applied. These concepts have been further explored in applications ranging from humanitarian response to search & rescue [28]–[30]. The activities have not only tapped into our connectivity to scale human analytics, but also for the social mobilization of human attention [31], [32]. A recent direct derivative of the effort described here can be seen in Digital Globe Inc.'s “Tomnod” (Mongolian word meaning “big eye”) survey for the missing Malaysia Airlines flight MH370, where over 8 million participants surveyed over 1 million km2 of ultra-high resolution satellite imagery for anomalies. The shear mass of participation in this example provides a glimpse of the potential of our networked society. These crowdsourcing activities help us dive into the unknown and extract the unexpected. However, beyond that they present a fundamentally new construct for how we, as a digitally connected society, interact with information. The ability to focus and route networks of human attention at such massive scales, coupled with the functional ability for meaningful micro-contributions at individual scales, presents yet another evolutionary step in our collective ability to reason. 
 
 Acknowledgments We thank N. Ricklin, S. Har-Noy, K. Stamatiou, and the entire Valley of the Khans (VOTK) project team; S. Bira, and T. Ishdorj of the International Association for Mongol Studies and F. Hiebert of the National Geographic Society for co-leadership in field expeditions; D. Vanoni, K. Ponto, D. Lomas, J. Lewis, V. deSa, F. Kuester, S. Belongie, R. Rao, and E.A. Spencer for critical discussions and contributions; S. Poulton and A. Bucci with National Geographic Digital Media; and the entire Digitaria team. We sincerely thank the National Geographic Society, the Waitt Institute for Discovery, the Digital Globe Foundation (formerly the GeoEye Foundation), and the National Science Foundation.
– Armchair history buffs, take note: A University of California scientist needs all the help he can get finding the tomb of Genghis Khan. The only thing participants need to join the Valley of the Khans Project is a computer and an Internet connection, reports the Washington Post. Under the program, volunteers go to the Field Expedition Mongolia website to sift through satellite images of Mongolia and flag things that look unusual. “What a computer can’t do is look for ‘weird things,’ but when you ask a human brain, you don’t have to tell it what ‘weird’ is; we know,” says Albert Lin. About 7,000 volunteers are already taking part in the program, which was developed with National Geographic. Lin and other explorers on the ground visit sites that get enough flags, but so far the resting place of Genghis and his family remains a mystery.