To overcome inconsistencies in partially cached websites, Archive-It.org was developed in 2005 by the Internet Archive as a means of allowing institutions and content creators to voluntarily harvest and preserve collections of digital content, and create digital archives.
(meta data) TITLE: Archive-It: Crawling the Web Together | Internet Archive Blogs 178 captures 01 Nov 2014 - 11 May 2022 About this capture The Wayback Machine - https://web.archive.org/web/20171012212827/http://blog.archive.org/2014/10/27/archive-it-crawling-the-web-together/ Internet Archive Blogs A blog from the team at archive.org Skip to content ← Media, Money & Elections: 2014 Philly Political Media Ad Watch Archive of Contemporary Music and the Internet Archive Team up to Create a Music Library → # Archive-It: Crawling the Web Together Posted on October 27, 2014 by jeff kaplan A post by the Archive-It team Today Phase 1 of the 5.0 release of the Archive-It web application was released for use by the 326 partners using the Archive-It service. In 1996 when the Internet Archive was founded, we used automated crawlers to capture the web, snapping up millions of web pages and preserving them for history. Ironically, our digital record of humankind was being driven by computer algorithms. As the years went by, it became clear that we needed people and communities to capture and save what is really and truly important. So in February 2006 we launched the Archive-It service, 1.0, which allowed traditional librarians and archivists to become web archivists by initiating focused, curated crawls of the live web using a simple web application with partner/tech support. Launching Archive-It meant we could help our colleagues create their own web collections for their own libraries and also foster a community around web archiving to work together to build a global digital public library at www.archive.org. Now, as we expand to the next generation of Archive-It with our 5.0 release, we hope to provide even greater tools for collection development. Released this week, 5.0 phase 1 highlights a shiny new user interface and significantly enhanced post-crawl reports that include infographics with visual representations of the data. Figure 1: Screenshot from the Reports section of the new Archive-It 5.0 user interface Back in 2006 there was little understanding of web archiving and many organizations were questioning whether this was a valid activity that could or should be a part of their larger institutional collecting strategies. After all, the challenges were staggering: the quality of web content was all over the map; conflicting policies and organizational structures posed challenges; no one had yet established best practices for selecting the content, how to handle metadata, or how to integrate this new type of content into other holdings and existing catalogs at the institution. Also, back then we could not have predicted the extent to which material that once existed in physical form would now only appear on the web in digital form. We launched the Archive-It service with a small band of believers and supporters, among them librarians and archivists from Indiana University, University of Texas at Austin, Library of Virginia, Montana State Library, and North Carolina State Archives and State Library. Partners were very patient with us and with Archive-It 1.0, which was bare bones. Collaborating and working with the library and archive community has always been a top priority for the Internet Archive, and a defining characteristic of the Archive-It service. There have been many times during the past 8+ years when we have not known the answer to a question and we say: "Let's ask the community and see what they think!" And the community has always gotten back to us with supportive answers – both illustrative and specific. Figure 2: Screenshot from the North Carolina State Government Web Site Archive of the North Carolina State Archives and State Library of North Carolina. As time went on, the community of web archivists grew and we were able to produce some compelling answers to the question: why web archive? Here are just a few: To create a thematic or topical web archive To fulfill a mandate to preserve institutional memory and history To archive state or local agency publications no longer being deposited in print form To archive records to meet university or government retention policies To preserve an historical record of an institution's web and /or social media presence To capture a website before re-design or it is taken offline To archive online art, exhibitions, and artists' materials Figure 3: Screenshot from the Latin American Government Documents Archive, LAGDA of the University of Texas at Austin. Figure 4: Screenshot from the Catalogues Raisonnés collection of the New York Art Resources Consortium (NYARC). To date in 2014, 326 Archive-It partners have created 2700 public collections on a diversity and range of topics, subjects, events and domains. These collections have become integral to these organizations' collecting strategies and have helped to raise awareness and understanding about why web archiving is so important. We like to say that the Archive-It service is both a partner and a vendor. We are a service provider and we strive to consistently deliver a high level of customer support — which we believe partners notice and appreciate. We also strive to be a partner to our community and work collaboratively on initiatives that we share together; a few of which are: a) collaborative efforts around archiving spontaneous events (like the 2011 Japanese Earthquake collection), b) teaching web archiving in graduate level MLIS programs and professional development workshops and c) the K12 Web Archiving program (now in its 7th year) where we work with 3rd to 12 graders around the county and ask them what they would like to archive for future generations. As one of the student archivists put it, "500 years from now, kids will think we were really cool." Many of the features and functionality that we see in the Archive-It service today are a direct result of a partner making a suggestion or request. Through face to face brainstorming sessions, online surveys, webinars, and support tickets, partners have expressed their ideas as well as offered constructive criticism. And we have listened. We hope that as the service continues to grow and we launch Archive-It 5.0 that many of our partners will see themselves in Archive-It. Their collections will continue to be valuable to researchers, historians, scholars and the general public for many years to come. Here are some links to just a few of those collections on the Archive-It website: Columbia University's collection on Human Rights: https://archive-it.org/collections/1068 National Museum of Women in the Arts's collection on Contemporary Women Artists on the Web: https://archive-it.org/collections/2973 University of Alberta's Circumpolar Collection: https://archive-it.org/collections/2475 Brigham Young University's Mormon Missionary Collection: https://archive-it.org/collections/3609 Stanford University's collection on Freedom of Information (FOIA): https://archive-it.org/collections/924 As we continue down this road – excited for the future and what comes next – we know that it takes a community to archive the web and we look forward to working with our partners to build libraries together. This entry was posted in Announcements, Archive-It, News. Bookmark the permalink. ← Media, Money & Elections: 2014 Philly Political Media Ad Watch Archive of Contemporary Music and the Internet Archive Team up to Create a Music Library → ### 3 Responses to Archive-It: Crawling the Web Together Madelaine Palko says: October 29, 2014 at 7:00 pm Such a great big project. I was checking and found an old website screenshot of mine from 2004. I had forgotten about it completely. I know you are mainly web-based but what about a Social Media interface? If only for promoting/fundraising your activities. Thanks. Pingback: Building Libraries Together: New Tools for a New Direction | Internet Archive Blogs Greg Bem says: November 5, 2014 at 6:24 pm Personally enjoy the continuation of the "collection" in archives. To "make sense" of the unlimited tides of digital data and digital information objects seems appropriately matched with the function of those collections. It also seems like Archive-It is an ideal aggregation platform for sharing those resources. I wonder how to better spread its word, though? Comments are closed. Search for: ### Recent Posts Syncing Catalogs with thousands of Libraries in 120 Countries through OCLC Boston Public Library's Sound Archives Coming to the Internet Archive for Preservation & Public Access Books from 1923 to 1941 Now Liberated! Internet Archive's Annual Bash this Wednesday! Get your tickets now before we run out! History is happening, and we're not just watching ### Recent Comments Brewster Kahle on Books from 1923 to 1941 Now Liberated!Brewster Kahle on Books from 1923 to 1941 Now Liberated!Chuck on Books from 1923 to 1941 Now Liberated!Warren on Books from 1923 to 1941 Now Liberated!Robert Nagle on Books from 1923 to 1941 Now Liberated! ### Categories Archive Version 2 Archive-It Audio Archive Books Archive Cool items Education Archive Image Archive Live Music Archive Movie Archive Open Library Software Archive Television Archive Video Archive Wayback Machine – Web Archive ### Archives Select Month October 2017 September 2017 August 2017 July 2017 June 2017 May 2017 April 2017 March 2017 February 2017 January 2017 December 2016 November 2016 October 2016 September 2016 August 2016 July 2016 June 2016 May 2016 April 2016 March 2016 February 2016 January 2016 December 2015 November 2015 October 2015 September 2015 July 2015 June 2015 May 2015 April 2015 March 2015 February 2015 January 2015 December 2014 November 2014 October 2014 September 2014 August 2014 July 2014 June 2014 May 2014 April 2014 March 2014 February 2014 January 2014 December 2013 November 2013 October 2013 September 2013 August 2013 July 2013 June 2013 May 2013 April 2013 March 2013 February 2013 January 2013 December 2012 November 2012 October 2012 September 2012 August 2012 July 2012 June 2012 May 2012 April 2012 March 2012 February 2012 January 2012 December 2011 November 2011 October 2011 September 2011 August 2011 July 2011 June 2011 May 2011 April 2011 March 2011 February 2011 January 2011 December 2010 November 2010 October 2010 September 2010 August 2010 July 2010 June 2010 May 2010 April 2010 March 2010 February 2010 January 2010 December 2009 October 2009 September 2009 August 2009 July 2009 June 2009 May 2009 April 2009 March 2009 February 2009 January 2009 December 2008 November 2008 September 2008 August 2008 June 2008 May 2008 April 2008 March 2008 February 2008 January 2008 December 2007 October 2007 September 2007 August 2007 July 2007 June 2007 May 2007 April 2007 March 2007 February 2007 January 2007 December 2006 November 2006 October 2006 September 2006 August 2006 February 2006 November 2005 October 2005 September 2005 March 2005 February 2005 January 2005 December 2004 October 2004 March 2004 ### Meta Log in Entries RSS Comments RSS
train24311
Wayback Machine
Technical details.
Software has been developed to "crawl" the web and download all publicly accessible World Wide Web pages, the Gopher hierarchy, the Netnews (Usenet) bulletin board system, and downloadable software. The information collected by these "crawlers" does not include all the information available on the Internet, since much of the data is restricted by the publisher or stored in databases that are not accessible.