1991, the year of the first call in the Libraries Programme, was a very different time; the network infrastructure was being built, computers were becoming more powerful, and the information society was being formed based on different technological solutions. Standards (SR, Z39.50, HTML, MP3) and protocols (TCP/IP, OSI) as to how bits should be transported and interpreted, how programs on computers should communicate and how one could find the relevant information (WAIS, Gopher, WWW) were being developed, tested and deployed. Questions were asked that pushed boundaries, experiments were conducted, which delivered new possibilities, and progress on standardization was made. Much happened in the 1990s in a short time span.
Researchers and librarians inside the numerous institutions benefitting from participation in the first two framework programmes also had an experience of not only being part of a European adventure, but being part of a totally transformative period, where the technological possibilities were exploding.
1991 was the year in which the first World Wide Web (WWW) servers outside CERN were switched on and the year, where Tim Berners-Lee posted a short summary of the WWW project. 1991 was implemented by CERN (European Organization for Nuclear Research), it was the year in which Wide Area Information Servers (WAIS) were invented, Gopher was released and the first version of the pretty good privacy (PGP) code was released. It was a hectic time with a lot of experimentation and it was a time, in which researchers, librarians, and technicians took risks to explore possibilities.
Experiments, prototyping and implementations happened in the research communities, among established and new companies, and in institutions such as archives and libraries. A lot was learned. At the same time; however, technical failures were unavoidable; both because decisions were made as the technology was being developed and at the same time as the standardization work was being initiated, but also because the official politics on standards collided with the development outside the corridors of the European Commission.
One such example was the battle between the two competing standards for the network, OSI (Open Systems Interconnection) and TCP/IP (Transmission Control Protocol/Internet Protocol); a competition which resulted in parallel development of applications and initiation of gateway-projects. In 1991 the Search and Retrieve (SR) and Interlibrary Loan (ILL) protocols were completed. They were the first international standards for application protocols specially designed for the information community. SR was implemented on top of the OSI framework for communication—in parallel with the American development of a similar protocol, Z39.50. Another example from this period is the competition between Gopher and WWW for dominance as the mechanism of identifying digital documents and serving them to the end-user.
Support was given to projects working with what turned out to be pre-mature technology and with protocols, which changed and developed as the community accepted and adopted the stateless WEB with the underlying TCP/IP and HTTP protocol and the HTML standard. Were they wasted? I would (and will) at any time argue for a "no!". From the perspective of a participant in this process it is evident that without the courage and money to experiment and gain experience, we would never have advanced and learned to master the technology. By being part of the game, the library community not only learned and used the technological possibilities, but also encouraged the European society toward information openness and pushed forward the agenda for the open science and open culture.
The projects resulting from the first call addressed different aspects of the technological possibilities. They experimented with possibilities including interconnecting library catalogues and with ways to localize and deliver digitized objects such as articles, video and high quality sound.
One of the projects in which I was involved, JUKE-BOX, carried multiple risks as it attempted to push boundaries to "apply telematics technology to improve public access to audio archives": the methodology for compressing the music and the communication technology guaranteeing a sufficient bandwidth to carry the music. The success of the project enabled three sound archives to gain an early start on the use of what turned out to be the winning music delivery technology.
Looking back at and reading about the projects done in the early and mid 90s, one thing strikes me more than anything else—the willingness to take a risk. This high risk appetite was evident in the perception of the European Commission and their proposal evaluators, researchers, and in the cultural institutions acting as content providers, end-users, and technology testbeds. It is remarkable how willing the Libraries Programme was to fund projects which experimented with the latest technological development, often doing so before they were fully developed. I am confident that these years of experimentation and competence building did much to stimulate the digital evolution in the library sector at large—led by the adventurous academic and national libraries, which due to their proximity to the research community had good access to the required technical competences.
Key to these successes was certainly the pragmatic attitude of the members of the library team at the European Commission's Luxembourg Offices, who helped in ensuring valuable results. My impression working with the Luxembourg office in those early years was one of trust in the project partners, acceptance of experimentation, and one of a search for pragmatic solutions.
I first met Pat Manson at a workshop arranged by Ariane Iljon (Head of Unit in Directorate General XIII, responsible for the Libraries Programme), and her staff in 1992. I don't remember the exact topic, but it must have been around interoperability of library catalogues. I remember making an argument in favour of the TCP/IP protocol as opposed to the OSI protocol favoured by the European Commission. Perhaps, in another context this might have been a risky move, as the official Commission policy was to use OSI. I therefore expected that my remark would end pre-maturely my engagement with the Commission in Luxembourg; but on the contrary, as I was told later, it opened the door. In my many years of contact and engagement with Pat and the staff in Luxembourg, I came to appreciate their openness to alternative views and their willingness to engage stakeholders in forming new strategies and defining activities.
In this article, I will address this period and some of the main technological questions which were the focus of EU co-funded research and development projects. These included addressing questions as to how to cope with different protocols, the library quest to offer user-friendly access to the processes of identifying and obtaining documents, and sound transmission experiments.
This essay reflects how rapid the development evolved and how uncertainty about the right technological choice evolved to certainty. It all started during the Third Framework Programme (FP3) during the years 1991–1994 and was settled during the Fourth Framework Programme (FP4)1994–1998.
The first call in the FP3 libraries programme was structured into four action lines within which around 50 shared-cost cooperative projects were launched.
Looking at the titles of the projects funded FP3 (call 91–93)2 it is clear, that the main interests of the participating institutions and companies were in the market stimulus for products and services for libraries (action line 4) with projects such as "MARC Optical Recognition", "Electronic Library SGML Applications", "Bibliographic Records and Images: a CD-ROM of Incunabula Editions" and "Advanced Tools for Accessing Library Catalogues" but also experimental projects working with audio, video or generally multimedia. I shall return to one of these projects, JUKE-BOX, as it demonstrates the openness towards experimentation and illustrates the engagement and spirit of that time. JUKE-BOX was one (if not the first) real user of the MPEG Layer 3 standard—what later was named the MP3-standard.
Action line 2, with only 8 projects, was the home of projects like Electronic Document Interchange for Libraries and Booksellers in Europe, 3 SR-target projects, where I will return to one, SR Origin Communication Kernel (SOCKER) and a gateway-project between the search- and retrieval standards as they were developed in US and in Europe, EUROPAGATE.
I enjoy reflecting on the enthusiasm and energy characterizing this period and I appreciate the effort of the office in Luxembourg to ensure the budget to make this happened. Libraries were in the front when it came to adopt and adjust to the technological possibilities.
OSI (Open Systems Interconnection) is a reference model developed by the International Standard Organisation (ISO) with heavy support from the European telecompanies for how applications can communicate over a network. Applications associated with OSI are FTAM (File Transfer Access and Management), X.400 for the mail system and X.500 for the required directory service etc.
TCP/IP is an implemented protocol with only four layers in contrast to the seven layers in the OSI protocols. Applications associated with TCP/IP are FTP (File Transfer Protocol), SMTP (Simple Mail Transfer System), DNS (Domain Name System), and HTTP—the latter being the basis behind the World Wide Web.
"Resolution: The Federal Networking Council (FNC) agrees that the following language reflects our definition of the term "Internet".
The intention behind the OSI reference model was to encourage vendors and developers to develop interoperable communication products—some argue as a reaction to big vendors such IBM [e.g., Systems Network Architecture (SNA)] and DEC [e.g., Digital Data Communications Message Protocol (DDCMP)] who offered proprietary solutions which increased costs and limited cross-system interoperability. The main idea in OSI is that the process of communication between two end points in a telecommunication network can be divided into layers of specialized functions. Protocols developed according to the OSI Reference Model are referred to as "OSI protocols" or as belonging to the "OSI suite of protocols".
TCP/IP on the other hand was a practical implementation of a network protocol initiated in the USA at the inspiration of Bob Kahn.
There are similarities between the two and there are differences.
"For a number of years there was a consensus that, ultimately, networking based on OSI standards would supersede TCP/IP. This would be primarily the results of government policy, through GOSIP (Government Open Systems Interconnection Profiles), and also because of the greater functionality offered by the OSI applications such as X.400, FTAM, and Virtual Terminal.
It is no longer certain, however, that this will necessarily be the case. Since 1990 there has been a very rapid growth in the size of the TCP/IP Internet. In particular, the use of TCP/IP for wide area networking is now almost universal in the North American academic community, and has increasing presence in the European academic community. This is significant because it means that most of the active research into networking and interoperability issues is being carried out in a TCP/IP environment. Additionally, the U.S National Research and Education Network (NREN), the planned, high capacity network of the latter 1990's, is almost certain to be based on TCP/IP, at least initially.
The research libraries initially led the uptake of the network within the library community as illustrated in Figure 1, which illustrates the situation in 1995, the year after the end of the European Commission's Framework Programme Three. As the figure makes evident, the research libraries quickly followed their research institution and offered Internet access to many. The situation among the public libraries was different with only 71 out of 244 libraries being connected. The availability of a homepage was even poorer, with only 2% of the public libraries having a homepage.
Statistics for access to the Internet and for availability of homepages by the end of 1995. The figure is based on numbers from an investigation done by the Library Authorities (Biblioteksstyrelsen, 1996).
In the early 1990s, the networks were mainly used to transfer files and to support the exchange of electronic mail, as is illustrated in Figure 2.
For a long time file transfer and mail were the dominant reasons to move bits over the Internet, as illustrated here, based on figures from Merit (from Christensen-Dalsgaard, 2005; the original webpage has been removed). This situation began to change in 1992 with the advent of new protocols such as Gopher and WWW.
One may say that an objective of the first call of the Libraries Programme was to change this. As can also be seen from the figure, traffic soon was driven by other applications such as gopher and WWW.
Below are two stories relating to this change—one relating to locating information and one about delivering non-text-based information.
The information landscape in 1991 was rapidly growing with hundreds of databases and library catalogues coming online for the first time, but many were still distributed on CD-ROMs. This was particularly the case in Higher Education where the 2 Mbps European Research Network backbone was considered a remarkable transmission capacity. But the gradual increase of the bandwidth, and a growing desire to use the network to provide or access up-to-date information—or just information—stimulated a series of initiatives to change network provision. The result was Gopher, Veronica, etc., which greatly influenced the possibilities for libraries, and their information providers and users to distribute, access, and share information.
A parallel trend focused on capitalizing on decades of investment in detailed cataloguing of physical objects, based on cross searching the information in these catalogues leading to the development of protocols for search and retrieval, transforming such services as interlibrary loan.
I will here focus on three in many ways parallel developments all in different manners addressing the burning question of identifying and retrieving relevant information.
SR/Z39.50: The library oriented, structured approach to extract information from library systems.
Gopher: A structured way of extracting in a systemized manner information on a file basis.
WWW: A hypertext approach to interrelate information and information servers.
Let me provide some examples of each of these initiatives.
In 1991 the Search and Retrieve (SR) and Interlibrary Loan (ILL) standards were completed; these were the first international standards for application protocols specially designed for the information community. SR was implemented on top of the OSI framework for communication—in parallel with the American development of a similar protocol, Z39.50, which was approved as an ANSI standard in 1992. The development of these two standards was co-ordinated closely (the same editor drafted both standards) and the 1992 version of the Z39.50 standard has the SR standard defined as a compatible sub-set of Z39.50.
One objective of the SR-protocol was to allow the construction of so-called SR-clients, which in a unified manner could access a number of different databases, the so-called SR-target. These could be OPACs (Online Public Access Catalogue), CD-ROMs or other databases. The SR-clients even allowed users to make simultaneous searches of different databases.
As I noted earlier, this was not the case at the outset of Framework Programme 3. As the protocol landscape changed, projects such as EUROPAGATE redefined their activities and some, such as SOCKER, provided input to the creation of a common user-interface to different resources, and through their experiments produced valuable information in areas such as stateless and statefull protocols.
The prime contractor of SOCKER was UNI-C, the Danish Computer company for research and education, where I was employed at the time. As the operator of the Danish Research Network we were responsible for the TCP/IP-based research network—and all colleagues at our institution expected the TCP/IP protocol to win. To secure the value and reusability of our research and services, we adopted a strategy of using ISODE (ISO Development Layer), which allowed us to run OSI over TCP/IP.
The experience in SOCKER paved the way for an ambitious initiative in Denmark, The Danish Electronic Library (deff.dk). Here the objective was to use the Z39.50 protocol to produce a unified search of the catalogues of Danish Libraries. For a short time, the service was operational. Its long-term continuity was cut short by a combination of library vendors never fully supporting the required functionality and the growth in computer power which enabled other initiatives based on unified indexes to take off.
"The SRW (Search & Retrieve Web Service) initiative is part of an international collaborative effort to develop a standard web-based text-searching interface. It draws heavily on the abstract models and functionality of Z39.50, but removes much of the complexity. SRW is built using common web development tools (WSDL, SOAP, HTTP and XML) and development of SRW interfaces to data repositories is significantly easier than for Z39.50. In addition, such arcane record formats as MARC and GRS-1 have been replaced with XML.
In a period, up to about 1994 we witnessed something that may well be called a strategy war between Gopher and WWW. The battle is illustrated by the two curves based on figures from The Internet Society. Here one can see that Gopher clearly had more supporters in 1993, but this completely turns in 1994. A strong contributory factor was the development of Mosaic, which was launched in November 1993.
Gopher and, eventually, WWW, address an issue of access which arose at the time as the research community was fast in adopting the possibilities offered by the digitalization and hundreds if not thousands of collections where made accessible in public repositories known as anonymous FTP servers. FTP is the Internet-standard high-speed file transfer protocol, used for exchange of private information by trusted parties with passwords as well as for publishing information without passwords, i.e., anonymously. The burning question was how to identify the needle in the FTP haystack or haystacks.
The first step towards a solution for identifying material came in '89, when the first search engine on the Internet, Archie, was launched. Archie (ARCHIE server) was developed at McGill University to index the contents of all FTP servers and to provide keyword searching of the index. Its approach was simple but powerful: Every night re-indexed roughly one thirtieth of the servers; the result was a database that was completely refreshed every month. Archie provided users with the ability to input the name of a file (or program) and then obtain information as to what machine it resided on. Archie was a Unix product and required some technical knowledge, which was not a problem for IT people, but probably a barrier to wider deployment.
WAIS, Wide Area Information System, was launched almost at the same time as Archie. WAIS was a joint project of Apple Computer, Dow Jones, KPMG Peat Marwick, and Thinking Machines Corporation; its development is attributed to Brewster Kale, then at Thinking Machines. WAIS provides a uniform interface to many full-text databases, together with a sophisticated "relevance search" capability.
The idea of the WAIS was to create an index on the basis of the full text of the documents. There were several different implementations of the WAIS with different levels of complexity and capability. At its height, pointers to over 600 databases, including Usenet's FAQ and the full documentation behind all standardization proposals about the Internet were maintained. As with Archie the use of WAIS required some technical knowledge.
The first initiative aimed at offering the not so technically proficient with these kinds of services was Gopher.
Gopher began as the University of Minnesota's version of PennInfo, a menu-driven campus-wide information system (CWIS). Gopher's simplicity as a distributed, client/server CWIS led to its rapid adoption by other institutions, some of which developed new client or server software for desktop or host computers and contributed them to the Gopher software archive (accessible via anonymous FTP, naturally). Soon thereafter, Minnesota offered to provide a menu of all Gopher servers that any other Gopher could access. The result was what the research community had been wanting for years: an interoperating set of information systems linking several hundred organizations around the world, all with a common user interface!
At that time, it felt like a revolution. You started via the interface gopher.denet.dk (shown in Figure 5) on a machine in Lyngby and after a few menu choices you ended up at a server in the United States or Australia. Navigation happened via a series of lists and using the keyboard, "U" would bring you up one level up, "M" would give the main menu and "Q" would quit the session. Easy, logical and straightforward—we felt then.
Gopher was only a tool to browse through hierarchical structures and did not allow search (see Figure 4). This came with a different tool, Veronica, which gave the opportunity to scour Gopherspace. Often a menu item in Gopher listings was "Search Gopherspace with Veronica".
Opening screen of gopher.denet.dk (Figure from a talk).
Many initiatives were taken to improve Gopher and for several years it looked like Gopher was the winning strategy and in response Gopher servers were implemented in many libraries. As was illustrated in Figure 3, it was overtaken in March 1994 by the World Wide Web or short: WWW.
As is widely known, the World Wide Web was proposed by Tim Berners-Lee (TBL) in 1989, who at that time was working at the European Centre for Nuclear Physics, CERN. The challenges TBL had to go through and the mistrust of the directors of CERN is also a well-known story.
The proposal was implemented and in November the same year unveiled as the first WWW program. The first implementation was a browser, which also supported WYSIWYG (What You See Is What You Get) editing. The implementation was done on a revolutionary new computer, NEXT (which unfortunately did not survive) with the possibility of full screen menus, where you could point and click. The following year, software was written so that it could run on other platforms. The first use of WWW in Denmark was as a front end to the library at the Technical University in Denmark in November 1992 (see Figure 5). It was hand-carried by two librarians, who up to then worked at CERN and had participated in the development there. Those, who saw the implementation, were very impressed and enthusiastic; however, WWW's graphical interface only worked on the NEXT computer and the enthusiasm when seeing the line oriented version turned into caution—and the work based on Gopher applications continued.
WWW-based front end for the National Technological Library of Denmark in 1992.
A dramatic step forward came with the advance of user-friendly browsers for the UNIX and the PC-community. In 1993 Marc Andersen and his team invented the first graphical browser, Mosaic. The creation of this methodology for navigation was a game changer. From now on, the use of the WWW outnumbered all other navigation methods.
An example often used in 1994 and 1995 to show the attractiveness of WWW was the navigation interface of the Norwegian Research Network, UNINETT. I have shown the original Danish version from 1994 in Figure 6 and the coloured version from 1995 in Figure 7.
The navigation webpage from the Norwegian Research Network, UNINETT. This is the original version from 1994, which only exists as a black and white overhead in the author's files.
The navigation webpage from the Norwegian Research Network, UNINETT, in its English version from 1995 (private communication).
The quest of good use of the network led to the project JUKE-BOX. It was funded in the middle of the development of the compression algorithms for music, which were needed to transmit digitized audio-visual material over the net—and via the input channel of playing devices such as the DVI, which was being developed at that time by Philips. A lot of actors worked on different aspects of the MPEG standard; when it came to Audio, Fraunhofer was a clear leader.
JUKE-BOX's aim was to allow researchers to search for and retrieve music in different archives and then to compare different versions especially different versions of the same piece. In1991 there was little experience with compressing music sufficiently for it to be transmitted via the public networks—and expecting it to sound well.
As an extra complication, the choice of network was not that simple. While the use of TCP/IP based networks was investigated it was concluded, that although all believed that future services would be based on TCI/IP, using this protocol at the time would pose unacceptably high risks. The JUKE-BOX project therefore ended up choosing to use native ISDN for the music transfer to ensure sufficient and constant bandwidth. The search was done using the SR-software developed in SOCKER and was using the TCP/IP based network (a facility added in the follow-up project SR-TARGET/PARAGON (SR Target Development as a paragon for Catalogue Systems).
After selecting the transport protocol, the project then faced the challenge of compressing high quality music into 64 kbps per Channel—in total 128 kbps as we wanted everything to be in stereo. The first tests of using the existing standard, MPEG layer II, did not go well—the quality was too poor to be relevant for the purpose of the project. We therefore got in contact with Fraunhofer and borrowed an encoding card (it was huge—a full size card for a standard PC) for real life tests of the quality. We wanted to test this new approach being developed by Fraunhofer and based on pseudo-acoustic methods. I vividly remember the afternoon when in the basement of the State and University Library we tested the encoding on several types of music—and the sense of exuberant relief , when the critical testers decided that the methodology worked!
It is therefore quite interesting to actually look at the timeline for the development of what became the mp3-format—and I am amazed to realize, that we must have been among the first customers.
1992:MP3 is a reality. The ISO includes it as one of a number of possibilities for encoding audio. But for some time, Brandenburg says, it didn't catch on.
1993–1994:The system is in place. Where are the users?
It required a whole full size computer with a special, very expensive encoding/decoding card to use the system. It is quite mind-blowing looking at the small devices used today! But we showed that it worked—and it led to the beginning of a digital music adventure for the project coordinator, The State and University Library.
In the evaluation, a number of challenges were identified. One was the handling of copyright—which still is a major problem!
The Libraries Programme was formulated in a time characterized by political optimism and the onset of a technological revolution in Europe. The initial period saw a battle between the TCP/IP protocol, which was easy to implement, and the more complicated OSI stack as the basis of the services running over the network. The TCP/IP protocol was backed by all the governmental institutions in USA and a number of university networks in Europe (among these Denmark). The telecommunications companies in Europe pushed and lobbied for the more secure, but also more complicated and more expensive OSI stack. The result was a policy for the use of network which made the life difficult for the developers. It was a battle between technologies over speed of implementation and security, and guarantee of a given service level.
The battle is illustrated by the two examples, SOCKER and JUKEBOX. In SOCKER we wanted to use the TCP/IP protocol as we as the university computing centre were users of TCP/IP and believed it provided a future model for accessing research library catalogues. On the downside, we had to implement a session-oriented protocol over a stateless TCP/IP network. In the JUKE-BOX project network services posed a different problem as we needed a minimum guaranteed bandwidth in order to transmit high quality music. The TCP/IP based bandwidth offered across Europe at that time was limited and the OSI based network was quite expensive. To support JUKE-BOX we needed both to compress the signal—and to use ISDN-lines to guarantee the quality.
Thus the situation was not black and white: on one side we saw TCP/IP as the winning approach but on the other hand, we wanted the quality and guaranteed bandwidth provided by the OSI stack.
The point I want to make in this paper is, that despite limitation as well in terms of standards, methodologies and capacity, the Commission was ready to allow experiments on the edge of the frontiers both in terms network development and in terms of technology. The choices seem obvious today, but at the time they were not. Had the Commission insisted on projects following their policies and projections, many initiatives would have been outdated even before they ended. Errors were made, activities became redundant and had to be re-focused, but there was room for all this—and the researchers and their institutions learned a lot and became better prepared for the next wave of discoveries.
I am happy, that risks were taken and the Commission backed the projects—also in their reformulation and adjustment to new, radically different situations. I am happy, that the Commission have heard the critical voices and quickly incorporated the new reality into their strategies for the subsequent Framework Programmes. I think the library sector would be in a quite different state if they had not had the opportunity to be part of these early experiments.
In writing this paper I have tried to check references and revisit sources of information quoted. What I discovered was that most sources of the data behind the graphs are no longer accessible and many of the references in older papers and websites have disappeared. This could be the topic of a completely different story; a story which I know interests Pat—namely the story about digital preservation—not only of the finalized artifacts but also of the components/numbers, which were needed to create them.
1See e.g. the introduction at CORDIS: http://cordis.europa.eu/libraries/en/intro.html.
Biblioteksstyrelsen. (1996). State-of-the-art of information technologies and Danish libraries. Udgivet af Biblioteksstyrelsen, maj 1996. Note 2017: Has not been recovered; however the numbers are supported by the report State-of-the-art of Information Technologies in Nordic Libraries, http://cordis.europa.eu/libraries/en/nordinfo.html.
Christensen-Dalsgaard, B.L. (2005). Internettet og biblioteker - The Internet and the libraries (in Danish). In E. Kolding Nielsen, N.C. Nielsen, & S.B. Larsen (Eds.), Kommunikation erstatter transport - Den digitale revolution i danske forskningsbiblioteker 1980–2005. (Danish humanist texts and studies, nr. 30) (pp. 317–341). Copenhagen: Museum Tusculanums Forlag.
Lee, C. (2000). Where have all the Gophers gone? Why the Web beat Gopher in the battle for protocol mind share. Retrieved January 15, 2016, from https://ils.unc.edu/callee/gopherpaper.htm.
NITRD. (1995). Definition of "Internet". The Networking and Information Technology Research and Development (NITRD) Program. Retrieved November 7, 2016, from https://www.nitrd.gov/fnc/Internet_res.aspx.
Telematics for Libraries. (1996). Europagate workshop/SR concertation meeting. Retrieved November 7, 2016, from http://cordis.europa.eu/libraries/en/sr-conc.html.
Turner, F., Tallim, P., & Zeemann, J. (1992). OSI for libraries: Standards to services (1992). UDT Series on data communication technologies and standards for libraries. Retrieved November 7, 2016, from http://www.ifla.org/archive/udt/reports/rep3/rep3.htm.
Vitiello, G. (2014). International librarianship in Europe (1990–2000). Paper presented at the IFLA conference, Lyon. Retrieved November 7, 2016, from http://library.ifla.org/946/1/071-vitiello-en.pdf.