<commit_before> <chapter id="introduction"><title>Introduction</title>

  <para>
   &yaz; is a C/C++ library for information retrieval applications
   using the Z39.50/SRU protocols for information retrieval.
  </para>
  
  <para>
   Properties of &yaz;:
   <itemizedlist>
    <listitem><para>
      Complete 
      <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
      Amendments and Z39.50-2002 revision is supported.
     </para></listitem>
    <listitem><para>
      Supports 
      <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
      version 1.2 (over HTTP and HTTPS).
     </para></listitem>
    <listitem><para>
      Includes BER encoders/decoders for the 
      <ulink url="&url.ill;">ISO ILL</ulink>
      protocol.
     </para></listitem>
    <listitem><para>
      Supports 
      <ulink url="&url.solr;">SOLR</ulink> Web Service (client side only)
     </para></listitem>
    <listitem><para>
      Supports the following transports: BER over TCP/IP
      (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
      BER over unix local socket, and 
      <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
     </para></listitem>
    <listitem><para>
      Secure Socket Layer support using
      <ulink url="&url.gnutls;">GNU TLS</ulink> or
      <ulink url="&url.openssl;">OpenSSL</ulink>.
      If enabled, &yaz; uses HTTPS transport (for SOAP) or
      "Secure BER" (for Z39.50).
     </para></listitem>
    <listitem><para>
      Offers 
      <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
      Z39.50, SRU and SOLR Web Service.
     </para></listitem>
    <listitem><para>
      The &yaz; library offers a set of useful utilities
      related to the protocols, such as MARC (ISO2709) parser,
      CCL (ISO8777) parser, 
      <ulink url="&url.cql;">CQL</ulink>
      parser, memory management routines, character set conversion.
     </para></listitem>
    <listitem><para>
      Portable code. &yaz; compiles out-of-the box on most Unixes and
      on Windows using Microsoft Visual C++.
     </para></listitem>
    <listitem><para>
      Fast operation. The C based BER encoders/decoders as well
      as the server component of &yaz; is very fast.
     </para></listitem>
    <listitem><para>
      Liberal license that allows for commercial use of &yaz;.
     </para></listitem>
   </itemizedlist>
  </para>

  <sect1 id="introduction.reading"><title>Reading this Manual</title>
   <para>Most implementors only need to read a fraction of the
    material in thie manual, so a quick walkthrough of the chapters
    is in order.
   </para>
   <itemizedlist>
    <listitem>
     <para>
      <xref linkend="installation"/> contains installation 
      instructions for &yaz;. You don't need reading this
      if you expect to download &yaz; binaries.
      However, the chapter contains information about how
      to make <emphasis>your</emphasis> application link
      with &yaz;.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
      This is definitely worth a read if you wish to develop a Z39.50/SRU
      client.
     </para>
    </listitem>
    
    <listitem>
     <para>
      <xref linkend="server"/> describes the generic frontend server
      and explains how to develop server Z39.50/SRU applications for &yaz;.
      Obviously worth reading if you're to develop a server.
    </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
      client. If you're developer and wish to test your server
      or a server from another party, you might find this chapter
      useful.
    </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="asn"/> documents the most commonly used Z39.50
      C data structures offered by the &yaz; API. Client
      developers using ZOOM and non-Z39.50 implementors may skip this.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="soap"/> describes how SRU and SOAP is used 
      in &yaz;. Only if you're developing SRU applications
      this section is a must.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="tools"/> contains sections for the various
      tools offered by &yaz;. Scan through the material quickly
      and see what's relevant to you! SRU implementors
      might find the <link linkend="cql">CQL</link> section
      particularly useful.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="odr"/> goes through the details of the
      ODR module which is the work horse that encodes and decodes
      BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
      need reading this.
      Most other Z39.50 implementors only need to read the first two
      sections (<xref linkend="odr.introduction"/> and
      <xref linkend="odr.use"/>).
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="comstack"/> describes the network layer module
      COMSTACK. Implementors using ZOOM or the generic frontend server
      may skip this. Others, presumably, handling client/server
     communication on their own should read this.
     </para>
    </listitem>

   </itemizedlist>
  </sect1>
  <sect1 id="introduction.api"><title>The API</title>
   
   <para>
    The <ulink url="&url.yaz;">&yaz;</ulink>
    toolkit offers several different levels of access to the
    <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
    <ulink url="&url.ill;">ILL</ulink> and
    <ulink url="&url.sru;">SRU</ulink>
    protocols.
    The level that you need to use depends on your requirements, and
    the role (server or client) that you want to implement.
    If you're developing a client application you should consider the
    <link linkend="zoom">ZOOM</link> API.
    It is, by far, the easiest way to develop clients in C.
    Server implementers should consider the 
    <link linkend="server">generic frontend server</link>.
    None of those high-level APIs support the whole protocol, but
    they do include most facilities used in existing Z39.50 applications.
   </para>
   <para>
    If you're using 'exotic' functionality (meaning anything not included in
    the high-level APIs), developing non-standard extensions to Z39.50 or
    you're going to develop an ILL application you'll have to learn the lower
    level APIs of &yaz;.
   </para>
   <para>
    The YAZ toolkit modules is shown in figure <xref linkend="yaz.layer"/>.
   </para>
   <figure id="yaz.layer">
    <title>YAZ layers</title>
    <mediaobject>
     <imageobject>
      <imagedata fileref="apilayer.png" format="PNG"/>
     </imageobject>
     <imageobject>
      <imagedata fileref="apilayer.eps" format="EPS"/>
     </imageobject>
    </mediaobject>
   </figure>
   <para>
    There are four layers.
    <itemizedlist>
     <listitem>
      <para>A client or server application (or both).
       This layer includes ZOOM and the generic frontend server.
      </para>
     </listitem>
     <listitem>
      <para>
       The second layer provides a C represenation of the 
       protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
       SRU.
      </para>
     </listitem>
     <listitem>
      <para>
       The third layer encodes and decodes protocol data units to
       simple packages (buffer with certain length). The &odr; module
       encodes and decodes BER whereas the HTTP modules encodes and
       decodes HTTP ruquests/responses.
      </para>
     </listitem>
     <listitem>
      <para>
       The lowest layer is &comstack; which exchanges the encoded packages
       with a peer process over a network.
      </para>
     </listitem>
    </itemizedlist>
   </para>
   <para>
    The &asn; module represents the ASN.1 definition of
    the Z39.50 protocol. It establishes a set of type and
    structure definitions, with one structure for each of the top-level
    PDUs, and one structure or type for each of the contained ASN.1 types.
    For primitive types, or other types that are defined by the ASN.1
    standard itself (such as the EXTERNAL type), the C representation is
    provided by the &odr; (Open Data Representation) subsystem.
  </para>
   <para>
     &odr; is a basic mechanism for representing an
    ASN.1 type in the C programming language, and for implementing BER
    encoders and decoders for values of that type. The types defined in
    the &asn; module generally have the prefix <literal>Z_</literal>, and
    a suffix corresponding to the name of the type in the ASN.1
    specification of the protocol (generally Z39.50-1995). In the case of
    base types (those originating in the ASN.1 standard itself), the prefix
    <literal>Odr_</literal> is sometimes seen. Either way, look for
    the actual definition in either <filename>z-core.h</filename> (for the types
    from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
    types).
    The &asn; library also provides functions (which are, in turn,
    defined using &odr; primitives) for encoding and decoding data values.
    Their general form is
    
    <funcsynopsis>
     <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
      <paramdef>ODR <parameter>o</parameter></paramdef>
      <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
      <paramdef>int <parameter>optional</parameter></paramdef>
      <paramdef>const char *<parameter>name</parameter></paramdef>
     </funcprototype>
    </funcsynopsis>
    (note the lower-case &quot;z&quot; in the function name)
   </para>
   
   <note>
    <para>
     If you are using the premade definitions of the &asn; module, and you
     are not adding new protocol of your own, the only parts of &odr; that you
     need to worry about are documented in
     <xref linkend="odr.use"/>.
    </para>
   </note>
   
  <para>
    When you have created a BER-encoded buffer, you can use the &comstack;
    subsystem to transmit (or receive) data over the network. The &comstack;
    module provides simple functions for establishing a connection
    (passively or actively, depending on the role of your application),
    and for exchanging BER-encoded PDUs over that connection. When you
    create a connection endpoint, you need to specify what transport to
    use (TCP/IP, SSL or UNIX sockets).
    For the remainder of the connection's lifetime, you don't have
    to worry about the underlying transport protocol at all - the &comstack;
    will ensure that the correct mechanism is used.
   </para>
   <para>
    We call the combined interfaces to &odr;, &asn;, and &comstack; the service
    level API. It's the API that most closely models the Z39.50
   service/protocol definition, and it provides unlimited access to all
    fields and facilities of the protocol definitions.
   </para>
   <para>
    The reason that the &yaz; service-level API is a conglomerate of the
    APIs from three different submodules is twofold. First, we wanted to allow
    the user a choice of different options for each major task. For instance,
    if you don't like the protocol API provided by &odr;/&asn;, you
    can use SNACC or BERUtils instead, and still have the benefits of the
    transparent transport approach of the &comstack; module. Secondly,
    we realize that you may have to fit the toolkit into an existing
    event-processing structure, in a way that is incompatible with
    the &comstack; interface or some other part of &yaz;.
   </para>
  </sect1>
 </chapter>

 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document:"yaz.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->
<commit_msg>Add SOLR version. mentioning SOLR togetheer with Z39.50 and SRU.<commit_after> <chapter id="introduction"><title>Introduction</title>

  <para>
   &yaz; is a C/C++ library for information retrieval applications
   using the Z39.50/SRU/SOLR protocols for information retrieval.
  </para>
  
  <para>
   Properties of &yaz;:
   <itemizedlist>
    <listitem><para>
      Complete 
      <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
      Amendments and Z39.50-2002 revision is supported.
     </para></listitem>
    <listitem><para>
      Supports 
      <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
      version 1.2 (over HTTP and HTTPS).
     </para></listitem>
    <listitem><para>
      Includes BER encoders/decoders for the 
      <ulink url="&url.ill;">ISO ILL</ulink>
      protocol.
     </para></listitem>
    <listitem><para>
      Supports 
      <ulink url="&url.solr;">SOLR</ulink> Web Service version 1.4.x (client side only)
     </para></listitem>
    <listitem><para>
      Supports the following transports: BER over TCP/IP
      (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
      BER over unix local socket, and 
      <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
     </para></listitem>
    <listitem><para>
      Secure Socket Layer support using
      <ulink url="&url.gnutls;">GNU TLS</ulink> or
      <ulink url="&url.openssl;">OpenSSL</ulink>.
      If enabled, &yaz; uses HTTPS transport (for SOAP) or
      "Secure BER" (for Z39.50).
     </para></listitem>
    <listitem><para>
      Offers 
      <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
      Z39.50, SRU and SOLR Web Service.
     </para></listitem>
    <listitem><para>
      The &yaz; library offers a set of useful utilities
      related to the protocols, such as MARC (ISO2709) parser,
      CCL (ISO8777) parser, 
      <ulink url="&url.cql;">CQL</ulink>
      parser, memory management routines, character set conversion.
     </para></listitem>
    <listitem><para>
      Portable code. &yaz; compiles out-of-the box on most Unixes and
      on Windows using Microsoft Visual C++.
     </para></listitem>
    <listitem><para>
      Fast operation. The C based BER encoders/decoders as well
      as the server component of &yaz; is very fast.
     </para></listitem>
    <listitem><para>
      Liberal license that allows for commercial use of &yaz;.
     </para></listitem>
   </itemizedlist>
  </para>

  <sect1 id="introduction.reading"><title>Reading this Manual</title>
   <para>Most implementors only need to read a fraction of the
    material in thie manual, so a quick walkthrough of the chapters
    is in order.
   </para>
   <itemizedlist>
    <listitem>
     <para>
      <xref linkend="installation"/> contains installation 
      instructions for &yaz;. You don't need reading this
      if you expect to download &yaz; binaries.
      However, the chapter contains information about how
      to make <emphasis>your</emphasis> application link
      with &yaz;.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
      This is definitely worth a read if you wish to develop a Z39.50/SRU
      client.
     </para>
    </listitem>
    
    <listitem>
     <para>
      <xref linkend="server"/> describes the generic frontend server
      and explains how to develop server Z39.50/SRU applications for &yaz;.
      Obviously worth reading if you're to develop a server.
    </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
      client. If you're developer and wish to test your server
      or a server from another party, you might find this chapter
      useful.
    </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="asn"/> documents the most commonly used Z39.50
      C data structures offered by the &yaz; API. Client
      developers using ZOOM and non-Z39.50 implementors may skip this.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="soap"/> describes how SRU and SOAP is used 
      in &yaz;. Only if you're developing SRU applications
      this section is a must.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="tools"/> contains sections for the various
      tools offered by &yaz;. Scan through the material quickly
      and see what's relevant to you! SRU implementors
      might find the <link linkend="cql">CQL</link> section
      particularly useful.
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="odr"/> goes through the details of the
      ODR module which is the work horse that encodes and decodes
      BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
      need reading this.
      Most other Z39.50 implementors only need to read the first two
      sections (<xref linkend="odr.introduction"/> and
      <xref linkend="odr.use"/>).
     </para>
    </listitem>

    <listitem>
     <para>
      <xref linkend="comstack"/> describes the network layer module
      COMSTACK. Implementors using ZOOM or the generic frontend server
      may skip this. Others, presumably, handling client/server
     communication on their own should read this.
     </para>
    </listitem>

   </itemizedlist>
  </sect1>
  <sect1 id="introduction.api"><title>The API</title>
   
   <para>
    The <ulink url="&url.yaz;">&yaz;</ulink>
    toolkit offers several different levels of access to the
    <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
    <ulink url="&url.ill;">ILL</ulink> and
    <ulink url="&url.sru;">SRU</ulink>
    protocols.
    The level that you need to use depends on your requirements, and
    the role (server or client) that you want to implement.
    If you're developing a client application you should consider the
    <link linkend="zoom">ZOOM</link> API.
    It is, by far, the easiest way to develop clients in C.
    Server implementers should consider the 
    <link linkend="server">generic frontend server</link>.
    None of those high-level APIs support the whole protocol, but
    they do include most facilities used in existing Z39.50 applications.
   </para>
   <para>
    If you're using 'exotic' functionality (meaning anything not included in
    the high-level APIs), developing non-standard extensions to Z39.50 or
    you're going to develop an ILL application you'll have to learn the lower
    level APIs of &yaz;.
   </para>
   <para>
    The YAZ toolkit modules is shown in figure <xref linkend="yaz.layer"/>.
   </para>
   <figure id="yaz.layer">
    <title>YAZ layers</title>
    <mediaobject>
     <imageobject>
      <imagedata fileref="apilayer.png" format="PNG"/>
     </imageobject>
     <imageobject>
      <imagedata fileref="apilayer.eps" format="EPS"/>
     </imageobject>
    </mediaobject>
   </figure>
   <para>
    There are four layers.
    <itemizedlist>
     <listitem>
      <para>A client or server application (or both).
       This layer includes ZOOM and the generic frontend server.
      </para>
     </listitem>
     <listitem>
      <para>
       The second layer provides a C represenation of the 
       protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
       SRU.
      </para>
     </listitem>
     <listitem>
      <para>
       The third layer encodes and decodes protocol data units to
       simple packages (buffer with certain length). The &odr; module
       encodes and decodes BER whereas the HTTP modules encodes and
       decodes HTTP ruquests/responses.
      </para>
     </listitem>
     <listitem>
      <para>
       The lowest layer is &comstack; which exchanges the encoded packages
       with a peer process over a network.
      </para>
     </listitem>
    </itemizedlist>
   </para>
   <para>
    The &asn; module represents the ASN.1 definition of
    the Z39.50 protocol. It establishes a set of type and
    structure definitions, with one structure for each of the top-level
    PDUs, and one structure or type for each of the contained ASN.1 types.
    For primitive types, or other types that are defined by the ASN.1
    standard itself (such as the EXTERNAL type), the C representation is
    provided by the &odr; (Open Data Representation) subsystem.
  </para>
   <para>
     &odr; is a basic mechanism for representing an
    ASN.1 type in the C programming language, and for implementing BER
    encoders and decoders for values of that type. The types defined in
    the &asn; module generally have the prefix <literal>Z_</literal>, and
    a suffix corresponding to the name of the type in the ASN.1
    specification of the protocol (generally Z39.50-1995). In the case of
    base types (those originating in the ASN.1 standard itself), the prefix
    <literal>Odr_</literal> is sometimes seen. Either way, look for
    the actual definition in either <filename>z-core.h</filename> (for the types
    from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
    types).
    The &asn; library also provides functions (which are, in turn,
    defined using &odr; primitives) for encoding and decoding data values.
    Their general form is
    
    <funcsynopsis>
     <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
      <paramdef>ODR <parameter>o</parameter></paramdef>
      <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
      <paramdef>int <parameter>optional</parameter></paramdef>
      <paramdef>const char *<parameter>name</parameter></paramdef>
     </funcprototype>
    </funcsynopsis>
    (note the lower-case &quot;z&quot; in the function name)
   </para>
   
   <note>
    <para>
     If you are using the premade definitions of the &asn; module, and you
     are not adding new protocol of your own, the only parts of &odr; that you
     need to worry about are documented in
     <xref linkend="odr.use"/>.
    </para>
   </note>
   
  <para>
    When you have created a BER-encoded buffer, you can use the &comstack;
    subsystem to transmit (or receive) data over the network. The &comstack;
    module provides simple functions for establishing a connection
    (passively or actively, depending on the role of your application),
    and for exchanging BER-encoded PDUs over that connection. When you
    create a connection endpoint, you need to specify what transport to
    use (TCP/IP, SSL or UNIX sockets).
    For the remainder of the connection's lifetime, you don't have
    to worry about the underlying transport protocol at all - the &comstack;
    will ensure that the correct mechanism is used.
   </para>
   <para>
    We call the combined interfaces to &odr;, &asn;, and &comstack; the service
    level API. It's the API that most closely models the Z39.50
   service/protocol definition, and it provides unlimited access to all
    fields and facilities of the protocol definitions.
   </para>
   <para>
    The reason that the &yaz; service-level API is a conglomerate of the
    APIs from three different submodules is twofold. First, we wanted to allow
    the user a choice of different options for each major task. For instance,
    if you don't like the protocol API provided by &odr;/&asn;, you
    can use SNACC or BERUtils instead, and still have the benefits of the
    transparent transport approach of the &comstack; module. Secondly,
    we realize that you may have to fit the toolkit into an existing
    event-processing structure, in a way that is incompatible with
    the &comstack; interface or some other part of &yaz;.
   </para>
  </sect1>
 </chapter>

 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document:"yaz.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->

{'fraction_non_alphanumeric': 0.09794893260778569, 'fraction_numerical': 0.0107994976977815, 'mean_word_length': 2.869614512471655, 'pattern_counts': {'":': 0, '<': 487, '<?xml version=': 0, '>': 487, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7516685', 'n_tokens_mistral': 7437, 'n_tokens_neox': 7077, 'n_words': 2725}