<commit_before><?xml version="1.0" encoding="UTF-8"?>

<book xmlns:ci="http://apache.org/cocoon/include/1.0">
    <bookinfo>
        <graphic fileref="logo.jpg"/>

        <title>Open Source Native XML Database</title>
        <author>
            <firstname><NAME>.</firstname>
            <surname>Meier</surname>
            <affiliation>
                <address format="linespecific">
                    <email><EMAIL></email>
                </address>
            </affiliation>
        </author>
    </bookinfo>
    <ci:include src="sidebar.xml"/>
    <chapter>
        <title>XQuery</title>
        <section>
            <title>Current Status</title>

            <para>eXist implements the current XQuery working draft as of
                November, 2003 with the exception of the XML Schema related
                features. Currently, eXist passes most of the official XQuery 
                use cases. According to the Bumblebee XQuery test harness, 92% of all use
                cases are passed (excluding the "strong" use cases, which are
                currently beyond eXist's scope).</para>

            <para>The XQuery engine is a complete rewrite and replaces the old
                XPath engine. Older code based on XPath should be compatible
                with the XQuery engine, though some aspects (e.g. the rules for
                automatic type conversions) have changed. The query engine
                doesn't make a difference between XPath and XQuery syntax
                internally. All interfaces should work as before, i.e. you can
                pass an XQuery expression wherever you used to pass an XPath
                query.</para>

            <para>Basically, eXist implements all features described in the
            XQuery specification, except the following:</para>

            <itemizedlist>
                <listitem>Unrecognized keywords: 
                <command>typeswitch</command>, <command>treat as</command>,
                <command>castable as</command>.
                </listitem>

				<listitem>Item types
                <listitem>Schema related features: <command>validate</command>,
                <command>import schema</command>, <command>declare validation</command>.
                </listitem>

                <listitem>Since schema support is missing, the value of a node
                    is always treated as xs:untypedAtomic.</listitem>

				<listitem>For the same reason, one cannot specify a type name
					in an element or attribute test. The test:
					element(test) works, but: element(test, xs:integer)
					results in a syntax error.</listitem>

                <listitem>The atomic types xs:gYear, xs:gMonth etc. are not
                    recognized.</listitem>

            </itemizedlist>

            <para>Another limitation applies to <emphasis>constructed document
            fragments</emphasis>: it is currently not possible to query document
            fragments that have been constructed within the XQuery, i.e. a query
            like the following will fail:</para>

            <synopsis>let $x := &lt;test>Test&lt;/test> return $x/test</synopsis>

            <para>At this stage of development, the constructed in-memory DOM
                tree can not be queried in the same way as the persistent
                database DOM (the in-memory DOM is not yet complete and will be
                extended in a future release). You have to store the fragment
                into the database first. You can use the provided <ulink
                        url="#xmldb">xmldb extension functions</ulink> for
                that.</para>

            <para>A list of all the functions currently known to the system can
                be viewed <a href="xquery/functions.xq">here</a>. This list is
                dynamically generated by an XQuery script, so the database needs
                to be running. The function description is directly extracted
                from the signature provided by each class implementing the
                <classname>Function</classname> interface. </para>
            

        </section>
        <section id="ext">
            <title>Extensions</title>

            <para>eXist offers a number of additional functions and two
                additional operators. These extensions can be grouped into three
                main categories:</para>

            <orderedlist>
                <listitem>
                    Functions to specify the input document set for an expression.
                </listitem>
                <listitem>
                    Extensions for full-text searching.
                </listitem>
                <listitem>
                    Functions for manipulating database contents.
                </listitem>
                <listitem>
                    Utility functions.
                </listitem>
                <listitem>
                    XSL transformation functions.
                </listitem>
                <listitem>
                    HTTP related functions.
                </listitem>
            </orderedlist>

            <para>We start with the additional functions for specifying
            the input document set.</para>

            <section>
                <title>Specifying the Input Document Set</title>
                <para>A database may contain an unlimited set of collections and documents.
                If you use the XML:DB API to query the db, only the documents in the current 
                XML:DB collection are processed by default. However, there are four additional 
                functions to change this behavior: <command>doc()</command>, 
                <command moreinfo="none">document()</command>, 
                <command moreinfo="none">collection()</command> and <command>xcollection()</command>.</para>

                <para>The <command>collection()</command> and <command>doc()</command> 
            functions are standard XQuery/XPath functions.
            <command>xcollection()</command> and <command>document()</command> 
            represent eXist-specific extensions.</para>

                <para>eXist interprets the arguments to <command>collection</command> and <command>doc</command>
            as absolute or relative paths, leading to some collection or
            document within the database. Relative paths are resolved relative to the base
            URI property of the static XQuery context, which usually points to a database
            collection.</para>

                <note>
                    <para>Currently, the eXist-specific, document related functions are in the XQuery/XPath
                core library functions namespace (i.e. http://www.w3.org/2003/05/xpath-functions).
                This is definitely wrong, but previous versions of eXist did not use namespaces for
                functions, so older eXist-based applications would complain. However, in the future, the eXist-specific 
                functions will move to their own namespace.</para>
                </note>

                <para>While <command>doc()</command> is restricted to a single document-URI argument, 
            <command>document</command> accepts multiple
            document paths to be included into the input node set. Second, calling 
            <command>document</command> without an argument includes <emphasis>every</emphasis>
            document node in the current database instance. Some examples:</para>

                <synopsis format="linespecific">document()//SPEAKER</synopsis>
                <synopsis format="linespecific">document('/db/test/abc.xml', '/db/test/def.xml')//title</synopsis>

                <para>The <command moreinfo="none">collection()</command> function specifies the 
            collection whose documents are to be included into query evaluation. By default, 
            documents found in sub-collections of the specified collection are included. This 
            means that for three collections, <filename
                            moreinfo="none">/db/test</filename>, 
            <filename moreinfo="none">/db/test/abc</filename> and 
            <filename moreinfo="none">/db/test/def</filename>, 
            <command moreinfo="none">collection('/db/test')</command> will include all resources 
            found in <filename moreinfo="none">/db/test</filename>, 
            <filename moreinfo="none">/db/test/abc</filename> and 
            <filename moreinfo="none">/db/test/def</filename>. You may use the 
            <command moreinfo="none">xcollection()</command> function to change this behavior, 
            e.g.</para>
                <synopsis format="linespecific">xcollection('/db/test')//title</synopsis>
                <para>will only include resources found in <filename
                            moreinfo="none">/db/test</filename>, but not in <filename
                            moreinfo="none">/db/test/abc</filename> or <filename
                            moreinfo="none">/db/test/def</filename>.</para>
                <note>
                    <para>Beginning with version 1.0beta1, the collection and document functions are implemented
                as ordinary functions. They may occur at any position where a function call is allowed.</para>
                </note>
            </section>
            <section>
                <title>Querying Text</title>
                <para>The standard XPath/XQuery function library contains most
                of the well-known string manipulation functions provided by
                every programming languages. However, these functions are rather
                insufficient when searching for a set of keywords or a phrase inside a
                larger portion of text or mixed content.</para>

                <para>This is a weak point if you have to work with document-centric 
                (as opposed to data-centric) documents. For many types of documents, the 
                provided standard
                functions will not yield satisfactory results. For example, you might remember 
                to have read something about "XML" and "databases" in some chapter of a book, 
                but you may not be sure where exactly it was. Using standard XPath, you 
                could try a query like:</para>

                <synopsis format="linespecific">//chapter[contains(., 'XML') and contains(., 'databases')]</synopsis>

                <para>Query execution will probably be quite slow, because the XPath engine has to scan 
                the entire character content of all chapter nodes and their descendants. Still you can't be sure 
                to find all matches - for example, "databases" might have been written with a capital letter 
                at the start of the sentence.</para>
                <para>eXist offers additional operators and extension functions for efficient, index-based 
                access to the full-text content of nodes. With eXist, you could formulate 
                your query as follows:</para>
                <synopsis format="linespecific">//chapter[near(., 'XML database?', 50)]</synopsis>
                <para>This query will return all chapters containing both keywords in the correct 
                order and with less than 50 words between them. Additionally, the wildcard in "database?" 
                will match the singular as well as the plural form and the search is done case-insensitive.</para>
                <para>Since the query is index-based, it will usually be a magnitude faster than the standard 
                XPath query above. Let's walk through each of eXist's extensions:</para>
                <section>
                    <title>Operators</title>
                    <para>In cases where the order and distance of search terms is not important, eXist offers 
                    two additional operators for simple keyword queries: 
                    <command moreinfo="none">&amp;=</command> and <command
                                moreinfo="none">|=</command>. 
                    </para>

                    <variablelist>
                        <varlistentry>
                            <term>node-set &amp;= 'string of keywords'</term>
                            <listitem>
                                <para>selects context nodes containing all of the keywords in the right-hand 
                                argument in any order. The default tokenizer is used to split the right-hand 
                                argument into single tokens, i.e. any punctuation or white space is skipped. 
                                Wildcards are allowed. Keywords are compared case-insensitive.</para>
                            </listitem>
                        </varlistentry>
                        <varlistentry>
                            <term>node-set |= 'string of keywords'</term>
                            <listitem>
                                <para>as above, but selects context nodes containing <emphasis>any</emphasis> 
                                of the keywords in the right-hand argument.</para>
                            </listitem>
                        </varlistentry>
                    </variablelist>

                    <note>
                        <para>The strings are split into tokens using the
                        default tokenizer function. The current implementation
                        will work well for all european languages. For
                        non-european languages, things are a bit more
                        complicated. In these cases, eXist uses the predefined
                        unicode codepoints to decide where the string will be
                        split.</para>
                    </note>

                    <para>Both operators accept simple wildcards in the keyword string. A 
                    <command moreinfo="none">?</command> matches zero or one character, 
                    <command moreinfo="none">*</command> matches zero or more characters. 
                    A character range <command moreinfo="none">[abc]</command> matches any 
                    of the characters in the range. You may use a backslash to escape wildcard 
                    characters.</para>

                    <para>To match more complex patterns, full regular expression syntax is 
                    supported through additional functions, which will be described below.</para>

                    <para>Please note that there is an important semantic difference between the two expressions:</para>
                    <synopsis format="linespecific">document()//SPEECH[LINE &amp;= "cursed spite"]</synopsis>
                    <para>and</para>
                    <synopsis format="linespecific">document()//SPEECH[LINE &amp;= "cursed" and LINE &amp;="spite"]</synopsis>

                    <para>The first expression selects all distinct LINE nodes that contain both terms. The second 
                    one selects all context nodes (SPEECH) that have line children containing either or 
                    both of the terms. The second expression is supposed to yield more results than the 
                    first one. To make both expressions select nearly the same nodes, you would have to 
                    change the first one into</para>
                    <synopsis format="linespecific">document()//SPEECH[. &amp;= "cursed spite"]</synopsis>
                    <para>(but this will also include, say SPEAKER or STAGEDIR children of the SPEECH parent)</para>
                </section>
                <section>
                    <title>near()</title>
                    <para>As shown above, the <command
                                moreinfo="none">near()</command> function behaves quite 
                    similar to the <command moreinfo="none">&amp;=</command> operator, but it additionally pays 
                    attention to the order of search terms and their distance in the source document. The syntax is:</para>
                    <synopsis format="linespecific">near(node-list, 'string of keywords' [, max-distance])</synopsis>
                    <para>The functions measures the distance between two search terms by counting the words in 
                    between them. A maximum distance of 1 is assumed by default, which means that the search terms 
                    should occur next to each other. Other values may be specified in the optional third argument, 
                    e.g.:</para>
                    <synopsis format="linespecific">document()//SPEECH[near(., 'love marriage', 25)</synopsis>
                    <para>will return any SPEECH elements containing the words "love" and "marriage" and less 
                    than 25 words between them.</para>
                    <para>Like <command moreinfo="none">&amp;=</command>, <command
                                moreinfo="none">near()</command> 
                    accepts wildcards in the keyword string. As well, punctuation and whitespace will be skipped 
                    according to the default tokenization rules.</para>
                </section>
                <section>
                    <title>match-all()/match-any()</title>
                    <para>These functions are variations of the <command
                                moreinfo="none">&amp;=</command> and 
                    <command moreinfo="none">|=</command> operators which interpret their arguments as 
                    regular expressions. <emphasis>But</emphasis>: contrary to the <command>matches</command>
                    function in the XQuery core library,
                    <command>match-all</command> and
                    <command>match-any</command> try to match the regular
                    expression argument against the keywords contained in the
                    full-text index, not against the entire text. Assume a document containing the
                    following paragraph:</para>

                    <synopsis>&lt;para&gt;Peter lives in Frankfurt&lt;/para&gt;</synopsis>

                    <para>Then the following expression:</para>

                    <synopsis>match-all(para, "li[vf]e.?", "frank.*")</synopsis>

                    <para>will match because the paragraph contains two keywords
                    matching the specified regular expression patterns.</para>

                    <para>
                        <command moreinfo="none">match-all()</command> corresponds to 
                    <command moreinfo="none">&amp;=</command> in that it
                    will select context nodes with keywords
                    matching all of the specified regular expressions. <command
                                moreinfo="none">match-any()</command> 
                    will select nodes with keywords matching <emphasis>any</emphasis> of the specified regexp.</para>

                    <para>Since tokenization doesn't work correctly with regular expression patterns, each 
                    keyword has to be specified as a separate argument, so the syntax looks like:</para>

                    <synopsis format="linespecific">match-all(node-set, 'regexp' [, 'regexp' ...])</synopsis>

                    <para>Please note that the functions will try to match the regexp against the entire 
                    keyword. For example, the expression</para>
                    <synopsis format="linespecific">//SPEECH[match-all(LINE, 'li[vf]e')]</synopsis>
                    <para>will match 'live', 'life', but not 'lives'.</para>
                    <para>eXist uses <ulink
                                url="http://jakarta.apache.org/oro/">Apache's ORO library </ulink> for regular expressions. A complete description of the supported regexp syntax can be found in the documentation of the library.</para>
                </section>
            </section>
            <section id="xmldb">
                <title>Manipulating Database Contents</title>

                <para>The xmldb extension functions can be used to create
                new database collections, or to store query output back
                into the database. For example, assume that someone has sent us
                a large file containing a number of RDF metadata records.
                Unfortunately, we don't want to store the metadata records
                as one, single file. Instead, our application expects each
                record to be in its own document. We thus have to transform
                the document and split it into smaller units.</para>

                <para>Using an XSLT stylesheet would be one alternative.
                However, it is rather memory-intensive. So why not use XQuery
                to do the job? The XQuery script below shows how to split the
                large RDF file into a number of smaller documents:</para>

                <example>
                    <title>Splitting a Document</title>
                    <programlisting>
<![CDATA[
xquery version "1.0";

declare namespace  xdb="http://exist-db.org/xquery/xmldb";
declare namespace util="http://exist-db.org/xquery/util";

xdb:register-database("org.exist.xmldb.DatabaseImpl", true()),
let $root := xdb:collection("xmldb:exist:///db", "admin", ""),
    $out := xdb:create-collection($root, "output")
for $rec in /rdf:RDF/* return
    xdb:store($out, concat(util:md5($rec/@rdf:about), ".xml"),
        <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
            {$rec}
        </rdf:RDF>
    )
]]>
                    </programlisting>
                </example>
                <para>The extension functions are based on the XML:DB API, so we
                first have to register a database driver with
                <command>xdb:register-database</command>. Then we retrieve the
                root collection with <command>xdb:collection</command>. The
                variable returned by <command>xdb:collection</command> holds
                the collection as a Java object. We can pass this variable to
                <command>xdb:create-collection</command> to create a new
                sub-collection, called "output".</para>

                <para>The <command>for</command> expression iterates through all
                child elements of the top RDF element. In each iteration, we use
                <command>xdb:store</command> to write out the current child
                node into a new document.</para>

                <para>Obviously, a unique document name is required for each
                document. In this example, the URI contained in the rdf:about attribute is
                unique, so we simply compute an MD5 key from it, append ".xml" and use it as
                document name.</para>
            </section>

            <section>
                <title>Utility Functions</title>

                <para>The function namespace http://exist-db.org/xquery/util
                contains a number of common utility functions. For example,
                the <command>util:md5</command> function used above is defined 
                here.</para>

                <para>We will not discuss all available functions. They 
                documented in the <a href="xquery/functions.xq">functions table</a>.
                However, there's one function which needs some further
                explanation:</para>

                <para>
                    <command>util:eval</command> is used to dynamically
                execute a constructed XQuery expression inside a running XQuery
                script. This is very handy sometimes, for example, in a
                web-based application, where queries are dynamically built from
                the request parameters the user has passed to the application.</para>

                <para>In this case, there would be one XQuery script, which is responsible
                for evaluating the user supplied parameters.
                This script uses the parameters from the HTTP request to create another
                XQuery expression. It then passes the constructed query string
                to <command>util:eval</command> to evaluate the expression,
                post-processes the returned results and displays them to the
                user. More information about how to write web applications with XQuery 
                can be found in the <ulink url="devguide.xml">Developer's Guide</ulink>.
                Below is a simple example to add or subtract two numbers:
                </para>

                <example>
                    <title>Adding/Subtracting Two Numbers</title>

                    <programlisting>
<![CDATA[xquery version "1.0";

declare namespace request="http://exist-db.org/xquery/request";
declare namespace util="http://exist-db.org/xquery/util";

declare function local:do-query() as element()
{
    let $n1 := request:request-parameter("n1", ""),
		$n2 := request:request-parameter("n2", ""),
		$op := request:request-parameter("op", "")
	return
		if($n1 = "" or $n2 = "") then
			<p>Please enter two operands.</p>
		else
			let $query := concat($n1, " ", $op, " ", $n2)
			return
				<p>{$query} = {util:eval($query)}</p>
};

<html>
	<body>
		<h1>Enter two numbers</h1>

		<form action="{request:request-uri()}" method="get">
			<table border="0" cellpadding="5">
			<tr>
				<td>First number:</td>
				<td><input name="n1" size="4"/></td>
			</tr>
			<tr>
				<td>Operator:</td>
				<td>
					<select name="op">
						<option name="+">+</option>
						<option name="-">-</option>
					</select>
				</td>
			</tr>
			<tr>
				<td>Second number:</td>
				<td><input name="n2" size="4"/></td>
			</tr>
			<tr>
				<td colspan="2"><input type="submit"/></td>
			</tr>
			</table>
		</form>

		{ local:do-query() }
	</body>
</html>]]>
                    </programlisting>
                </example>
            </section>

            <section>
                <title>XSL Transformations</title>

                <para>eXist has a function to directly apply an XSL stylesheet
                    to an XML fragment within the XQuery script. This can be
                    very convenient, for example, if your requirements are
                    simple and you don't want to install Cocoon with your web
                    application. The function signature is as follows:</para>

                <synopsis>transform:transform($input as node()?, $stylesheet as
                    xs:string, $parameters as node()?) as node()?</synopsis>

                <para>
                    <command>transform:transform</command> expects the node to
                    be transformed in the first argument $input. If $input is
                    the empty sequence, the function returns immediately.</para>

                <para>The XSL stylesheet will be read from the location
                    specified in $stylesheet. It should either be an absolute
                    URI or a relative path. If a relative path is specified, the
                    function will first try to find a file matching the path. If
                    this fails, <command>transform:transform</command> appends
                    the path to the current base URI property from the XQuery
                    context and retries.</para>

                <para>The stylesheet will be compiled into a template using the
                    standard Java APIs (javax.xml.transform). The template is
                    shared between all instances of the function and will only
                    be reloaded if modified since the last invocation.</para>

                <para>The $options parameter can be used to pass stylesheet
                    parameters to the XSL processor. Parameters are passed as an
                    XML fragment:</para>

                <synopsis><![CDATA[<parameters>
    <param name="param1" value="value1"/>
    <param name="param2" value="value2"/>
</parameters>]]></synopsis>

                <para>This will set the stylesheet parameter
                    <option>param1</option> to the string value
                    <option>value1</option>. In the stylesheet, the parameter
                    can then be referenced with</para>

                <synopsis><![CDATA[<xsl:param name="param1"/>]]></synopsis>
            </section>

            <section>
                <title>HTTP Related Functions</title>

                <para>The example above reads HTTP request parameters using the
                function <command>request:request-parameter</command>. Obviously, 
                functions in the http://exist-db.org/xquery/request namespace 
                are only usable if the query is executed through the
                XQueryGenerator or the XQueryServlet (please have a look at the 
                <ulink url="devguide.xml">Developer's Guide</ulink> for more 
                information).</para>

                <para>
                    <command>request:request-parameter</command> expects two
                arguments: the first argument denotes the name of the parameter,
                the second specifies a default value, which is returned if the
                parameter is not set. The function returns a sequence containing
                the values for the parameter.</para>

                <para>
                    <command>request:request-uri</command> returns the URI of
                the current request. You may call</para>

                <synopsis>request:encode-url(request:request-uri())</synopsis>

                <para>to encode the current session id into the URI.</para>

                <para>There are also two functions to read and set session
                attributes. Any XQuery value or Java object value can be stored 
                into a session attribute, for example, to cache query results.
                Please have a look at the provided <ulink
                            url="examples.xml">examples</ulink>.</para>

                <para>The function call
                <option>request:create-session()</option> will create a new HTTP
                session if none has been created before.</para>
            </section>
        </section>
        <section>
            <title>Java Binding</title>
            <para>eXist supports calls to arbitrary Java methods from within XQuery. The 
        	binding mechanism follows the short-cut technique introduced by 
        	<ulink url="http://saxon.sf.net">Saxon</ulink>. The class where the external function
        	will be found is identified by the namespace URI of the function call. The namespace
        	URI should start with the prefix <option>java:</option> followed by the fully qualified
        	class name of the class. For example, the following code snippet calls the static method
        	<methodname>sqrt</methodname> of class <classname>java.lang.Math</classname>:</para>
            <example>
                <title>Calling a Static Method</title>
                <programlisting><![CDATA[
declare namespace math="java:java.lang.Math";
math:sqrt(2)
				]]></programlisting>
            </example>
            <para>If the function name contains a hyphen, the letter following the hyphen is converted
        	to upper-case and the hyphen is removed (camel case), i.e. <code>to-string()</code> will call
        	the Java method <code>toString()</code>.</para>
            <para>If more than one method in the class matches the given name and parameter count,
        	eXist tries to select the method that best fits the passed parameter types at runtime.
        	The result of the method call can be assigned to an XQuery variable. If possible, it will
        	be mapped to the corresponding XML schema type. Otherwise, it's type is the built-in type
        	<option>object</option>.</para>
            <para>
                <emphasis>Java constructors</emphasis> are called using the function <option>new</option>. Again, a matching
        	constructor is selected by looking at the parameter count and types. The returned value is a
        	new Java object with the built-in type <option>object</option>.</para>
            <para>
                <emphasis>Instance methods</emphasis> are called by supplying a valid Java object as
        	first parameter. The Java object has to be an instance of the given class. For example, the
        	following snippet lists all files and directories in the current directory:</para>
            <example>
                <title>List Contents of the Current Directory</title>
                <programlisting><![CDATA[
declare namespace file="java:java.io.File";

<files>
    {
        for $f in file:list-files( file:new(".") )
        let $n := file:get-name($f)
        order by $n
        return
            if (file:is-directory($f)) then
                <directory name="{ $n }"/>
            else
                <file name="{ $n }" size="{ file:length($f) }"/>
    }
</files>       			
        		]]></programlisting>
            </example>
        </section>

        <section>
            <title>Writing your own Modules</title>

            <para>eXist supports XQuery's notion of library modules, i.e. you
                can declare an XQuery file as a module and later import it using
                the <command>import module</command> directive.</para>

            <para>The XQuery engine imports each module only once during
                compilation. The compiled module is then made available through
                the static XQuery context.</para>

            <para>A library module is basically just a collection of function
                definitions and global variable declarations. eXist knows two
                types of modules: external modules which are themselves written
                in XQuery, and internal modules implemented in Java. The
                standard XPath/XQuery functions and all extension functions
                described above are defined through internal modules.</para>

            <para>Users may provide additional Java modules. This is not too
                difficult, though the API may still change in the future. To
                register a module, eXist needs to know the namespace URI by
                which the module is identified and the list of functions it
                supplies. Basically, this works by passing a driver class to
                the XQuery engine. The driver class should implement interface
                <classname>org.exist.xpath.InternalModule</classname>.</para>

            <para>Class <classname>org.exist.xpath.AbstractInternalModule</classname>
                already provides an implementation skeleton. The class
                constructor expects an array of function definitions for all
                functions that should be registered. A function definition
                (class <classname>FunctionDef</classname>) has two properties:
                the static signature of the function (as an instance of
                <classname>FunctionSignature</classname>) and the Java Class
                that implements the function.</para>

            <para>A function is a class extending
                <classname>org.exist.xpath.Function</classname> or
                <classname>org.exist.xpath.BasicFunction</classname>. Functions
                without special requirements (e.g. overloading) should subclass BasicFunction. As
                an example, we define a simple function below:</para>

            <example>
                <title>A Basic Function</title>
                <programlisting>
<![CDATA[
public class EchoFunction extends BasicFunction {

public final static FunctionSignature signature =
new FunctionSignature(
	new QName("echo", ExampleModule.NAMESPACE_URI, ExampleModule.PREFIX),
	"A useless example function. It just echoes the input parameters.",
    new SequenceType[] {
        new SequenceType(Type.STRING, Cardinality.ZERO_OR_MORE)
    },
	new SequenceType(Type.STRING, Cardinality.ZERO_OR_MORE));

public EchoFunction(XQueryContext context) {
	super(context, signature);
}

public Sequence eval(Sequence[] args, Sequence contextSequence)
throws XPathException {
	// is argument the empty sequence?
	if (args[0].getLength() == 0)
		return Sequence.EMPTY_SEQUENCE;
	// iterate through the argument sequence and echo each item
	ValueSequence result = new ValueSequence();
	for (SequenceIterator i = args[0].iterate(); i.hasNext();) {
		String str = i.nextItem().getStringValue();
		result.add(new StringValue("echo: " + str));
	}
	return result;
}

}]]></programlisting>
            </example>

            <para>First, every function class has to provide a function signature. The
                function signature defines the QName by which the function is
                identified, a documentation string, the sequence types of
                all arguments and the sequence type of the returned value. In
                the example above, we accept a single argument of type xs:string
                and a cardinality of ZERO_OR_MORE. In other words, we accept any sequence
                of strings containing zero or more items.</para>

            <para>Next, the subclass overwrites the <command>eval</command>
                method, which has two arguments: the first contains the values
                of all arguments passed to the function, the second passes the
                current context sequence (which might be null).</para>

            <para>The argument values in the array <option>args</option> have
                already been checked to match the sequence types defined in the
                function signature. We thus don't have to recheck the length of
                the array: if more or less than one argument were passed to the
                function, an exception would have been thrown before eval gets
                called.</para>

            <para>In XQuery all values are passed as sequences. A sequence
                consists of one or more items. Every item is either an atomic
                value or a node. Furthermore, a single item is also a
                sequence.</para>

            <para>The function signature defines that any sequence containing
                zero or more strings is acceptable for our method. We thus have
                to check if the empty sequence has been passed. In this case,
                the function call returns immediately. Otherwise, we iterate
                through each item in the sequence, prepend "echo: " to its
                string value and add it to the result sequence.</para>

            <para>Next, we want to add the function to a new module. We thus
                have to provide a driver class:</para>

            <example>
                <title>Driver Class</title>
                <programlisting><![CDATA[
public class ExampleModule extends AbstractInternalModule {

public final static String NAMESPACE_URI = 
    "http://exist-db.org/xquery/examples";
	
public final static String PREFIX = "example";
	
private final static FunctionDef[] functions = {
	new FunctionDef(EchoFunction.signature, EchoFunction.class)
};
	
public ExampleModule() {
	super(functions);
}

public String getNamespaceURI() {
	return NAMESPACE_URI;
}

public String getDefaultPrefix() {
	return PREFIX;
}

}]]></programlisting>
            </example>

            <para>The driver class defines a namespace URI and a default prefix
                for the module. Functions are registered by passing an array of
                FunctionDef to the constructor.</para>

            <para>Finally, we are now able to use the new module in an XQuery
                script:</para>

            <example>
                <title>Importing the Module</title>
                <programlisting>
<![CDATA[xquery version "1.0";

import module "http://exist-db.org/xquery/examples"
at "java:org.exist.examples.xquery.ExampleModule";

example:echo(("Hello", "World!"))
]]></programlisting>
            </example>

            <para>The query engine recognizes the "java:" prefix in the location
                URI and treats the remaining part of the string as a fully
                qualified class name leading to the driver class of the
                module.</para>
        </section>
        
        <section>
            <title>Collations</title>
            
            <para>Collations are used to compare strings in a locale-sensitive fashion.
            XQuery allows to specify collations at several places by means of a collation 
            URI, for example, in the <command>order by</command> clause of a FLWOR expression
            and all string-related functions. However, the concrete form of the URI is
            implementation defined.</para>
            
            <para>eXist recognizes the following URIs:</para>
            
            <orderedlist>
				<listitem>
					<synopsis>http://www.w3.org/2004/07/xpath-functions/collation/codepoint</synopsis>
					<para>Selects the unicode codepoint collation. This is the default if no collation 
						is specified. Basically, it means that only the standard Java implementations 
						of the comparison and string search functions are used.</para>
				</listitem>
				<listitem>
					<synopsis>http://exist-db.org/collation?lang=xxx&amp;strength=xxx&amp;decomposition=xxx</synopsis>
					<para>or just</para>
					<synopsis>?lang=xxx&amp;strength=xxx&amp;decomposition=xxx</synopsis>
					<para><option>lang</option> selects a locale. The parameter should have the same form as in xml:lang, 
						for example: "de" or "de-DE" to select a german locale.</para>
					<para><option>strength</option> (optional): value should be one of "primary", "secondary", "tertiary" 
						or "identical".</para>
					<para>decomposition (optional): one of "none", "full" or "standard".</para>
				</listitem>
            </orderedlist>
			
			<para>An example for selecting a german locale for sorting:</para>
			
			<synopsis>for $w in ("das", "daß", "Buch", "Bücher", "Bauer", "Bäuerin", "Jagen", "Jäger") 
				order by $w collation "?lang=de-DE" 
				return $w</synopsis>
				
			<para>returns:</para>
			
			<synopsis>Bauer, Bäuerin, Buch, Bücher, das, daß, Jagen, Jäger</synopsis>
			
			<para>You can also change the default collation:</para>
			
			<synopsis>declare default collation "?lang=de-DE"; 
				"Bäuerin" &lt; "Bier"</synopsis>
				
			<para>returns "true". If you use the default codepoint collation instead, the comparison
			would evaluate to "false".</para>
		</section>
		
		<section>
			<title>XQuery Pragmas</title>
			
			<para>A pragma is used to provide additional information to the XQuery 
            engine. This is very handy to specify things like output properties. The 
            syntax is</para>

            <synopsis>(::pragma QName contents ::)</synopsis>

            <para>A pragma can be specified anywhere that whitespace is allowed. It is basically 
            handled like a comment. eXist understands the following pragmas:</para>

            <synopsis>(::pragma exist:timeout query-timeout ::)</synopsis>

            <para>Specifies the maximum amount of time (in ms) a query can take before it is 
            cancelled by the XQuery engine.</para>

            <synopsis>(::pragma exist:output-size-limit size-hint ::)</synopsis>

            <para>Defines a limit for the max. size of a document fragment created within an 
            XQuery. The limit is just an estimation, specified in terms of the 
            accumulated number of nodes contained in all generated fragments. This can be 
            used to prevent users from consuming too much memory if they are allowed to 
            pass in their own XQueries.</para>

            <synopsis>(::pragma exist:serialize options-list ::)</synopsis>

            <para>exist:serialize can be used to set various output options dynamically. Options 
            affect the way in which generated results are serialized to XML. For example, 
            you can turn on/off XInclude expansion here. The options are the same that 
            can be passed via XPathQueryService.setProperty() and 
            XQueryService.setProperty() in Java. They are defined in 
            javax.xml.transform.OutputKeys and <ulink url="api/org/exist/storage/serializers/EXistOutputKeys.html">
            EXistOutputKeys</ulink>. Valid options include:</para>

            <itemizedlist>
                <listitem>indent=yes|no</listitem>
                <listitem>omit-xml-declaration=yes|no</listitem>
                <listitem>expand-xincludes=yes|no</listitem>
                <listitem>highlight-matches=elements|attributes|both</listitem>
                <listitem>process-xsl-pi=yes|no</listitem>
            </itemizedlist>
            
            For example, to disable XInclude expansion and indent the output you can use 
            the pragma:

            <synopsis>(::pragma exist:serialize expand-xincludes=no indent=yes ::)</synopsis>

            <para>Internally, the XQuery parser removes the pragma (like a comment) from the 
            query during lexical analysis. However, the pragma is added to the 
            XQueryContext class and can be retrieved from there. Unknown pragmas are 
            simply ignored.</para>

		</section>
    </chapter>
</book>
<commit_msg>*** empty log message ***<commit_after>﻿<?xml version="1.0" encoding="UTF-8"?>

<book xmlns:ci="http://apache.org/cocoon/include/1.0">
    <bookinfo>
        <graphic fileref="logo.jpg"/>

        <title>Open Source Native XML Database</title>
        <author>
            <firstname><NAME>.</firstname>
            <surname>Meier</surname>
            <affiliation>
                <address format="linespecific">
                    <email><EMAIL></email>
                </address>
            </affiliation>
        </author>
    </bookinfo>
    <ci:include src="sidebar.xml"/>
    <chapter>
        <title>XQuery</title>
        <section>
            <title>Current Status</title>

            <para>eXist implements the current XQuery working draft as of
                November, 2003 with the exception of the XML Schema related
                features. Currently, eXist passes most of the official XQuery 
                use cases. According to the Bumblebee XQuery test harness, 92% of all use
                cases are passed (excluding the "strong" use cases, which are
                currently beyond eXist's scope).</para>

            <para>The XQuery engine is a complete rewrite and replaces the old
                XPath engine. Older code based on XPath should be compatible
                with the XQuery engine, though some aspects (e.g. the rules for
                automatic type conversions) have changed. The query engine
                doesn't make a difference between XPath and XQuery syntax
                internally. All interfaces should work as before, i.e. you can
                pass an XQuery expression wherever you used to pass an XPath
                query.</para>

            <para>Basically, eXist implements all features described in the
            XQuery specification, except the following:</para>

            <itemizedlist>
                <listitem>Unrecognized keywords: 
                <command>typeswitch</command>, <command>treat as</command>,
                <command>castable as</command>.
                </listitem>
                <listitem>Schema related features: <command>validate</command>,
                <command>import schema</command>, <command>declare validation</command>.
                </listitem>

                <listitem>Since schema support is missing, the value of a node
                    is always treated as xs:untypedAtomic.</listitem>

				<listitem>For the same reason, one cannot specify a type name
					in an element or attribute test. The test:
					element(test) works, but: element(test, xs:integer)
					results in a syntax error.</listitem>

                <listitem>The atomic types xs:gYear, xs:gMonth etc. are not
                    recognized.</listitem>

            </itemizedlist>

            <para>Another limitation applies to <emphasis>constructed document
            fragments</emphasis>: it is currently not possible to query document
            fragments that have been constructed within the XQuery, i.e. a query
            like the following will fail:</para>

            <synopsis>let $x := &lt;test>Test&lt;/test> return $x/test</synopsis>

            <para>At this stage of development, the constructed in-memory DOM
                tree can not be queried in the same way as the persistent
                database DOM (the in-memory DOM is not yet complete and will be
                extended in a future release). You have to store the fragment
                into the database first. You can use the provided <ulink
                        url="#xmldb">xmldb extension functions</ulink> for
                that.</para>

            <para>A list of all the functions currently known to the system can
                be viewed <a href="xquery/functions.xq">here</a>. This list is
                dynamically generated by an XQuery script, so the database needs
                to be running. The function description is directly extracted
                from the signature provided by each class implementing the
                <classname>Function</classname> interface. </para>
            

        </section>
        <section id="ext">
            <title>Extensions</title>

            <para>eXist offers a number of additional functions and two
                additional operators. These extensions can be grouped into three
                main categories:</para>

            <orderedlist>
                <listitem>
                    Functions to specify the input document set for an expression.
                </listitem>
                <listitem>
                    Extensions for full-text searching.
                </listitem>
                <listitem>
                    Functions for manipulating database contents.
                </listitem>
                <listitem>
                    Utility functions.
                </listitem>
                <listitem>
                    XSL transformation functions.
                </listitem>
                <listitem>
                    HTTP related functions.
                </listitem>
            </orderedlist>

            <para>We start with the additional functions for specifying
            the input document set.</para>

            <section>
                <title>Specifying the Input Document Set</title>
                <para>A database may contain an unlimited set of collections and documents.
                If you use the XML:DB API to query the db, only the documents in the current 
                XML:DB collection are processed by default. However, there are four additional 
                functions to change this behavior: <command>doc()</command>, 
                <command moreinfo="none">document()</command>, 
                <command moreinfo="none">collection()</command> and <command>xcollection()</command>.</para>

                <para>The <command>collection()</command> and <command>doc()</command> 
            functions are standard XQuery/XPath functions.
            <command>xcollection()</command> and <command>document()</command> 
            represent eXist-specific extensions.</para>

                <para>eXist interprets the arguments to <command>collection</command> and <command>doc</command>
            as absolute or relative paths, leading to some collection or
            document within the database. Relative paths are resolved relative to the base
            URI property of the static XQuery context, which usually points to a database
            collection.</para>

                <note>
                    <para>Currently, the eXist-specific, document related functions are in the XQuery/XPath
                core library functions namespace (i.e. http://www.w3.org/2003/05/xpath-functions).
                This is definitely wrong, but previous versions of eXist did not use namespaces for
                functions, so older eXist-based applications would complain. However, in the future, the eXist-specific 
                functions will move to their own namespace.</para>
                </note>

                <para>While <command>doc()</command> is restricted to a single document-URI argument, 
            <command>document</command> accepts multiple
            document paths to be included into the input node set. Second, calling 
            <command>document</command> without an argument includes <emphasis>every</emphasis>
            document node in the current database instance. Some examples:</para>

                <synopsis format="linespecific">document()//SPEAKER</synopsis>
                <synopsis format="linespecific">document('/db/test/abc.xml', '/db/test/def.xml')//title</synopsis>

                <para>The <command moreinfo="none">collection()</command> function specifies the 
            collection whose documents are to be included into query evaluation. By default, 
            documents found in sub-collections of the specified collection are included. This 
            means that for three collections, <filename
                            moreinfo="none">/db/test</filename>, 
            <filename moreinfo="none">/db/test/abc</filename> and 
            <filename moreinfo="none">/db/test/def</filename>, 
            <command moreinfo="none">collection('/db/test')</command> will include all resources 
            found in <filename moreinfo="none">/db/test</filename>, 
            <filename moreinfo="none">/db/test/abc</filename> and 
            <filename moreinfo="none">/db/test/def</filename>. You may use the 
            <command moreinfo="none">xcollection()</command> function to change this behavior, 
            e.g.</para>
                <synopsis format="linespecific">xcollection('/db/test')//title</synopsis>
                <para>will only include resources found in <filename
                            moreinfo="none">/db/test</filename>, but not in <filename
                            moreinfo="none">/db/test/abc</filename> or <filename
                            moreinfo="none">/db/test/def</filename>.</para>
                <note>
                    <para>Beginning with version 1.0beta1, the collection and document functions are implemented
                as ordinary functions. They may occur at any position where a function call is allowed.</para>
                </note>
            </section>
            <section>
                <title>Querying Text</title>
                <para>The standard XPath/XQuery function library contains most
                of the well-known string manipulation functions provided by
                every programming languages. However, these functions are rather
                insufficient when searching for a set of keywords or a phrase inside a
                larger portion of text or mixed content.</para>

                <para>This is a weak point if you have to work with document-centric 
                (as opposed to data-centric) documents. For many types of documents, the 
                provided standard
                functions will not yield satisfactory results. For example, you might remember 
                to have read something about "XML" and "databases" in some chapter of a book, 
                but you may not be sure where exactly it was. Using standard XPath, you 
                could try a query like:</para>

                <synopsis format="linespecific">//chapter[contains(., 'XML') and contains(., 'databases')]</synopsis>

                <para>Query execution will probably be quite slow, because the XPath engine has to scan 
                the entire character content of all chapter nodes and their descendants. Still you can't be sure 
                to find all matches - for example, "databases" might have been written with a capital letter 
                at the start of the sentence.</para>
                <para>eXist offers additional operators and extension functions for efficient, index-based 
                access to the full-text content of nodes. With eXist, you could formulate 
                your query as follows:</para>
                <synopsis format="linespecific">//chapter[near(., 'XML database?', 50)]</synopsis>
                <para>This query will return all chapters containing both keywords in the correct 
                order and with less than 50 words between them. Additionally, the wildcard in "database?" 
                will match the singular as well as the plural form and the search is done case-insensitive.</para>
                <para>Since the query is index-based, it will usually be a magnitude faster than the standard 
                XPath query above. Let's walk through each of eXist's extensions:</para>
                <section>
                    <title>Operators</title>
                    <para>In cases where the order and distance of search terms is not important, eXist offers 
                    two additional operators for simple keyword queries: 
                    <command moreinfo="none">&amp;=</command> and <command
                                moreinfo="none">|=</command>. 
                    </para>

                    <variablelist>
                        <varlistentry>
                            <term>node-set &amp;= 'string of keywords'</term>
                            <listitem>
                                <para>selects context nodes containing all of the keywords in the right-hand 
                                argument in any order. The default tokenizer is used to split the right-hand 
                                argument into single tokens, i.e. any punctuation or white space is skipped. 
                                Wildcards are allowed. Keywords are compared case-insensitive.</para>
                            </listitem>
                        </varlistentry>
                        <varlistentry>
                            <term>node-set |= 'string of keywords'</term>
                            <listitem>
                                <para>as above, but selects context nodes containing <emphasis>any</emphasis> 
                                of the keywords in the right-hand argument.</para>
                            </listitem>
                        </varlistentry>
                    </variablelist>

                    <note>
                        <para>The strings are split into tokens using the
                        default tokenizer function. The current implementation
                        will work well for all european languages. For
                        non-european languages, things are a bit more
                        complicated. In these cases, eXist uses the predefined
                        unicode codepoints to decide where the string will be
                        split.</para>
                    </note>

                    <para>Both operators accept simple wildcards in the keyword string. A 
                    <command moreinfo="none">?</command> matches zero or one character, 
                    <command moreinfo="none">*</command> matches zero or more characters. 
                    A character range <command moreinfo="none">[abc]</command> matches any 
                    of the characters in the range. You may use a backslash to escape wildcard 
                    characters.</para>

                    <para>To match more complex patterns, full regular expression syntax is 
                    supported through additional functions, which will be described below.</para>

                    <para>Please note that there is an important semantic difference between the two expressions:</para>
                    <synopsis format="linespecific">document()//SPEECH[LINE &amp;= "cursed spite"]</synopsis>
                    <para>and</para>
                    <synopsis format="linespecific">document()//SPEECH[LINE &amp;= "cursed" and LINE &amp;="spite"]</synopsis>

                    <para>The first expression selects all distinct LINE nodes that contain both terms. The second 
                    one selects all context nodes (SPEECH) that have line children containing either or 
                    both of the terms. The second expression is supposed to yield more results than the 
                    first one. To make both expressions select nearly the same nodes, you would have to 
                    change the first one into</para>
                    <synopsis format="linespecific">document()//SPEECH[. &amp;= "cursed spite"]</synopsis>
                    <para>(but this will also include, say SPEAKER or STAGEDIR children of the SPEECH parent)</para>
                </section>
                <section>
                    <title>near()</title>
                    <para>As shown above, the <command
                                moreinfo="none">near()</command> function behaves quite 
                    similar to the <command moreinfo="none">&amp;=</command> operator, but it additionally pays 
                    attention to the order of search terms and their distance in the source document. The syntax is:</para>
                    <synopsis format="linespecific">near(node-list, 'string of keywords' [, max-distance])</synopsis>
                    <para>The functions measures the distance between two search terms by counting the words in 
                    between them. A maximum distance of 1 is assumed by default, which means that the search terms 
                    should occur next to each other. Other values may be specified in the optional third argument, 
                    e.g.:</para>
                    <synopsis format="linespecific">document()//SPEECH[near(., 'love marriage', 25)</synopsis>
                    <para>will return any SPEECH elements containing the words "love" and "marriage" and less 
                    than 25 words between them.</para>
                    <para>Like <command moreinfo="none">&amp;=</command>, <command
                                moreinfo="none">near()</command> 
                    accepts wildcards in the keyword string. As well, punctuation and whitespace will be skipped 
                    according to the default tokenization rules.</para>
                </section>
                <section>
                    <title>match-all()/match-any()</title>
                    <para>These functions are variations of the <command
                                moreinfo="none">&amp;=</command> and 
                    <command moreinfo="none">|=</command> operators which interpret their arguments as 
                    regular expressions. <emphasis>But</emphasis>: contrary to the <command>matches</command>
                    function in the XQuery core library,
                    <command>match-all</command> and
                    <command>match-any</command> try to match the regular
                    expression argument against the keywords contained in the
                    full-text index, not against the entire text. Assume a document containing the
                    following paragraph:</para>

                    <synopsis>&lt;para&gt;Peter lives in Frankfurt&lt;/para&gt;</synopsis>

                    <para>Then the following expression:</para>

                    <synopsis>match-all(para, "li[vf]e.?", "frank.*")</synopsis>

                    <para>will match because the paragraph contains two keywords
                    matching the specified regular expression patterns.</para>

                    <para>
                        <command moreinfo="none">match-all()</command> corresponds to 
                    <command moreinfo="none">&amp;=</command> in that it
                    will select context nodes with keywords
                    matching all of the specified regular expressions. <command
                                moreinfo="none">match-any()</command> 
                    will select nodes with keywords matching <emphasis>any</emphasis> of the specified regexp.</para>

                    <para>Since tokenization doesn't work correctly with regular expression patterns, each 
                    keyword has to be specified as a separate argument, so the syntax looks like:</para>

                    <synopsis format="linespecific">match-all(node-set, 'regexp' [, 'regexp' ...])</synopsis>

                    <para>Please note that the functions will try to match the regexp against the entire 
                    keyword. For example, the expression</para>
                    <synopsis format="linespecific">//SPEECH[match-all(LINE, 'li[vf]e')]</synopsis>
                    <para>will match 'live', 'life', but not 'lives'.</para>
                    <para>eXist uses <ulink
                                url="http://jakarta.apache.org/oro/">Apache's ORO library </ulink> for regular expressions. A complete description of the supported regexp syntax can be found in the documentation of the library.</para>
                </section>
            </section>
            <section id="xmldb">
                <title>Manipulating Database Contents</title>

                <para>The xmldb extension functions can be used to create
                new database collections, or to store query output back
                into the database. For example, assume that someone has sent us
                a large file containing a number of RDF metadata records.
                Unfortunately, we don't want to store the metadata records
                as one, single file. Instead, our application expects each
                record to be in its own document. We thus have to transform
                the document and split it into smaller units.</para>

                <para>Using an XSLT stylesheet would be one alternative.
                However, it is rather memory-intensive. So why not use XQuery
                to do the job? The XQuery script below shows how to split the
                large RDF file into a number of smaller documents:</para>

                <example>
                    <title>Splitting a Document</title>
                    <programlisting>
<![CDATA[
xquery version "1.0";

declare namespace  xdb="http://exist-db.org/xquery/xmldb";
declare namespace util="http://exist-db.org/xquery/util";

xdb:register-database("org.exist.xmldb.DatabaseImpl", true()),
let $root := xdb:collection("xmldb:exist:///db", "admin", ""),
    $out := xdb:create-collection($root, "output")
for $rec in /rdf:RDF/* return
    xdb:store($out, concat(util:md5($rec/@rdf:about), ".xml"),
        <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
            {$rec}
        </rdf:RDF>
    )
]]>
                    </programlisting>
                </example>
                <para>The extension functions are based on the XML:DB API, so we
                first have to register a database driver with
                <command>xdb:register-database</command>. Then we retrieve the
                root collection with <command>xdb:collection</command>. The
                variable returned by <command>xdb:collection</command> holds
                the collection as a Java object. We can pass this variable to
                <command>xdb:create-collection</command> to create a new
                sub-collection, called "output".</para>

                <para>The <command>for</command> expression iterates through all
                child elements of the top RDF element. In each iteration, we use
                <command>xdb:store</command> to write out the current child
                node into a new document.</para>

                <para>Obviously, a unique document name is required for each
                document. In this example, the URI contained in the rdf:about attribute is
                unique, so we simply compute an MD5 key from it, append ".xml" and use it as
                document name.</para>
            </section>

            <section>
                <title>Utility Functions</title>

                <para>The function namespace http://exist-db.org/xquery/util
                contains a number of common utility functions. For example,
                the <command>util:md5</command> function used above is defined 
                here.</para>

                <para>We will not discuss all available functions. They 
                documented in the <a href="xquery/functions.xq">functions table</a>.
                However, there's one function which needs some further
                explanation:</para>

                <para>
                    <command>util:eval</command> is used to dynamically
                execute a constructed XQuery expression inside a running XQuery
                script. This is very handy sometimes, for example, in a
                web-based application, where queries are dynamically built from
                the request parameters the user has passed to the application.</para>

                <para>In this case, there would be one XQuery script, which is responsible
                for evaluating the user supplied parameters.
                This script uses the parameters from the HTTP request to create another
                XQuery expression. It then passes the constructed query string
                to <command>util:eval</command> to evaluate the expression,
                post-processes the returned results and displays them to the
                user. More information about how to write web applications with XQuery 
                can be found in the <ulink url="devguide.xml">Developer's Guide</ulink>.
                Below is a simple example to add or subtract two numbers:
                </para>

                <example>
                    <title>Adding/Subtracting Two Numbers</title>

                    <programlisting>
<![CDATA[xquery version "1.0";

declare namespace request="http://exist-db.org/xquery/request";
declare namespace util="http://exist-db.org/xquery/util";

declare function local:do-query() as element()
{
    let $n1 := request:request-parameter("n1", ""),
		$n2 := request:request-parameter("n2", ""),
		$op := request:request-parameter("op", "")
	return
		if($n1 = "" or $n2 = "") then
			<p>Please enter two operands.</p>
		else
			let $query := concat($n1, " ", $op, " ", $n2)
			return
				<p>{$query} = {util:eval($query)}</p>
};

<html>
	<body>
		<h1>Enter two numbers</h1>

		<form action="{request:request-uri()}" method="get">
			<table border="0" cellpadding="5">
			<tr>
				<td>First number:</td>
				<td><input name="n1" size="4"/></td>
			</tr>
			<tr>
				<td>Operator:</td>
				<td>
					<select name="op">
						<option name="+">+</option>
						<option name="-">-</option>
					</select>
				</td>
			</tr>
			<tr>
				<td>Second number:</td>
				<td><input name="n2" size="4"/></td>
			</tr>
			<tr>
				<td colspan="2"><input type="submit"/></td>
			</tr>
			</table>
		</form>

		{ local:do-query() }
	</body>
</html>]]>
                    </programlisting>
                </example>
            </section>

            <section>
                <title>XSL Transformations</title>

                <para>eXist has a function to directly apply an XSL stylesheet
                    to an XML fragment within the XQuery script. This can be
                    very convenient, for example, if your requirements are
                    simple and you don't want to install Cocoon with your web
                    application. The function signature is as follows:</para>

                <synopsis>transform:transform($input as node()?, $stylesheet as
                    xs:string, $parameters as node()?) as node()?</synopsis>

                <para>
                    <command>transform:transform</command> expects the node to
                    be transformed in the first argument $input. If $input is
                    the empty sequence, the function returns immediately.</para>

                <para>The XSL stylesheet will be read from the location
                    specified in $stylesheet. It should either be an absolute
                    URI or a relative path. If a relative path is specified, the
                    function will first try to find a file matching the path. If
                    this fails, <command>transform:transform</command> appends
                    the path to the current base URI property from the XQuery
                    context and retries.</para>

                <para>The stylesheet will be compiled into a template using the
                    standard Java APIs (javax.xml.transform). The template is
                    shared between all instances of the function and will only
                    be reloaded if modified since the last invocation.</para>

                <para>The $options parameter can be used to pass stylesheet
                    parameters to the XSL processor. Parameters are passed as an
                    XML fragment:</para>

                <synopsis><![CDATA[<parameters>
    <param name="param1" value="value1"/>
    <param name="param2" value="value2"/>
</parameters>]]></synopsis>

                <para>This will set the stylesheet parameter
                    <option>param1</option> to the string value
                    <option>value1</option>. In the stylesheet, the parameter
                    can then be referenced with</para>

                <synopsis><![CDATA[<xsl:param name="param1"/>]]></synopsis>
            </section>

            <section>
                <title>HTTP Related Functions</title>

                <para>The example above reads HTTP request parameters using the
                function <command>request:request-parameter</command>. Obviously, 
                functions in the http://exist-db.org/xquery/request namespace 
                are only usable if the query is executed through the
                XQueryGenerator or the XQueryServlet (please have a look at the 
                <ulink url="devguide.xml">Developer's Guide</ulink> for more 
                information).</para>

                <para>
                    <command>request:request-parameter</command> expects two
                arguments: the first argument denotes the name of the parameter,
                the second specifies a default value, which is returned if the
                parameter is not set. The function returns a sequence containing
                the values for the parameter.</para>

                <para>
                    <command>request:request-uri</command> returns the URI of
                the current request. You may call</para>

                <synopsis>request:encode-url(request:request-uri())</synopsis>

                <para>to encode the current session id into the URI.</para>

                <para>There are also two functions to read and set session
                attributes. Any XQuery value or Java object value can be stored 
                into a session attribute, for example, to cache query results.
                Please have a look at the provided <ulink
                            url="examples.xml">examples</ulink>.</para>

                <para>The function call
                <option>request:create-session()</option> will create a new HTTP
                session if none has been created before.</para>
            </section>
        </section>
        <section>
            <title>Java Binding</title>
            <para>eXist supports calls to arbitrary Java methods from within XQuery. The 
        	binding mechanism follows the short-cut technique introduced by 
        	<ulink url="http://saxon.sf.net">Saxon</ulink>. The class where the external function
        	will be found is identified by the namespace URI of the function call. The namespace
        	URI should start with the prefix <option>java:</option> followed by the fully qualified
        	class name of the class. For example, the following code snippet calls the static method
        	<methodname>sqrt</methodname> of class <classname>java.lang.Math</classname>:</para>
            <example>
                <title>Calling a Static Method</title>
                <programlisting><![CDATA[
declare namespace math="java:java.lang.Math";
math:sqrt(2)
				]]></programlisting>
            </example>
            <para>If the function name contains a hyphen, the letter following the hyphen is converted
        	to upper-case and the hyphen is removed (camel case), i.e. <code>to-string()</code> will call
        	the Java method <code>toString()</code>.</para>
            <para>If more than one method in the class matches the given name and parameter count,
        	eXist tries to select the method that best fits the passed parameter types at runtime.
        	The result of the method call can be assigned to an XQuery variable. If possible, it will
        	be mapped to the corresponding XML schema type. Otherwise, it's type is the built-in type
        	<option>object</option>.</para>
            <para>
                <emphasis>Java constructors</emphasis> are called using the function <option>new</option>. Again, a matching
        	constructor is selected by looking at the parameter count and types. The returned value is a
        	new Java object with the built-in type <option>object</option>.</para>
            <para>
                <emphasis>Instance methods</emphasis> are called by supplying a valid Java object as
        	first parameter. The Java object has to be an instance of the given class. For example, the
        	following snippet lists all files and directories in the current directory:</para>
            <example>
                <title>List Contents of the Current Directory</title>
                <programlisting><![CDATA[
declare namespace file="java:java.io.File";

<files>
    {
        for $f in file:list-files( file:new(".") )
        let $n := file:get-name($f)
        order by $n
        return
            if (file:is-directory($f)) then
                <directory name="{ $n }"/>
            else
                <file name="{ $n }" size="{ file:length($f) }"/>
    }
</files>       			
        		]]></programlisting>
            </example>
        </section>

        <section>
            <title>Writing your own Modules</title>

            <para>eXist supports XQuery's notion of library modules, i.e. you
                can declare an XQuery file as a module and later import it using
                the <command>import module</command> directive.</para>

            <para>The XQuery engine imports each module only once during
                compilation. The compiled module is then made available through
                the static XQuery context.</para>

            <para>A library module is basically just a collection of function
                definitions and global variable declarations. eXist knows two
                types of modules: external modules which are themselves written
                in XQuery, and internal modules implemented in Java. The
                standard XPath/XQuery functions and all extension functions
                described above are defined through internal modules.</para>

            <para>Users may provide additional Java modules. This is not too
                difficult, though the API may still change in the future. To
                register a module, eXist needs to know the namespace URI by
                which the module is identified and the list of functions it
                supplies. Basically, this works by passing a driver class to
                the XQuery engine. The driver class should implement interface
                <classname>org.exist.xpath.InternalModule</classname>.</para>

            <para>Class <classname>org.exist.xpath.AbstractInternalModule</classname>
                already provides an implementation skeleton. The class
                constructor expects an array of function definitions for all
                functions that should be registered. A function definition
                (class <classname>FunctionDef</classname>) has two properties:
                the static signature of the function (as an instance of
                <classname>FunctionSignature</classname>) and the Java Class
                that implements the function.</para>

            <para>A function is a class extending
                <classname>org.exist.xpath.Function</classname> or
                <classname>org.exist.xpath.BasicFunction</classname>. Functions
                without special requirements (e.g. overloading) should subclass BasicFunction. As
                an example, we define a simple function below:</para>

            <example>
                <title>A Basic Function</title>
                <programlisting>
<![CDATA[
public class EchoFunction extends BasicFunction {

public final static FunctionSignature signature =
new FunctionSignature(
	new QName("echo", ExampleModule.NAMESPACE_URI, ExampleModule.PREFIX),
	"A useless example function. It just echoes the input parameters.",
    new SequenceType[] {
        new SequenceType(Type.STRING, Cardinality.ZERO_OR_MORE)
    },
	new SequenceType(Type.STRING, Cardinality.ZERO_OR_MORE));

public EchoFunction(XQueryContext context) {
	super(context, signature);
}

public Sequence eval(Sequence[] args, Sequence contextSequence)
throws XPathException {
	// is argument the empty sequence?
	if (args[0].getLength() == 0)
		return Sequence.EMPTY_SEQUENCE;
	// iterate through the argument sequence and echo each item
	ValueSequence result = new ValueSequence();
	for (SequenceIterator i = args[0].iterate(); i.hasNext();) {
		String str = i.nextItem().getStringValue();
		result.add(new StringValue("echo: " + str));
	}
	return result;
}

}]]></programlisting>
            </example>

            <para>First, every function class has to provide a function signature. The
                function signature defines the QName by which the function is
                identified, a documentation string, the sequence types of
                all arguments and the sequence type of the returned value. In
                the example above, we accept a single argument of type xs:string
                and a cardinality of ZERO_OR_MORE. In other words, we accept any sequence
                of strings containing zero or more items.</para>

            <para>Next, the subclass overwrites the <command>eval</command>
                method, which has two arguments: the first contains the values
                of all arguments passed to the function, the second passes the
                current context sequence (which might be null).</para>

            <para>The argument values in the array <option>args</option> have
                already been checked to match the sequence types defined in the
                function signature. We thus don't have to recheck the length of
                the array: if more or less than one argument were passed to the
                function, an exception would have been thrown before eval gets
                called.</para>

            <para>In XQuery all values are passed as sequences. A sequence
                consists of one or more items. Every item is either an atomic
                value or a node. Furthermore, a single item is also a
                sequence.</para>

            <para>The function signature defines that any sequence containing
                zero or more strings is acceptable for our method. We thus have
                to check if the empty sequence has been passed. In this case,
                the function call returns immediately. Otherwise, we iterate
                through each item in the sequence, prepend "echo: " to its
                string value and add it to the result sequence.</para>

            <para>Next, we want to add the function to a new module. We thus
                have to provide a driver class:</para>

            <example>
                <title>Driver Class</title>
                <programlisting><![CDATA[
public class ExampleModule extends AbstractInternalModule {

public final static String NAMESPACE_URI = 
    "http://exist-db.org/xquery/examples";
	
public final static String PREFIX = "example";
	
private final static FunctionDef[] functions = {
	new FunctionDef(EchoFunction.signature, EchoFunction.class)
};
	
public ExampleModule() {
	super(functions);
}

public String getNamespaceURI() {
	return NAMESPACE_URI;
}

public String getDefaultPrefix() {
	return PREFIX;
}

}]]></programlisting>
            </example>

            <para>The driver class defines a namespace URI and a default prefix
                for the module. Functions are registered by passing an array of
                FunctionDef to the constructor.</para>

            <para>Finally, we are now able to use the new module in an XQuery
                script:</para>

            <example>
                <title>Importing the Module</title>
                <programlisting>
<![CDATA[xquery version "1.0";

import module "http://exist-db.org/xquery/examples"
at "java:org.exist.examples.xquery.ExampleModule";

example:echo(("Hello", "World!"))
]]></programlisting>
            </example>

            <para>The query engine recognizes the "java:" prefix in the location
                URI and treats the remaining part of the string as a fully
                qualified class name leading to the driver class of the
                module.</para>
        </section>
        
        <section>
            <title>Collations</title>
            
            <para>Collations are used to compare strings in a locale-sensitive fashion.
            XQuery allows to specify collations at several places by means of a collation 
            URI, for example, in the <command>order by</command> clause of a FLWOR expression
            and all string-related functions. However, the concrete form of the URI is
            implementation defined.</para>
            
            <para>eXist recognizes the following URIs:</para>
            
            <orderedlist>
				<listitem>
					<synopsis>http://www.w3.org/2004/07/xpath-functions/collation/codepoint</synopsis>
					<para>Selects the unicode codepoint collation. This is the default if no collation 
						is specified. Basically, it means that only the standard Java implementations 
						of the comparison and string search functions are used.</para>
				</listitem>
				<listitem>
					<synopsis>http://exist-db.org/collation?lang=xxx&amp;strength=xxx&amp;decomposition=xxx</synopsis>
					<para>or just</para>
					<synopsis>?lang=xxx&amp;strength=xxx&amp;decomposition=xxx</synopsis>
					<para><option>lang</option> selects a locale. The parameter should have the same form as in xml:lang, 
						for example: "de" or "de-DE" to select a german locale.</para>
					<para><option>strength</option> (optional): value should be one of "primary", "secondary", "tertiary" 
						or "identical".</para>
					<para>decomposition (optional): one of "none", "full" or "standard".</para>
				</listitem>
            </orderedlist>
			
			<para>An example for selecting a german locale for sorting:</para>
			
			<synopsis>for $w in ("das", "daß", "Buch", "Bücher", "Bauer", "Bäuerin", "Jagen", "Jäger") 
				order by $w collation "?lang=de-DE" 
				return $w</synopsis>
				
			<para>returns:</para>
			
			<synopsis>Bauer, Bäuerin, Buch, Bücher, das, daß, Jagen, Jäger</synopsis>
			
			<para>You can also change the default collation:</para>
			
			<synopsis>declare default collation "?lang=de-DE"; 
				"Bäuerin" &lt; "Bier"</synopsis>
				
			<para>returns "true". If you use the default codepoint collation instead, the comparison
			would evaluate to "false".</para>
		</section>
		
		<section>
			<title>XQuery Pragmas</title>
			
			<para>A pragma is used to provide additional information to the XQuery 
            engine. This is very handy to specify things like output properties. The 
            syntax is</para>

            <synopsis>(::pragma QName contents ::)</synopsis>

            <para>A pragma can be specified anywhere that whitespace is allowed. It is basically 
            handled like a comment. eXist understands the following pragmas:</para>

            <synopsis>(::pragma exist:timeout query-timeout ::)</synopsis>

            <para>Specifies the maximum amount of time (in ms) a query can take before it is 
            cancelled by the XQuery engine.</para>

            <synopsis>(::pragma exist:output-size-limit size-hint ::)</synopsis>

            <para>Defines a limit for the max. size of a document fragment created within an 
            XQuery. The limit is just an estimation, specified in terms of the 
            accumulated number of nodes contained in all generated fragments. This can be 
            used to prevent users from consuming too much memory if they are allowed to 
            pass in their own XQueries.</para>

            <synopsis>(::pragma exist:serialize options-list ::)</synopsis>

            <para>exist:serialize can be used to set various output options dynamically. Options 
            affect the way in which generated results are serialized to XML. For example, 
            you can turn on/off XInclude expansion here. The options are the same that 
            can be passed via XPathQueryService.setProperty() and 
            XQueryService.setProperty() in Java. They are defined in 
            javax.xml.transform.OutputKeys and <ulink url="api/org/exist/storage/serializers/EXistOutputKeys.html">
            EXistOutputKeys</ulink>. Valid options include:</para>

            <itemizedlist>
                <listitem>indent=yes|no</listitem>
                <listitem>omit-xml-declaration=yes|no</listitem>
                <listitem>expand-xincludes=yes|no</listitem>
                <listitem>highlight-matches=elements|attributes|both</listitem>
                <listitem>process-xsl-pi=yes|no</listitem>
            </itemizedlist>
            
            For example, to disable XInclude expansion and indent the output you can use 
            the pragma:

            <synopsis>(::pragma exist:serialize expand-xincludes=no indent=yes ::)</synopsis>

            <para>Internally, the XQuery parser removes the pragma (like a comment) from the 
            query during lexical analysis. However, the pragma is added to the 
            XQueryContext class and can be retrieved from there. Unknown pragmas are 
            simply ignored.</para>

		</section>
    </chapter>
</book>

{'fraction_non_alphanumeric': 0.08242462765899153, 'fraction_numerical': 0.0018192378927526385, 'mean_word_length': 1.9608670257641638, 'pattern_counts': {'":': 0, '<': 1456, '<?xml version=': 2, '>': 1460, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 44}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5958722', 'n_tokens_mistral': 22883, 'n_tokens_neox': 20675, 'n_words': 8420}