<commit_before><?xml version="1.0" encoding="utf-8"?>
<!-- $Id$ -->
<book>
    <bookinfo>
        <graphic fileref="logo.jpg"/>
        <productname>Open Source Native XML Database</productname>
        <title>Versioning Module</title>
        <orgname>The eXist Project</orgname>
        <date>August 2011</date>
        <author>
            <firstname>Wolfgang</firstname>
            <surname>Meier</surname>
	    <affiliation>
	      <address format="linespecific"><email>wolfgang at exist-db.org</email></address>
	    </affiliation>
        </author>
        <author>
            <firstname>Leif-Jöran</firstname>
            <surname>Olsson</surname>
            <affiliation>
                <address format="linespecific"><email>ljo at exist-db.org</email></address>
            </affiliation>
	</author>
    </bookinfo>
    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="sidebar.xml"/>
    <chapter>
        <title>Versioning Extension</title>

        <abstract>
            <para>eXist-db provides a basic document versioning extension. The
                extension will track all changes to a document by storing a diff between the
                revisions. Older versions can be restored on the fly and even queried in memory.
                There's also basic support to detect and intercept conflicting writes.</para>

            <para>However, eXist-db has no control over the client. It does not know where a document
                update comes from and cannot directly communicate with the user. The versioning
                extension should be seen more like a toolbox than a complete solution. Advanced
                functionality (e.g. merging and conflict resolution) will require support from the
                end-user applications, which is outside of eXist-db's reach.</para>
            
            <para>The versioning extension has been created with human editors in mind, who will 
                typically be changing a document through an editor or some form-based frontend.
                It should work well with documents up to several megabytes in size. 
                However, the versioning will not track machine-generated node-level edits using XUpdate or XQuery 
                update extensions.</para>
        </abstract>

        <section>
            <title>Components</title>

            <para>The versioning extensions consist of the following components:</para>

            <variablelist>
                <varlistentry>
                    <term>VersioningTrigger</term>
                    <listitem>
                        <para>a trigger which has to be registered with a collection and
                        implements the core versioning functionality</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term>VersioningFilter</term>
                    <listitem>
                        <para>a serialization filter which adds special version attributes to every
                            serialized document. The attributes are used to detect conflicting writes.</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term><filename>versioning.xqm</filename></term>
                    <listitem>
                        <para>an XQuery module which provides a function library for end-user
                            applications, including functions like v:doc (used to
                            restore a given revision on the fly).</para>
                    </listitem>
                </varlistentry>
            </variablelist>
        </section>

        <section>
            <title>Setup</title>
            
            <para>Versioning can be enabled for any collection in the collection hierarchy. It
                is not necessary to use versioning for all collections. To enable versioning,
                a trigger has to be registered with the top-level collection. This is done
                through the same collection configuration files that are used for defining
                indexes.</para>
            
            <section>
                <title>Register the versioning trigger</title>
                
                <para>To enable version for a collection, you have to edit the
                collection's <filename>.xconf</filename> configuration file, which has to be 
                stored below the <filename>/db/system/config</filename> collection. As described
                in the <ulink url="indexing.xml">Configuring Indexes</ulink> document, the
                <filename>/db/system/config</filename> collection mirrors the hierarchical
                structure of the main collection tree.</para>
                
                <para>Within the collection's <filename>.xconf</filename>, you should register
                    the trigger class <classname>org.exist.versioning.VersioningTrigger</classname>
                    for the "store", "update" and "remove" events:</para>
                
                <example>
                    <title>Registering trigger in collection.xconf</title>
                    <programlisting language="xml"><![CDATA[<collection xmlns="http://exist-db.org/collection-config/1.0">
    <index>
        <fulltext default="none" attributes="no">
        </fulltext>
    </index>
    <triggers>
        <trigger event="store,remove,update"
            class="org.exist.versioning.VersioningTrigger">
            <parameter name="overwrite" value="yes"/>
        </trigger>
    </triggers>
</collection>]]></programlisting>
                </example>
                
                <para>If you store above document into <filename>/db/system/config/db/collection.xconf</filename>,
                it will enable versioning for the entire database.</para> 
                
                <note>
                    <para>Note that a <filename>collection.xconf</filename> at a lower level in the hierarchy will
                        <emphasis>overwrite</emphasis> any configuration on higher levels, including the
                        trigger definitions. Triggers are not inherited from ancestor configurations. If the 
                        new configuration doesn't define a trigger, the trigger map will be empy.</para>
                    
                    <para>When working with nested collection configurations,
                        you need to make sure that the trigger definitions are present in all
                        <filename>collection.xconf</filename> files.</para>
                </note>
                
                <para>VersioningTrigger accepts one
                    parameter, <option>overwrite</option>: if this is set to "no", the trigger will
                    check for potential write conflicts. For example, if two users opened the same
                    document and are editing it, it may happen that the first user saves his changes
                    without the second user recognizing it. The second user also made changes and if
                    eXist did allow him to store his version, he would just overwrite the modifications
                    already committed by the first user.</para>
                
                <para>The <option>overwrite="no"</option> setting
                    prevents this. However, eXist has no control over the client. It does not know
                    where the conflicting document came from. All it can do is to reject the write
                    attempt and raise an error. The error should then be handled by the client.
                    Right now there are no clients to support this. More work will be required
                    in this area. However, clients can already use the supplied XQuery functions to
                    check for write conflicts (see below).</para>
                
            </section>
            
            <section>
                <title>Enabling the serialization filter</title>
                
                <para>In order to detect conflicting writes, the versioning extension needs
                    to keep track of the base revision to which changes were applied. It does this
                    by inserting special metadata attributes into a document when it is retrieved
                    from the database. For this purpose, a <emphasis>custom filter</emphasis> has
                    to be registered with eXist's serializer. This is done in the <sgmltag>serializer</sgmltag>
                    section in the main configuration file, <filename>conf.xml</filename>. Add a
                    <sgmltag>custom-filter</sgmltag> child tag to the <sgmltag>serializer</sgmltag>
                    element and set its <option>class</option> attribute to 
                    <classname>org.exist.versioning.VersioningFilter</classname>:</para>
                
                <example>
                    <title>Registering the versioning filter in conf.xml</title>
                    <programlisting language="xml"><![CDATA[<serializer add-exist-id="none" compress-output="no" enable-xinclude="yes"
            enable-xsl="no" indent="yes" match-tagging-attributes="no" 
            match-tagging-elements="no">
    <custom-filter class="org.exist.versioning.VersioningFilter"/>
</serializer>]]></programlisting>
                </example>
                
                <para>eXist-db needs to be restarted for the versioning filter to become active.</para>
            </section>
            
        </section>
        
        <section>
            <title>Checking versions through the admin webapp</title>
            
            <para>The admin web application provides a simple way to see the revision history of a
                document and restore old versions. From the sidebar menu, select 
                <guimenuitem>Browse Collections</guimenuitem>. Within the collection view
                you should find a revision number next to each document which has been changed
                since versioning was enabled. Unchanged documents will not have a revision number:</para>
            
            <screenshot>
                <graphic fileref="resources/versioning1.png"/>
            </screenshot>
            
            <para>The revision number is the last number in each row ("34" in the screenshot above).
            You can click on the revision number to see the history of changes for this document:</para>
            
            <screenshot>
                <graphic fileref="resources/versioning2.png"/>
            </screenshot>
            
            <para>Clicking on <guimenuitem>Diff</guimenuitem> shows the differences between the selected
                and the previous revision. The way in which the differences are recorded is specific
                to eXist. It uses eXist's unique node ids to identify the reference node to which a change
                applies (the <option>ref</option> attributes in the XML shown above).</para>
            
            <para>The <guimenuitem>Restore</guimenuitem> link will restore the corresponding revision.
                This is done in memory by recursively applying all diffs since the base revision.</para>
        </section>
        
        <section>
            <title>Accessing the versioning information</title>
            
            <para>The versioning extension uses the collection <filename>/db/system/versions</filename>
                to store base revisions and diffs. The collection hierarchy below <filename>/db/system/versions</filename>
                again mirrors the main collection tree. For each versioned resource, you will find a
                document with suffix <filename>.base</filename>, which contains the base revision, i.e. the
                first version of the document. Each revision is stored in a document which starts with the
                original document name and ends with the revision number, e.g. 
                <filename>hamlet.xml.35</filename>:</para>
            
            <screenshot>
                <graphic fileref="resources/versioning3.png"/>
            </screenshot>
            
            <para>Like the admin webapp, you can use XQuery functions to access the revision history
                or restore a given revision. eXist provides a single XQuery module for that. For
                example, to view the history of a resource, use the following query:</para>
            
            <programlisting language="xquery"><![CDATA[import module namespace v="http://exist-db.org/versioning";
v:history(doc("/db/shakespeare/plays/hamlet.xml"))]]></programlisting>
            
            <para>This should return an XML fragment like the following:</para>
            
            <programlisting language="xml"><![CDATA[<v:history>
    <v:document>/db/shakespeare/plays/hamlet.xml</v:document>
    <v:revisions>
        <v:revision rev="35">
            <v:date>2011-08-02T14:51:33.777+02:00</v:date>
            <v:user>admin</v:user>
        </v:revision>
        <v:revision rev="36">
            <v:date>2011-08-02T14:57:41.629+02:00</v:date>
            <v:user>admin</v:user>
        </v:revision>
    </v:revisions>
</v:history>]]></programlisting>
            
            <para>The most important function is <function>v:doc</function>, which is used
            to restore an arbitrary revision of a document on the fly. You can use this
            function similar to the standard <function>fn:doc</function> to query the revision.
            For example:</para>
            
            <programlisting language="xquery"><![CDATA[import module namespace v="http://exist-db.org/versioning";
v:doc(doc("/db/shakespeare/plays/hamlet.xml"), 35)//SPEECH[SPEAKER="HAMLET"]]]></programlisting>
            
            <para>This will restore revision 35 of <filename>hamlet.xml</filename> and then find
                all <sgmltag>SPEECH</sgmltag> elements with a <sgmltag>SPEAKER</sgmltag> called 
                "HAMLET". Please note that no indexes will be available to the query engine when
                processing a restore document.</para>
        </section>
        
        <section>
            <title>Detecting write conflicts</title>
            
            <para>To avoid that a user overwrites the changes made by another user, eXist
                needs to know upon which revision the user's changes are based. To make this
                possible, the versioning filter adds a number of metadata attributes to the
                root element of a document when it is serialized (e.g. to be opened in an editor).
                The inserted metadata attributes are all in a separate versioning namespace
                and will never be stored in the db. The following fragment shows the added
                attributes:</para>
            
            <programlisting language="xml"><![CDATA[<PLAY xmlns:v="http://exist-db.org/versioning" v:revision="36" v:key="<KEY>4" 
v:path="/db/shakespeare/plays/hamlet.xml">...</PLAY>]]></programlisting>
            
            <para>As described above ("Setup"), when eXist detects a potential write conflict,
                it cannot do more than to reject the update and raise an error. However, there's
                an XQuery function which can be used by clients to check if newer revisions exist.
                You pass it the revision number and the unique key as given in the versioning
                attributes of the document root element. If the function returns the empty sequence,
                no newer revisions exist in the database. Otherwise, the function returns the
                version documents of each newer revision.
            </para>
            
            <programlisting language="xquery"><![CDATA[import module namespace v="http://exist-db.org/versioning";
v:find-newer-revision(doc("/db/shakespeare/plays/hamlet.xml"), 36, "12343e4940b24")]]></programlisting>
            
            <para>Once you made sure that you really want to store the document and overwrite the
                other revisions, simply remove the version attributes from the root element.</para>
            
            <para>Future releases of the versioning extension may provide a more advanced
                conflict handling, including an automatic merge of non-conflicting changes.</para>
        </section>
        
    </chapter>
</book><commit_msg>[ignore] versioning review before version 1.4.1 release. (you should store before committing)<commit_after><?xml version="1.0" encoding="utf-8"?>
<!-- $Id$ -->
<book>
    <bookinfo>
        <graphic fileref="logo.jpg"/>
        <productname>Open Source Native XML Database</productname>
        <title>Versioning Module</title>
        <orgname>The eXist Project</orgname>
        <date>August 2011</date>
        <author>
            <firstname>Wolfgang</firstname>
            <surname>Meier</surname>
	    <affiliation>
	      <address format="linespecific"><email>wolfgang at exist-db.org</email></address>
	    </affiliation>
        </author>
        <author>
            <firstname>Leif-Jöran</firstname>
            <surname>Olsson</surname>
            <affiliation>
                <address format="linespecific"><email>ljo at exist-db.org</email></address>
            </affiliation>
	</author>
    </bookinfo>
    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="sidebar.xml"/>
    <chapter>
        <title>Versioning Extension</title>

        <abstract>
            <para>eXist-db provides a basic document versioning extension. The
                extension will track all changes to a document by storing a diff between the
                revisions. Older versions can be restored on the fly and even queried in memory.
                There's also basic support to detect and intercept conflicting writes.</para>

            <para>However, eXist-db has no control over the client. It does not know where a document
                update comes from and cannot directly communicate with the user. The versioning
                extension should be seen more like a toolbox than a complete solution. Advanced
                functionality (e.g. merging and conflict resolution) will require support from the
                end-user applications, which is outside of eXist-db's reach.</para>
            
            <para>The versioning extension has been created with human editors in mind, who will 
                typically be changing a document through an editor or some form-based frontend.
                It should work well with documents up to several megabytes in size. 
                However, the versioning will not track machine-generated node-level edits using XUpdate or XQuery 
                update extensions.</para>
        </abstract>

        <section>
            <title>Components</title>

            <para>The versioning extensions consist of the following components:</para>

            <variablelist>
                <varlistentry>
                    <term>VersioningTrigger</term>
                    <listitem>
                        <para>a trigger which has to be registered with a collection and
                        implements the core versioning functionality</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term>VersioningFilter</term>
                    <listitem>
                        <para>a serialization filter which adds special version attributes to every
                            serialized document. The attributes are used to detect conflicting writes.</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term><filename>versioning.xqm</filename></term>
                    <listitem>
                        <para>an XQuery module which provides a function library for end-user
                            applications, including functions like v:doc (used to
                            restore a given revision on the fly).</para>
                    </listitem>
                </varlistentry>
            </variablelist>
        </section>

        <section>
            <title>Setup</title>
            
            <para>Versioning can be enabled for any collection in the collection hierarchy. It
                is not necessary to use versioning for all collections. To enable versioning,
                a trigger has to be registered with the top-level collection. This is done
                through the same collection configuration files that are used for defining
                indexes.</para>
            
            <section>
                <title>Register the versioning trigger</title>
                
                <para>To enable version for a collection, you have to edit the
                collection's <filename>.xconf</filename> configuration file, which has to be 
                stored below the <filename>/db/system/config</filename> collection. As described
                in the <ulink url="indexing.xml">Configuring Indexes</ulink> document, the
                <filename>/db/system/config</filename> collection mirrors the hierarchical
                structure of the main collection tree.</para>
                
                <para>Within the collection's <filename>.xconf</filename>, you should register
                    the trigger class <classname>org.exist.versioning.VersioningTrigger</classname>
                    for the "store", "update" and "remove" events:</para>
                
                <example>
                    <title>Registering trigger in collection.xconf</title>
                    <programlisting language="xml"><![CDATA[<collection xmlns="http://exist-db.org/collection-config/1.0">
    <index>
        <fulltext default="none" attributes="no">
        </fulltext>
    </index>
    <triggers>
        <trigger event="store,remove,update"
            class="org.exist.versioning.VersioningTrigger">
            <parameter name="overwrite" value="yes"/>
        </trigger>
    </triggers>
</collection>]]></programlisting>
                </example>
                
                <para>If you store above document into <filename>/db/system/config/db/collection.xconf</filename>,
                it will enable versioning for the entire database.</para> 
                
                <note>
                    <para>Note that a <filename>collection.xconf</filename> at a lower level in the hierarchy will
                        <emphasis>overwrite</emphasis> any configuration on higher levels, including the
                        trigger definitions. Triggers are not inherited from ancestor configurations. If the 
                        new configuration doesn't define a trigger, the trigger map will be empy.</para>
                    
                    <para>When working with nested collection configurations,
                        you need to make sure that the trigger definitions are present in all
                        <filename>collection.xconf</filename> files.</para>
                </note>
                
                <para>VersioningTrigger accepts one
                    parameter, <option>overwrite</option>: if this is set to "no", the trigger will
                    check for potential write conflicts. For example, if two users opened the same
                    document and are editing it, it may happen that the first user saves changes
                    without the second user recognizing it. If the second user also made changes and those changes were stored it would just overwrite the modifications
                    already committed by the first user.</para>
                
                <para>The <option>overwrite="no"</option> setting
                    prevents this. However, eXist-db has no control over the client. It does not know
                    where the conflicting document came from. All it can do is to reject the write
                    attempt and raise an error. The error should then be handled by the client.
                    Right now there are no clients to support this. More work will be required
                    in this area. However, clients can already use the supplied XQuery functions to
                    check for write conflicts (see below).</para>
                
            </section>
            
            <section>
                <title>Enabling the serialization filter</title>
                
                <para>In order to detect conflicting writes, the versioning extension needs
                    to keep track of the base revision to which changes were applied. It does this
                    by inserting special metadata attributes into a document when it is retrieved
                    from the database. For this purpose, a <emphasis>custom filter</emphasis> has
                    to be registered with eXist-db's serializer. This is done in the <sgmltag>serializer</sgmltag>
                    section in the main configuration file, <filename>conf.xml</filename>. Add a
                    <sgmltag>custom-filter</sgmltag> child tag to the <sgmltag>serializer</sgmltag>
                    element and set its <option>class</option> attribute to 
                    <classname>org.exist.versioning.VersioningFilter</classname>:</para>
                
                <example>
                    <title>Registering the versioning filter in conf.xml</title>
                    <programlisting language="xml"><![CDATA[<serializer add-exist-id="none" compress-output="no" enable-xinclude="yes"
            enable-xsl="no" indent="yes" match-tagging-attributes="no" 
            match-tagging-elements="no">
    <custom-filter class="org.exist.versioning.VersioningFilter"/>
</serializer>]]></programlisting>
                </example>
                
                <para>eXist-db needs to be restarted for the versioning filter to become active.</para>
            </section>
            
        </section>
        
        <section>
            <title>Checking versions through the admin webapp</title>
            
            <para>The admin web application provides a simple way to see the revision history of a
                document and restore old versions. From the sidebar menu, select 
                <guimenuitem>Browse Collections</guimenuitem>. Within the collection view
                you should find a revision number next to each document which has been changed
                since versioning was enabled. Unchanged documents will not have a revision number:</para>
            
            <screenshot>
                <graphic fileref="resources/versioning1.png"/>
            </screenshot>
            
            <para>The revision number is the last number in each row ("34" in the screenshot above).
            You can click on the revision number to see the history of changes for this document:</para>
            
            <screenshot>
                <graphic fileref="resources/versioning2.png"/>
            </screenshot>
            
            <para>Clicking on <guimenuitem>Diff</guimenuitem> shows the differences between the selected
                and the previous revision. The way in which the differences are recorded is specific
                to eXist-db. It uses unique node ids to identify the reference node to which a change
                applies (the <option>ref</option> attributes in the XML shown above).</para>
            
            <para>The <guimenuitem>Restore</guimenuitem> link will restore the corresponding revision.
                This is done in memory by recursively applying all diffs since the base revision.</para>
        </section>
        
        <section>
            <title>Accessing the versioning information</title>
            
            <para>The versioning extension uses the collection <filename>/db/system/versions</filename>
                to store base revisions and diffs. The collection hierarchy below <filename>/db/system/versions</filename>
                again mirrors the main collection tree. For each versioned resource, you will find a
                document with suffix <filename>.base</filename>, which contains the base revision, i.e. the
                first version of the document. Each revision is stored in a document which starts with the
                original document name and ends with the revision number, e.g. 
                <filename>hamlet.xml.35</filename>:</para>
            
            <screenshot>
                <graphic fileref="resources/versioning3.png"/>
            </screenshot>
            
            <para>Like the admin webapp, you can use XQuery functions to access the revision history
                or restore a given revision. eXist-db provides a single XQuery module for that. For
                example, to view the history of a resource, use the following query:</para>
            
            <programlisting language="xquery"><![CDATA[import module namespace v="http://exist-db.org/versioning";
v:history(doc("/db/shakespeare/plays/hamlet.xml"))]]></programlisting>
            
            <para>This should return an XML fragment like the following:</para>
            
            <programlisting language="xml"><![CDATA[<v:history>
    <v:document>/db/shakespeare/plays/hamlet.xml</v:document>
    <v:revisions>
        <v:revision rev="35">
            <v:date>2011-08-02T14:51:33.777+02:00</v:date>
            <v:user>admin</v:user>
        </v:revision>
        <v:revision rev="36">
            <v:date>2011-08-02T14:57:41.629+02:00</v:date>
            <v:user>admin</v:user>
        </v:revision>
    </v:revisions>
</v:history>]]></programlisting>
            
            <para>The most important function is <function>v:doc</function>, which is used
            to restore an arbitrary revision of a document on the fly. You can use this
            function similar to the standard <function>fn:doc</function> to query the revision.
            For example:</para>
            
            <programlisting language="xquery"><![CDATA[import module namespace v="http://exist-db.org/versioning";
v:doc(doc("/db/shakespeare/plays/hamlet.xml"), 35)//SPEECH[SPEAKER="HAMLET"]]]></programlisting>
            
            <para>This will restore revision 35 of <filename>hamlet.xml</filename> and then find
                all <sgmltag>SPEECH</sgmltag> elements with a <sgmltag>SPEAKER</sgmltag> called 
                "HAMLET". Please note that no indexes will be available to the query engine when
                processing a restore document.</para>
        </section>
        
        <section>
            <title>Detecting write conflicts</title>
            
            <para>To avoid that a user overwrites the changes made by another user, eXist-db
                needs to know upon which revision the user's changes are based. To make this
                possible, the versioning filter adds a number of metadata attributes to the
                root element of a document when it is serialized (e.g. to be opened in an editor).
                The inserted metadata attributes are all in a separate versioning namespace
                and will never be stored in the db. The following fragment shows the added
                attributes:</para>
            
            <programlisting language="xml"><![CDATA[<PLAY xmlns:v="http://exist-db.org/versioning" v:revision="36" v:key="<KEY>" 
v:path="/db/shakespeare/plays/hamlet.xml">...</PLAY>]]></programlisting>
            
            <para>As described above ("Setup"), when eXist-db detects a potential write conflict,
                it cannot do more than to reject the update and raise an error. However, there's
                an XQuery function which can be used by clients to check if newer revisions exist.
                You pass it the revision number and the unique key as given in the versioning
                attributes of the document root element. If the function returns the empty sequence,
                no newer revisions exist in the database. Otherwise, the function returns the
                version documents of each newer revision.
            </para>
            
            <programlisting language="xquery"><![CDATA[import module namespace v="http://exist-db.org/versioning";
v:find-newer-revision(doc("/db/shakespeare/plays/hamlet.xml"), 36, "12343e4940b24")]]></programlisting>
            
            <para>Once you made sure that you really want to store the document and overwrite the
                other revisions, simply remove the version attributes from the root element.</para>
            
            <para>Future releases of the versioning extension may provide a more advanced
                conflict handling, including an automatic merge of non-conflicting changes.</para>
        </section>
        
    </chapter>
</book>
{'fraction_non_alphanumeric': 0.0793037917408223, 'fraction_numerical': 0.0053090404512684385, 'mean_word_length': 1.8946127652143543, 'pattern_counts': {'":': 0, '<': 593, '<?xml version=': 2, '>': 593, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 38}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4771517', 'n_tokens_mistral': 8035, 'n_tokens_neox': 7482, 'n_words': 2878}