<commit_before><?xml version="1.0" encoding="UTF-8"?>
<appendix version="5.0" xml:id="upgrade" xmlns="http://docbook.org/ns/docbook"
          xmlns:xlink="http://www.w3.org/1999/xlink"
          xmlns:ns5="http://www.w3.org/1998/Math/MathML"
          xmlns:ns4="http://www.w3.org/2000/svg"
          xmlns:ns3="http://www.w3.org/1999/xhtml"
          xmlns:ns="http://docbook.org/ns/docbook">
  <title>Upgrade Notes and Known Issues</title>

  <section xml:id="upgrade-known-issues">
    <title>Known Issues</title>

    <para>Because Spring Roo integrates a large number of other technologies,
    invariably some people using Roo may experience issues when using certain
    combinations of technologies together. This section aims to list such
    known issues in an effort to help you avoid experiencing any problems. If
    you are able to contribute further information, a solution or workaround
    to any of these known issues, we'd certainly appreciate hearing from you
    via the <link linkend="resources-forums">community forums</link>.</para>

    <itemizedlist>
      <listitem>
        <para><emphasis>JDK compatibility</emphasis>: Spring Roo has been
        tested with Sun, IBM, JRockit and Apache Harmony JVMs for Java 5 and
        Java 6. We do not formally support other JVMs or other versions of
        JVMs. We have also had an <link
        xlink:href="https://jira.springsource.org/browse/ROO-347">issue</link>
        reported with versions of Java 6 before 1.6.0_17 due to Java bug <link
        xlink:href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6506304">6506304</link>
        and therefore recommend you always use the latest released version of
        Java 6 for your platform. There is also a known issue with OpenJDK.
        You can read about our testing of different JDKs in issue <link
        xlink:href="https://jira.springsource.org/browse/ROO-106">ROO-106</link>.</para>
      </listitem>

      <listitem>
        <para><emphasis>Human language support</emphasis>: Pluralisation
        within Roo delegates to the <link
        xlink:href="https://inflector.dev.java.net/">Inflector</link> library.
        Due to some issues with Inflector, only English pluralisation is
        supported. If you wish to override the plural selected by Inflector
        (and in turn used by Roo), you can specify a particular plural for
        either a Java type or Java field by using the <code>@RooPlural</code>
        annotation. Longer term it would be nice if someone ported the
        Inflector code into the Roo pluralisation add-on so that we can fix
        these issues and support other languages. We are receptive to
        contributions from the community along these lines.</para>
      </listitem>

      <listitem>
        <para><emphasis>Shell wrapping</emphasis>: In certain cases typing a
        long command into the shell that wraps over a single line may prevent
        you from being able to backspace to the prior line. This is caused by
        the JLine library (not Roo). We expect to rewrite the shell at some
        future time and will likely stop using JLine at that point.</para>
      </listitem>

      <listitem>
        <para><emphasis>Hibernate issues</emphasis>: Hibernate is one of the
        JPA providers we test with, however, Hibernate has issues with
        <code>--mappedSuperclass</code> as detailed in <link
        xlink:href="https://jira.springsource.org/browse/ROO-292">ROO-292</link>
        and <link
        xlink:href="https://jira.springsource.org/browse/ROO-747">ROO-747</link>.
        We recommend you do not use <code>--mappedSuperclass</code> in
        combination with Hibernate. We have found OpenJPA works reliably in
        all cases, so you might want to consider switching to OpenJPA if you
        are seriously impacted by this issue (the "<link
        linkend="command-index-jpa-setup">jpa setup</link>" command can be
        used multiple times, which is useful for experimentally switching
        between different JPA providers).</para>
      </listitem>

      <listitem>
        <para><emphasis>Integration testing limitations</emphasis>: The data
        on demand mechanism (which is used for integration tests) has limited
        JSR 303 (Bean Validator) compatibility. Roo supports fields using
        @NotNull, @Past and @Future, @Size, @Min, and @Max. No other validator
        annotations are formally supported, although many will work. To use
        other validator annotations, you may need to edit your
        DataOnDemand.java file and add a manual
        <code>getNewTransientEntity(int)</code> method. Refer to a generated
        <code>*_Roo_DataOnDemand.aj</code> file for an example. Alternately,
        do not use the integration test functionality in Roo unless you have
        relatively simple validation constraints or you are willing to provide
        this data on demand method.</para>
      </listitem>

      <listitem>
        <para><emphasis>Tomcat 5.5</emphasis>: Tomcat 5.5 can not be supported
        by the scaffolded Spring MVC Web UI. Tomcat 5.5 does not support the
        JSP 2.1 API. Roo makes extensive use of the JSP 2.1 API in the
        scaffolded Web UI (specifically expression language features).
        Furthermore, the JSP 2.0 API does not support JDK 5 enums (a feature
        that Roo would need). See <link
        xlink:href="https://jira.springsource.org/browse/ROO-680">ROO-680</link>
        for more details. The following <link
        xlink:href="http://forum.springsource.org/showpost.php?s=10e2df3cc266c9a85f8d473716d9b0c3&amp;p=287652&amp;postcount=15">
        forum post</link> offers a workaround for the JSP 2.1 incompatibility
        issue. Please be aware that this has not been tested by the Roo team
        and Tomcat 5.5 does officially support the JSP 2.0 API.</para>
      </listitem>

      <listitem>
        <para>Applications with a scaffolded Spring MVC UI are currently not
        deployable to Google App Engine due to incompatibilities in the JSP
        support and JSTL. See <link
        xlink:href="https://jira.springframework.org/browse/ROO-1006">ROO-1006</link>
        for details.</para>
      </listitem>

      <listitem>
        <para>Applications with a scaffolded GWT UI require a manual
        adjustment in
        <code>src/main/webapp/WEB-INF/spring/webmvc-config.xml</code> (this
        will not be required when using Spring Framework 3.0.5+):</para>

        <programlisting>&lt;mvc:default-servlet-handler <emphasis role="bold">default-servlet-name="_ah_default"</emphasis> /&gt;</programlisting>
      </listitem>
    </itemizedlist>
  </section>

  <section>
    <title>Version Numbering Approach</title>

    <para>Spring Roo observes version number standards based on the <link
    xlink:href="http://apr.apache.org/versioning.html">Apache Portable Runtime
    (APR) versioning guidelines</link> as well as the <link
    xlink:href="http://www.osgi.org">OSGi</link> specifications. In summary
    this means all Roo releases adopt the format of MAJOR.MINOR.PATCH.TYPE.
    Each segment is separated by a period without any spaces. The
    MAJOR.MINOR.PATCH are always integer numbers, and TYPE is an alphanumeric
    value. For example, Roo 1.0.3.M1 means major version 1, minor version 0,
    patch number 3 and release type M1.</para>

    <para>You can always rely on the natural sort order of the version numbers
    to arrive at the latest available version. For example, 1.0.4.RELEASE is
    more recent than 1.0.4.RC2. This is because "RELEASE" sorts alphabetically
    lower than "RC2". The TYPE segment can generally be broken into two
    further undelimited portions, being the release type and a numeric
    increment. For example, RC1 means release candidate 1 and RC4 means
    release candidate 4. One exception to this is RELEASE means the final
    general availability of that release. Other common release types include
    "A" for alpha and "M" for milestone.</para>

    <para>We make no guarantees regarding the compatibility of any release
    that has a TYPE other than "RELEASE". However, for "RELEASE" releases we
    aim to ensure you can use a given "RELEASE" with any other "RELEASE" which
    has the same MAJOR.MINOR version number. As such you should be able to
    switch from 1.0.4.RELEASE to 1.0.9.RELEASE without any changes. However,
    you might have trouble with 1.0.4.RELEASE to 1.0.9.RC1 as RC1 is a
    work-in-progress and we may not have identified all regression issues.
    Obviously this version portability is only our objective, and sometimes we
    need to make exceptions or may inadvertently overlook an issue. We
    appreciate you logging a <link linkend="resources-issue-tracking">bug
    report</link> if you identify a version regression that violates the
    conventions expressed in this section, so that at least we can confirm it
    and either attempt to remedy it on the next release of that MAJOR.MINOR
    version range or bring it to people's attention in the other sections of
    this appendix.</para>

    <para>When upgrading you should review the <link
    linkend="resources-issue-tracking">issue tracker</link> for what has
    changed since the last version. Because most releases include a large
    number of issues in the release notes, we attempt to highlight any major
    issues that may require your attention in the sections below. These notes
    are not all-encompassing but simply pointers to the main upgrade-related
    issues that most people should be aware of. They are also written assuming
    you are maintaining currency with the latest public releases of Spring Roo
    and therefore the changes you may need to make to your project are
    cumulative.</para>
  </section>

  <section xml:id="upgrade-any">
    <title>Upgrading To Any New Release</title>

    <para>Before upgrading any project to the next release of Spring Roo, you
    should always:</para>

    <itemizedlist>
      <listitem>
        <para>Run the <link linkend="command-index-backup">backup</link>
        command using your currently-installed (i.e. existing) version of
        Spring Roo. This will help create a ZIP of your project, which may
        help if you need to revert. Don't install the new version of Roo until
        you've firstly completed this backup. Naturally you can skip this step
        if you have an alternate backup technique and have confidence in
        it.</para>
      </listitem>

      <listitem>
        <para>Edit your project's <code>pom.xml</code> and verify the Spring
        Roo annotations JAR matches the new Roo release you are installing.
        Spring Roo 1.1.0.M3 and above will do this automatically on your
        behalf when you load it on an existing project.</para>
      </listitem>

      <listitem>
        <para>Edit your project's <code>pom.xml</code> and verify that major
        libraries match the new versions that are now used by Roo. The
        simplest approach to doing this is to create a new directory and use
        "<code>roo script clinic.roo</code>" and then <code>diff</code> your
        existing <code>pom.xml</code> against the newly-created Petclinic
        <code>pom.xml</code>.</para>
      </listitem>

      <listitem>
        <para>After modifying the <code>pom.xml</code> as described above, you
        will need to update your Eclipse <code>.classpath</code> file. The
        simplest way to achieve this is to use <code>mvn eclipse:clean
        eclipse:eclipse</code> from the command prompt, or use the <link
        linkend="command-index-perform-eclipse">perform eclipse</link> command
        at the <code>roo&gt;</code> shell prompt. You can skip this step if
        you use m2eclipse, as would be the case for any SpringSource Tool
        Suite user.</para>
      </listitem>
    </itemizedlist>

    <para>Please refer to the specific upgrade section of this appendix for
    further instructions concerning upgrading to a particular version of
    Roo.</para>

    <para>If you experience any difficulty with upgrading your projects,
    please use the <link linkend="resources-forums">community support
    forum</link> for assistance.</para>
  </section>

  <section>
    <title>Upgrading to 1.2.0.M1</title>

    <para>The main changes you need to be aware of when upgrading from Spring
    Roo 1.1.5.RELEASE to Spring Roo 1.2.0.M1 are as follows:</para>

    <itemizedlist>
      <listitem>
        <para>The presence of @RooWebScaffold does not automatically trigger
        Spring MVC JSON integration any more. The exposeJson attribute in this
        annotation has been depricated and will be removed for subsequent
        releases. To create Spring MVC JSON integration please see the <link
        linkend="base-json">JSON chapter</link> or simply use the <link
        linkend="command-index-web-mvc-json-all">web mvc json all</link>
        command.</para>
      </listitem>

      <listitem>
        <para>The presence of @RooWebScaffold does not automatically trigger
        Spring MVC Finder integration any more. The exposeFinders attribute in
        this annotation has been depricated and will be removed for subsequent
        releases. To create Spring MVC Finder integration please see <link
        linkend="base-web">MVC chapter</link> or simply use the <link
        linkend="command-index-web-mvc-finder-all">web mvc finder all</link>
        command.</para>
      </listitem>
    </itemizedlist>
  </section>

  <section xml:id="upgrade-1.1.3.RELEASE">
    <title>Upgrading to 1.1.3.RELEASE</title>

    <para>The main changes you need to be aware of when upgrading from Spring
    Roo 1.1.2.RELEASE to Spring Roo 1.1.3.RELEASE are as follows:</para>

    <itemizedlist>
      <listitem><commit_msg>ROO-2736: Release Spring Roo 1.2.0.M1 - added 1.2.0.M1 GWT upgrade instructions<commit_after><?xml version="1.0" encoding="UTF-8"?>
<appendix version="5.0" xml:id="upgrade" xmlns="http://docbook.org/ns/docbook"
          xmlns:xlink="http://www.w3.org/1999/xlink"
          xmlns:ns5="http://www.w3.org/1998/Math/MathML"
          xmlns:ns4="http://www.w3.org/2000/svg"
          xmlns:ns3="http://www.w3.org/1999/xhtml"
          xmlns:ns="http://docbook.org/ns/docbook">
  <title>Upgrade Notes and Known Issues</title>

  <section xml:id="upgrade-known-issues">
    <title>Known Issues</title>

    <para>Because Spring Roo integrates a large number of other technologies,
    invariably some people using Roo may experience issues when using certain
    combinations of technologies together. This section aims to list such
    known issues in an effort to help you avoid experiencing any problems. If
    you are able to contribute further information, a solution or workaround
    to any of these known issues, we'd certainly appreciate hearing from you
    via the <link linkend="resources-forums">community forums</link>.</para>

    <itemizedlist>
      <listitem>
        <para><emphasis>JDK compatibility</emphasis>: Spring Roo has been
        tested with Sun, IBM, JRockit and Apache Harmony JVMs for Java 5 and
        Java 6. We do not formally support other JVMs or other versions of
        JVMs. We have also had an <link
        xlink:href="https://jira.springsource.org/browse/ROO-347">issue</link>
        reported with versions of Java 6 before 1.6.0_17 due to Java bug <link
        xlink:href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6506304">6506304</link>
        and therefore recommend you always use the latest released version of
        Java 6 for your platform. There is also a known issue with OpenJDK.
        You can read about our testing of different JDKs in issue <link
        xlink:href="https://jira.springsource.org/browse/ROO-106">ROO-106</link>.</para>
      </listitem>

      <listitem>
        <para><emphasis>Human language support</emphasis>: Pluralisation
        within Roo delegates to the <link
        xlink:href="https://inflector.dev.java.net/">Inflector</link> library.
        Due to some issues with Inflector, only English pluralisation is
        supported. If you wish to override the plural selected by Inflector
        (and in turn used by Roo), you can specify a particular plural for
        either a Java type or Java field by using the <code>@RooPlural</code>
        annotation. Longer term it would be nice if someone ported the
        Inflector code into the Roo pluralisation add-on so that we can fix
        these issues and support other languages. We are receptive to
        contributions from the community along these lines.</para>
      </listitem>

      <listitem>
        <para><emphasis>Shell wrapping</emphasis>: In certain cases typing a
        long command into the shell that wraps over a single line may prevent
        you from being able to backspace to the prior line. This is caused by
        the JLine library (not Roo). We expect to rewrite the shell at some
        future time and will likely stop using JLine at that point.</para>
      </listitem>

      <listitem>
        <para><emphasis>Hibernate issues</emphasis>: Hibernate is one of the
        JPA providers we test with, however, Hibernate has issues with
        <code>--mappedSuperclass</code> as detailed in <link
        xlink:href="https://jira.springsource.org/browse/ROO-292">ROO-292</link>
        and <link
        xlink:href="https://jira.springsource.org/browse/ROO-747">ROO-747</link>.
        We recommend you do not use <code>--mappedSuperclass</code> in
        combination with Hibernate. We have found OpenJPA works reliably in
        all cases, so you might want to consider switching to OpenJPA if you
        are seriously impacted by this issue (the "<link
        linkend="command-index-jpa-setup">jpa setup</link>" command can be
        used multiple times, which is useful for experimentally switching
        between different JPA providers).</para>
      </listitem>

      <listitem>
        <para><emphasis>Integration testing limitations</emphasis>: The data
        on demand mechanism (which is used for integration tests) has limited
        JSR 303 (Bean Validator) compatibility. Roo supports fields using
        @NotNull, @Past and @Future, @Size, @Min, and @Max. No other validator
        annotations are formally supported, although many will work. To use
        other validator annotations, you may need to edit your
        DataOnDemand.java file and add a manual
        <code>getNewTransientEntity(int)</code> method. Refer to a generated
        <code>*_Roo_DataOnDemand.aj</code> file for an example. Alternately,
        do not use the integration test functionality in Roo unless you have
        relatively simple validation constraints or you are willing to provide
        this data on demand method.</para>
      </listitem>

      <listitem>
        <para><emphasis>Tomcat 5.5</emphasis>: Tomcat 5.5 can not be supported
        by the scaffolded Spring MVC Web UI. Tomcat 5.5 does not support the
        JSP 2.1 API. Roo makes extensive use of the JSP 2.1 API in the
        scaffolded Web UI (specifically expression language features).
        Furthermore, the JSP 2.0 API does not support JDK 5 enums (a feature
        that Roo would need). See <link
        xlink:href="https://jira.springsource.org/browse/ROO-680">ROO-680</link>
        for more details. The following <link
        xlink:href="http://forum.springsource.org/showpost.php?s=10e2df3cc266c9a85f8d473716d9b0c3&amp;p=287652&amp;postcount=15">
        forum post</link> offers a workaround for the JSP 2.1 incompatibility
        issue. Please be aware that this has not been tested by the Roo team
        and Tomcat 5.5 does officially support the JSP 2.0 API.</para>
      </listitem>

      <listitem>
        <para>Applications with a scaffolded Spring MVC UI are currently not
        deployable to Google App Engine due to incompatibilities in the JSP
        support and JSTL. See <link
        xlink:href="https://jira.springframework.org/browse/ROO-1006">ROO-1006</link>
        for details.</para>
      </listitem>

      <listitem>
        <para>Applications with a scaffolded GWT UI require a manual
        adjustment in
        <code>src/main/webapp/WEB-INF/spring/webmvc-config.xml</code> (this
        will not be required when using Spring Framework 3.0.5+):</para>

        <programlisting>&lt;mvc:default-servlet-handler <emphasis role="bold">default-servlet-name="_ah_default"</emphasis> /&gt;</programlisting>
      </listitem>
    </itemizedlist>
  </section>

  <section>
    <title>Version Numbering Approach</title>

    <para>Spring Roo observes version number standards based on the <link
    xlink:href="http://apr.apache.org/versioning.html">Apache Portable Runtime
    (APR) versioning guidelines</link> as well as the <link
    xlink:href="http://www.osgi.org">OSGi</link> specifications. In summary
    this means all Roo releases adopt the format of MAJOR.MINOR.PATCH.TYPE.
    Each segment is separated by a period without any spaces. The
    MAJOR.MINOR.PATCH are always integer numbers, and TYPE is an alphanumeric
    value. For example, Roo 1.0.3.M1 means major version 1, minor version 0,
    patch number 3 and release type M1.</para>

    <para>You can always rely on the natural sort order of the version numbers
    to arrive at the latest available version. For example, 1.0.4.RELEASE is
    more recent than 1.0.4.RC2. This is because "RELEASE" sorts alphabetically
    lower than "RC2". The TYPE segment can generally be broken into two
    further undelimited portions, being the release type and a numeric
    increment. For example, RC1 means release candidate 1 and RC4 means
    release candidate 4. One exception to this is RELEASE means the final
    general availability of that release. Other common release types include
    "A" for alpha and "M" for milestone.</para>

    <para>We make no guarantees regarding the compatibility of any release
    that has a TYPE other than "RELEASE". However, for "RELEASE" releases we
    aim to ensure you can use a given "RELEASE" with any other "RELEASE" which
    has the same MAJOR.MINOR version number. As such you should be able to
    switch from 1.0.4.RELEASE to 1.0.9.RELEASE without any changes. However,
    you might have trouble with 1.0.4.RELEASE to 1.0.9.RC1 as RC1 is a
    work-in-progress and we may not have identified all regression issues.
    Obviously this version portability is only our objective, and sometimes we
    need to make exceptions or may inadvertently overlook an issue. We
    appreciate you logging a <link linkend="resources-issue-tracking">bug
    report</link> if you identify a version regression that violates the
    conventions expressed in this section, so that at least we can confirm it
    and either attempt to remedy it on the next release of that MAJOR.MINOR
    version range or bring it to people's attention in the other sections of
    this appendix.</para>

    <para>When upgrading you should review the <link
    linkend="resources-issue-tracking">issue tracker</link> for what has
    changed since the last version. Because most releases include a large
    number of issues in the release notes, we attempt to highlight any major
    issues that may require your attention in the sections below. These notes
    are not all-encompassing but simply pointers to the main upgrade-related
    issues that most people should be aware of. They are also written assuming
    you are maintaining currency with the latest public releases of Spring Roo
    and therefore the changes you may need to make to your project are
    cumulative.</para>
  </section>

  <section xml:id="upgrade-any">
    <title>Upgrading To Any New Release</title>

    <para>Before upgrading any project to the next release of Spring Roo, you
    should always:</para>

    <itemizedlist>
      <listitem>
        <para>Run the <link linkend="command-index-backup">backup</link>
        command using your currently-installed (i.e. existing) version of
        Spring Roo. This will help create a ZIP of your project, which may
        help if you need to revert. Don't install the new version of Roo until
        you've firstly completed this backup. Naturally you can skip this step
        if you have an alternate backup technique and have confidence in
        it.</para>
      </listitem>

      <listitem>
        <para>Edit your project's <code>pom.xml</code> and verify the Spring
        Roo annotations JAR matches the new Roo release you are installing.
        Spring Roo 1.1.0.M3 and above will do this automatically on your
        behalf when you load it on an existing project.</para>
      </listitem>

      <listitem>
        <para>Edit your project's <code>pom.xml</code> and verify that major
        libraries match the new versions that are now used by Roo. The
        simplest approach to doing this is to create a new directory and use
        "<code>roo script clinic.roo</code>" and then <code>diff</code> your
        existing <code>pom.xml</code> against the newly-created Petclinic
        <code>pom.xml</code>.</para>
      </listitem>

      <listitem>
        <para>After modifying the <code>pom.xml</code> as described above, you
        will need to update your Eclipse <code>.classpath</code> file. The
        simplest way to achieve this is to use <code>mvn eclipse:clean
        eclipse:eclipse</code> from the command prompt, or use the <link
        linkend="command-index-perform-eclipse">perform eclipse</link> command
        at the <code>roo&gt;</code> shell prompt. You can skip this step if
        you use m2eclipse, as would be the case for any SpringSource Tool
        Suite user.</para>
      </listitem>
    </itemizedlist>

    <para>Please refer to the specific upgrade section of this appendix for
    further instructions concerning upgrading to a particular version of
    Roo.</para>

    <para>If you experience any difficulty with upgrading your projects,
    please use the <link linkend="resources-forums">community support
    forum</link> for assistance.</para>
  </section>

  <section>
    <title>Upgrading to 1.2.0.M1</title>

    <para>The main changes you need to be aware of when upgrading from Spring
    Roo 1.1.5.RELEASE to Spring Roo 1.2.0.M1 are as follows:</para>

    <itemizedlist>
      <listitem>
        <para>The presence of @RooWebScaffold does not automatically trigger
        Spring MVC JSON integration any more. The exposeJson attribute in this
        annotation has been depricated and will be removed for subsequent
        releases. To create Spring MVC JSON integration please see the <link
        linkend="base-json">JSON chapter</link> or simply use the <link
        linkend="command-index-web-mvc-json-all">web mvc json all</link>
        command.</para>
      </listitem>

      <listitem>
        <para>The presence of @RooWebScaffold does not automatically trigger
        Spring MVC Finder integration any more. The exposeFinders attribute in
        this annotation has been depricated and will be removed for subsequent
        releases. To create Spring MVC Finder integration please see <link
        linkend="base-web">MVC chapter</link> or simply use the <link
        linkend="command-index-web-mvc-finder-all">web mvc finder all</link>
        command.</para>
      </listitem>

		<listitem>
        <para>To update a Roo GWT project please run
		<link linkend="command-index-web-gwt-setup">web gwt setup</link>
		</para>
      </listitem>
    </itemizedlist>
  </section>

  <section xml:id="upgrade-1.1.3.RELEASE">
    <title>Upgrading to 1.1.3.RELEASE</title>

    <para>The main changes you need to be aware of when upgrading from Spring
    Roo 1.1.2.RELEASE to Spring Roo 1.1.3.RELEASE are as follows:</para>

    <itemizedlist>
      <listitem>
{'fraction_non_alphanumeric': 0.07925432946901388, 'fraction_numerical': 0.014813224559896952, 'mean_word_length': 2.992714285714286, 'pattern_counts': {'":': 0, '<': 407, '<?xml version=': 2, '>': 407, 'https://': 14, 'lorem ipsum': 0, 'www.': 10, 'xml': 34}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7589977', 'n_tokens_mistral': 8068, 'n_tokens_neox': 7555, 'n_words': 3181}