<commit_before>//
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.
//
==== DBMS

[CAUTION]
The changes reported below to support different DBMSes are not complete files, but only show the lines that need to be
updated.

===== PostgreSQL

[NOTE]
Apache Syncope {docVersion} is verified with PostgreSQL server >= 10.3 and JDBC driver >= {postgresqlJDBC}.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
quartz.sql=tables_postgres.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=org.postgresql.Driver
Master.url=jdbc:postgresql://localhost:5432/syncope
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.PostgresDictionary
Master.orm=META-INF/spring-orm.xml
....

[WARNING]
This assumes that you have a PostgreSQL instance running on localhost, listening on its default port 5432 with a
database `syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/fit/core-reference/src/main/resources/postgres/views.xml[views.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/tree/master/fit/core-reference/src/main/resources/postgres/views.xml[views.xml^]
endif::[]
and save it under `core/src/main/resources/`.

===== PostgreSQL (JSONB)

[NOTE]
With the configurations reported below, Apache Syncope will leverage the
https://www.postgresql.org/docs/current/datatype-json.html[JSONB^] column type for attribute storage.

[NOTE]
Apache Syncope {docVersion} is verified with PostgreSQL server >= 10.3 and JDBC driver >= {postgresqlJDBC}.

Add the following dependency to `core/pom.xml`:

[source,xml,subs="verbatim,attributes"]
----
<dependency>
  <groupId>org.apache.syncope.core</groupId>
  <artifactId>syncope-core-persistence-jpa-json</artifactId>
  <version>${syncope.version}</version>
</dependency>
----

In `persistence.properties`, replace as follows:

....
entity.factory=org.apache.syncope.core.persistence.jpa.entity.PGJPAJSONEntityFactory
plainSchema.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONPlainSchemaDAO
plainAttr.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONPlainAttrDAO
plainAttrValue.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONPlainAttrValueDAO
any.search.dao=org.apache.syncope.core.persistence.jpa.dao.PGJPAJSONAnySearchDAO
user.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONUserDAO
group.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONGroupDAO
anyObject.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONAnyObjectDAO
conf.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONConfDAO
....

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
quartz.sql=tables_postgres.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=org.postgresql.Driver
Master.url=jdbc:postgresql://localhost:5432/syncope?stringtype=unspecified
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.PostgresDictionary
Master.orm=META-INF/spring-orm-pgjsonb.xml
....

[WARNING]
This assumes that you have a PostgreSQL instance running on localhost, listening on its default port 5432 with a
database `syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/main/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/main/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
,
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/main/resources/pgjsonb/indexes.xml[indexes.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/main/resources/pgjsonb/indexes.xml[indexes.xml^]
endif::[]
and
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/main/resources/pgjsonb/views.xml[views.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/main/resources/pgjsonb/views.xml[views.xml^]
endif::[]
then save all under `core/src/main/resources/`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/test/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/test/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
and save it under `core/src/test/resources/`.

===== MySQL

[NOTE]
Apache Syncope {docVersion} is verified with MySQL server >= 5.7 and JDBC driver >= {mysqlJDBC}.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
quartz.sql=tables_mysql_innodb.sql
....

[WARNING]
This assumes that the InnoDB engine is enabled in your MySQL instance - if this is not the case, then change the value
for `quartz.sql` to `tables_mysql.sql`.

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=com.mysql.jdbc.Driver
Master.url=jdbc:mysql://localhost:3306/syncope?characterEncoding=UTF-8&relaxAutoCommit=true
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.MySQLDictionary(blobTypeName=LONGBLOB,timestampTypeName=DATETIME(3))
Master.orm=META-INF/spring-orm.xml
Master.audit.sql=audit_mysql_innodb.sql
....

[CAUTION]
It is important to set the collation to `utf8_general_ci` after creation of `syncope` database.

[WARNING]
This assumes that the InnoDB engine is enabled in your MySQL instance - if this is not the case, then change the value
for `Master.audit` to `audit.sql`.

[WARNING]
This assumes that you have a MySQL instance running on localhost, listening on its default port 3306 with a database
`syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

===== MariaDB

[NOTE]
Apache Syncope {docVersion} is verified with MariaDB server >= 10.3.7 and JDBC driver >= {mariadbJDBC}.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
quartz.sql=tables_mariadb.sql
....

[WARNING]
This assumes that the InnoDB engine is enabled in your MariaDB instance - if this is not the case, then change the value
for `quartz.sql` to `tables_mariadb.sql`.

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=org.mariadb.jdbc.Driver
Master.url=jdbc:mariadb://localhost:3306/syncope?characterEncoding=UTF-8
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.MariaDBDictionary(blobTypeName=LONGBLOB,timestampTypeName=DATETIME(3))
Master.orm=META-INF/spring-orm.xml
....

[CAUTION]
It is important to set the collation to `utf8_general_ci` after creation of `syncope` database.

[WARNING]
This assumes that you have a MariaDB instance running on localhost, listening on its default port 3306 with a database
`syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

===== Oracle Database

[NOTE]
Apache Syncope {docVersion} is verified with Oracle database >= 11g and JDBC driver >= ojdbc8 172.16.58.3.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
quartz.sql=tables_oracle.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=oracle.jdbc.OracleDriver
Master.url=jdbc:oracle:thin:@localhost:1521:orcl
Master.schema=SYNCOPE
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.OracleDictionary
Master.orm=META-INF/spring-orm-oracle.xml
Master.audit.sql=audit_oracle.sql
....

[WARNING]
This assumes that you have an Oracle instance running on localhost, listening on its default port 1521 with a database
`syncope` under tablespace `SYNCOPE`, fully accessible by user `syncope` with password `<PASSWORD>`.

===== MS SQL Server

[NOTE]
Apache Syncope {docVersion} is verified with MS SQL server >= 2017 and JDBC driver >= {sqlserverJDBC}8.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.MSSQLDelegate
quartz.sql=tables_sqlServer.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
Master.url=jdbc:sqlserver://localhost:1344;database=syncope;selectMethod=cursor;sendStringParametersAsUnicode=false
Master.schema=dbo
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.SQLServerDictionary
Master.orm=META-INF/spring-orm-sqlserver.xml
Master.audit.sql=audit_sqlserver.sql
....

[WARNING]
This assumes that you have a MS SQL Server instance running on localhost, listening on its default port 1344 with a
database `syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/fit/core-reference/src/main/resources/sqlserver/views.xml[views.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/tree/master/fit/core-reference/src/main/resources/sqlserver/views.xml[views.xml^]
endif::[]
and save it under `core/src/main/resources/`.
<commit_msg>Little clarification about PGJSONB setup<commit_after>//
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.
//
==== DBMS

[CAUTION]
The changes reported below to support different DBMSes are not complete files, but only show the lines that need to be
updated.

===== PostgreSQL

[NOTE]
Apache Syncope {docVersion} is verified with PostgreSQL server >= 10.3 and JDBC driver >= {postgresqlJDBC}.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
quartz.sql=tables_postgres.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=org.postgresql.Driver
Master.url=jdbc:postgresql://localhost:5432/syncope
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.PostgresDictionary
Master.orm=META-INF/spring-orm.xml
....

[WARNING]
This assumes that you have a PostgreSQL instance running on localhost, listening on its default port 5432 with a
database `syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/fit/core-reference/src/main/resources/postgres/views.xml[views.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/tree/master/fit/core-reference/src/main/resources/postgres/views.xml[views.xml^]
endif::[]
and save it under `core/src/main/resources/`.

===== PostgreSQL (JSONB)

[NOTE]
With the configurations reported below, Apache Syncope will leverage the
https://www.postgresql.org/docs/current/datatype-json.html[JSONB^] column type for attribute storage.

[NOTE]
Apache Syncope {docVersion} is verified with PostgreSQL server >= 10.3 and JDBC driver >= {postgresqlJDBC}.

Add the following dependency to `core/pom.xml`:

[source,xml,subs="verbatim,attributes"]
----
<dependency>
  <groupId>org.apache.syncope.core</groupId>
  <artifactId>syncope-core-persistence-jpa-json</artifactId>
  <version>${syncope.version}</version>
</dependency>
----

In `persistence.properties`, replace as follows:

....
entity.factory=org.apache.syncope.core.persistence.jpa.entity.PGJPAJSONEntityFactory
plainSchema.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONPlainSchemaDAO
plainAttr.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONPlainAttrDAO
plainAttrValue.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONPlainAttrValueDAO
any.search.dao=org.apache.syncope.core.persistence.jpa.dao.PGJPAJSONAnySearchDAO
user.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONUserDAO
group.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONGroupDAO
anyObject.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONAnyObjectDAO
conf.dao=org.apache.syncope.core.persistence.jpa.dao.JPAJSONConfDAO
....

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
quartz.sql=tables_postgres.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=org.postgresql.Driver
Master.url=jdbc:postgresql://localhost:5432/syncope?stringtype=unspecified
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.PostgresDictionary
Master.orm=META-INF/spring-orm-pgjsonb.xml
....

[WARNING]
This assumes that you have a PostgreSQL instance running on localhost, listening on its default port 5432 with a
database `syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/main/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/main/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
and save it under `core/src/main/resources/domains/`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/main/resources/pgjsonb/indexes.xml[indexes.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/main/resources/pgjsonb/indexes.xml[indexes.xml^]
endif::[]
and
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/main/resources/pgjsonb/views.xml[views.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/main/resources/pgjsonb/views.xml[views.xml^]
endif::[]
then save both under `core/src/main/resources/`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/core/persistence-jpa-json/src/test/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/blob/master/core/persistence-jpa-json/src/test/resources/pgjsonb/domains/MasterContent.xml[MasterContent.xml^]
endif::[]
and save it under `core/src/test/resources/domains/`.

===== MySQL

[NOTE]
Apache Syncope {docVersion} is verified with MySQL server >= 5.7 and JDBC driver >= {mysqlJDBC}.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
quartz.sql=tables_mysql_innodb.sql
....

[WARNING]
This assumes that the InnoDB engine is enabled in your MySQL instance - if this is not the case, then change the value
for `quartz.sql` to `tables_mysql.sql`.

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=com.mysql.jdbc.Driver
Master.url=jdbc:mysql://localhost:3306/syncope?characterEncoding=UTF-8&relaxAutoCommit=true
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.MySQLDictionary(blobTypeName=LONGBLOB,timestampTypeName=DATETIME(3))
Master.orm=META-INF/spring-orm.xml
Master.audit.sql=audit_mysql_innodb.sql
....

[CAUTION]
It is important to set the collation to `utf8_general_ci` after creation of `syncope` database.

[WARNING]
This assumes that the InnoDB engine is enabled in your MySQL instance - if this is not the case, then change the value
for `Master.audit` to `audit.sql`.

[WARNING]
This assumes that you have a MySQL instance running on localhost, listening on its default port 3306 with a database
`syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

===== MariaDB

[NOTE]
Apache Syncope {docVersion} is verified with MariaDB server >= 10.3.7 and JDBC driver >= {mariadbJDBC}.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
quartz.sql=tables_mariadb.sql
....

[WARNING]
This assumes that the InnoDB engine is enabled in your MariaDB instance - if this is not the case, then change the value
for `quartz.sql` to `tables_mariadb.sql`.

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=org.mariadb.jdbc.Driver
Master.url=jdbc:mariadb://localhost:3306/syncope?characterEncoding=UTF-8
Master.schema=
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.MariaDBDictionary(blobTypeName=LONGBLOB,timestampTypeName=DATETIME(3))
Master.orm=META-INF/spring-orm.xml
....

[CAUTION]
It is important to set the collation to `utf8_general_ci` after creation of `syncope` database.

[WARNING]
This assumes that you have a MariaDB instance running on localhost, listening on its default port 3306 with a database
`syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

===== Oracle Database

[NOTE]
Apache Syncope {docVersion} is verified with Oracle database >= 11g and JDBC driver >= ojdbc8 172.16.58.3.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
quartz.sql=tables_oracle.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=oracle.jdbc.OracleDriver
Master.url=jdbc:oracle:thin:@localhost:1521:orcl
Master.schema=SYNCOPE
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.OracleDictionary
Master.orm=META-INF/spring-orm-oracle.xml
Master.audit.sql=audit_oracle.sql
....

[WARNING]
This assumes that you have an Oracle instance running on localhost, listening on its default port 1521 with a database
`syncope` under tablespace `SYNCOPE`, fully accessible by user `syncope` with password `<PASSWORD>`.

===== MS SQL Server

[NOTE]
Apache Syncope {docVersion} is verified with MS SQL server >= 2017 and JDBC driver >= {sqlserverJDBC}8.

In `provisioning.properties`, replace as follows:

....
quartz.jobstore=org.quartz.impl.jdbcjobstore.MSSQLDelegate
quartz.sql=tables_sqlServer.sql
....

In `domains/Master.properties` (for the `Master` domain), replace as follows:

....
Master.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
Master.url=jdbc:sqlserver://localhost:1344;database=syncope;selectMethod=cursor;sendStringParametersAsUnicode=false
Master.schema=dbo
Master.username=syncope
Master.password=<PASSWORD>
Master.databasePlatform=org.apache.openjpa.jdbc.sql.SQLServerDictionary
Master.orm=META-INF/spring-orm-sqlserver.xml
Master.audit.sql=audit_sqlserver.sql
....

[WARNING]
This assumes that you have a MS SQL Server instance running on localhost, listening on its default port 1344 with a
database `syncope` fully accessible by user `syncope` with password `<PASSWORD>`.

Download
ifeval::["{snapshotOrRelease}" == "release"]
https://github.com/apache/syncope/blob/syncope-{docVersion}/fit/core-reference/src/main/resources/sqlserver/views.xml[views.xml^]
endif::[]
ifeval::["{snapshotOrRelease}" == "snapshot"]
https://github.com/apache/syncope/tree/master/fit/core-reference/src/main/resources/sqlserver/views.xml[views.xml^]
endif::[]
and save it under `core/src/main/resources/`.

{'fraction_non_alphanumeric': 0.13210793190872872, 'fraction_numerical': 0.007787033683448026, 'mean_word_length': 5.566290130796671, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 67, 'https://': 26, 'lorem ipsum': 0, 'www.': 4, 'xml': 64}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3527913', 'n_tokens_mistral': 7263, 'n_tokens_neox': 7024, 'n_words': 1582}