<commit_before>
---
title: "PGO YAML"
date: {docdate}
draft: false
weight: 31
---

# pgo.yaml Configuration
The *pgo.yaml* file contains many different configuration settings as described in this section of the documentation.

The *pgo.yaml* file is broken into major sections as described below:
## Cluster 

| Setting |Definition  |
|---|---|
|BasicAuth        | if set to *true* will enable Basic Authentication
|PrimaryNodeLabel        |newly created primary deployments will specify this node label if specified, unless you override it using the --node-label command line flag, if not set, no node label is specifed
|ReplicaNodeLabel        |newly created replica deployments will specify this node label if specified, unless you override it using the --node-label command line flag, if not set, no node label is specifed
|CCPImageTag        |newly created containers will be based on this image version (e.g. centos7-10.4-1.8.3), unless you override it using the --ccp-image-tag command line flag
|Port        | the PostgreSQL port to use for new containers (e.g. 5432)
|LogStatement        | postgresql.conf log_statement value (required field) (works with crunchy-postgres >= 2.2.0)
|LogMinDurationStatement        | postgresql.conf log_min_duration_statement value (required field) (works with crunchy-postgres >= 2.2.0)
|User        | the PostgreSQL normal user name
|Strategy        | sets the deployment strategy to be used for deploying a cluster, currently there is only strategy *1*
|Replicas        | the number of cluster replicas to create for newly created clusters
|Metrics        | boolean, if set to true will cause each new cluster to include crunchy-collect as a sidecar container for metrics collection, if set to false (default), users can still add metrics on a cluster-by-cluster basis using the pgo command flag --metrics
|Badger        | boolean, if set to true will cause each new cluster to include crunchy-pgbadger as a sidecar container for static log analysis, if set to false (default), users can still add pgbadger on a cluster-by-cluster basis using the pgo create cluster command flag --pgbadger
|Policies        | optional, list of policies to apply to a newly created cluster, comma separated, must be valid policies in the catalog
|PasswordAgeDays        | optional, if set, will set the VALID UNTIL date on passwords to this many days in the future when creating users or setting passwords, defaults to 60 days
|PasswordLength        | optional, if set, will determine the password length used when creating passwords, defaults to 8
|ArchiveMode        | optional, if set to true will enable archive logging for all clusters created, default is false.
|ArchiveTimeout        | optional, if set, will determine the archive timeout setting used when ArchiveMode is true, defaults to 60 seconds
|ServiceType        | optional, if set, will determine the service type used when creating primary or replica services, defaults to ClusterIP if not set, can be overridden by the user on the command line as well
|Backrest        | optional, if set, will cause clusters to have the pgbackrest volume PVC provisioned during cluster creation
|Autofail        | optional, if set, will cause clusters to be checked for auto failover in the event of a non-Ready status
|AutofailReplaceReplica        | optional, default is false, if set, will determine whether a replica is created as part of a failover to replace the promoted replica, the AutofailReplaceReplica setting in *pgo.yaml* is overrode with this command line flag if specified by a user.

## Storage
| Setting|Definition  |
|---|---|
|PrimaryStorage    |required, the value of the storage configuration to use for the primary PostgreSQL deployment
|XlogStorage    |optional, the value of the storage configuration to use for the pgwal (archive) volume for the Postgres container /pgwal volume, if not set, the PrimaryStorage setting is used
|BackupStorage    |required, the value of the storage configuration to use for backups, including the storage for pgbackrest repo volumes
|ReplicaStorage    |required, the value of the storage configuration to use for the replica PostgreSQL deployments
|ReplicaStorage    |required, the value of the storage configuration to use for the replica PostgreSQL deployments
|StorageClass        |for a dynamic storage type, you can specify the storage class used for storage provisioning(e.g. standard, gold, fast)
|AccessMode        |the access mode for new PVCs (e.g. ReadWriteMany, ReadWriteOnce, ReadOnlyMany). See below for descriptions of these.
|Size        |the size to use when creating new PVCs (e.g. 100M, 1Gi)
|Storage.storage1.StorageType        |supported values are either *dynamic*,  *create*,  if not supplied, *create* is used
|Fsgroup        | optional, if set, will cause a *SecurityContext* and *fsGroup* attributes to be added to generated Pod and Deployment definitions
|SupplementalGroups        | optional, if set, will cause a SecurityContext to be added to generated Pod and Deployment definitions
|MatchLabels        | optional, if set, will cause the PVC to add a *matchlabels* selector in order to match a PV, only useful when the StorageType is *create*, when specified a label of *key=value* is added to the PVC as a match criteria

## Storage Configuration Examples
In *pgo.yaml*, you will need to configure your storage configurations
depending on which storage you are wanting to use for
Operator provisioning of Persistent Volume Claims.  The examples
below are provided as a sample.  In all the examples you are
free to change the *Size* to meet your requirements of Persistent
Volume Claim size.

### HostPath Example

HostPath is provided for simple testing and use
cases where you only intend to run on a single
Linux host for your Kubernetes cluster.

```
  hostpathstorage:
    AccessMode:  ReadWriteMany
    Size:  1G
    StorageType:  create
```

### NFS Example

In the following NFS example, notice that the
*SupplementalGroups* setting is set, this can
be whatever GID you have your NFS mount set
to, typically we set this *nfsnobody* as below.
NFS file systems offer a *ReadWriteMany* access
mode.

```
  nfsstorage:
    AccessMode:  ReadWriteMany
    Size:  1G
    StorageType:  create
    SupplementalGroups:  65534
```

### Storage Class Example

In the following example, the important attribute to
set for a typical Storage Class is the  *Fsgroup* setting.
This value is almost always set to *26* which represents
the Postgres user ID that the Crunchy Postgres container
runs as.  Most Storage Class providers offer *ReadWriteOnce*
access modes, but refer to your provider documentation
for other access modes it might support.

```
  storageos:
    AccessMode:  ReadWriteOnce
    Size:  1G
    StorageType:  dynamic
    StorageClass:  fast
    Fsgroup:  26
```


## Container Resources
| Setting |Definition  |
|---|---|
|DefaultContainerResource    |optional, the value of the container resources configuration to use for all database containers, if not set, no resource limits or requests are added on the database container
|DefaultLoadResource    |optional, the value of the container resources configuration to use for pgo-load containers, if not set, no resource limits or requests are added on the database container
|DefaultLspvcResource    |optional, the value of the container resources configuration to use for pgo-lspvc containers, if not set, no resource limits or requests are added on the database container
|DefaultRmdataResource    |optional, the value of the container resources configuration to use for pgo-rmdata containers, if not set, no resource limits or requests are added on the database container
|DefaultBackupResource    |optional, the value of the container resources configuration to use for crunchy-backup containers, if not set, no resource limits or requests are added on the database container
|DefaultPgbouncerResource    |optional, the value of the container resources configuration to use for crunchy-pgbouncer containers, if not set, no resource limits or requests are added on the database container
|DefaultPgpoolResource    |optional, the value of the container resources configuration to use for crunchy-pgpool containers, if not set, no resource limits or requests are added on the database container
|RequestsMemory        | request size of memory in bytes
|RequestsCPU        | request size of CPU cores
|LimitsMemory        | request size of memory in bytes
|LimitsCPU        | request size of CPU cores

## Miscellaneous (Pgo)
| Setting |Definition  |
|---|---|
|PreferredFailoverNode        | optional, a label selector (e.g. hosttype=offsite) that if set, will be used to pick the failover target which is running on a host that matches this label if multiple targets are equal in replication status
|LSPVCTemplate        | the PVC lspvc template file that lists PVC contents
|LoadTemplate        | the load template file used for load jobs
|COImagePrefix        | image tag prefix to use for the Operator containers
|COImageTag        | image tag to use for the Operator containers
|Audit        | boolean, if set to true will cause each apiserver call to be logged with an *audit* marking

## Storage Configuration Details

You can define n-number of Storage configurations within the *pgo.yaml* file. Those Storage configurations follow these conventions -

 * they must have lowercase name (e.g. storage1)
 * they must be unique names (e.g. mydrstorage, faststorage, slowstorage)

These Storage configurations are referenced in the BackupStorage, ReplicaStorage, and PrimaryStorage configuration values. However, there are command line
options in the *pgo* client that will let a user override these default global values to offer you the user a way to specify very targeted storage configurations when needed (e.g. disaster recovery storage for certain backups).

You can set the storage AccessMode values to the following:

* *ReadWriteMany* - mounts the volume as read-write by many nodes
* *ReadWriteOnce* - mounts the PVC as read-write by a single node
* *ReadOnlyMany* - mounts the PVC as read-only by many nodes

These Storage configurations are validated when the *pgo-apiserver* starts, if a
non-valid configuration is found, the apiserver will abort.  These Storage values are only read at *apiserver* start time.

The following StorageType values are possible -

 * *dynamic* - this will allow for dynamic provisioning of storage using a StorageClass.
 * *create* - This setting allows for the creation of a new PVC for each PostgreSQL cluster using a naming convention of *clustername*.  When set, the *Size*, *AccessMode* settings are used in constructing the new PVC.

The operator will create new PVCs using this naming convention: *dbname* where *dbname* is the database name you have specified.  For example, if you run:

    pgo create cluster example1

It will result in a PVC being created named *example1* and in the case of a backup job, the pvc is named *example1-backup*

There are currently 3 sample pgo configuration files provided for users to use as a starting configuration -

 * `pgo.yaml.nfs` - this configuration specifies *create* storage to be used, this is used for NFS storage for example where you want to have a unique PVC created for each database
 * `pgo.yaml.storageclass` - this configuration specifies *dynamic* storage to be used, namely a *storageclass* that refers to a dynamic provisioning strorage such as StorageOS or Portworx, or GCE.

Note, when Storage Type is *create*, you can specify a storage configuration setting of *MatchLabels*, when set, this will cause a *selector* of *key=value* to be added into the PVC, this will let you target specific PV(s) to be matched for this cluster. Note, if a PV does not match the claim request, then the cluster will not start.  Users
that want to use this feature have to place labels on their PV resources as part of PG cluster creation before creating the PG cluster.  For example, users would add a label like this to their PV before they create the PG cluster:

    kubectl label pv somepv myzone=somezone

If you do not specify *MatchLabels* in the storage configuration, then no match filter is added and any available PV will be used to satisfy the PVC request.  This option does not apply to *dynamic* storage types.

Example PV creation scripts are provided that add labels to a set of PVs and can be used for testing:  `$COROOT/pv/create-pv-nfs-labels.sh`
 in that example, a label of **crunchyzone=red** is set on a set of PVs to test with.

The *pgo.yaml* includes a storage config named **nfsstoragered** that when used will demonstrate the label matching.  This feature allows you to support
n-number of NFS storage configurations and supports spreading a PG cluster across different NFS storage configurations.

## Container Resources Details
In the *pgo.yaml* configuration file you have the option to configure a default container resources configuration that when set will add CPU and memory resource limits and requests values into each database container when the container is created.

You can also override the default value using the `--resources-config` command flag when creating a new cluster:

    pgo create cluster testcluster --resources-config=large

Note, if you try to allocate more resources than your
host or Kube cluster has available then you will see your
pods wait in a *Pending* status. The output from a `kubectl describe pod`
command will show output like this in this event:
Events:

      Type     Reason            Age               From               Message
      ----     ------            ----              ----               -------
      Warning  FailedScheduling  49s (x8 over 1m)  default-scheduler  No nodes are available that match all of the predicates: Insufficient memory (1).

## Overriding Storage Configuration Defaults
 

    pgo create cluster testcluster --storage-config=bigdisk

 

That example will create a cluster and specify a storage configuration of *bigdisk* to be used for the primary database storage. The replica storage will default to the value of ReplicaStorage as specified in *pgo.yaml*.

    pgo create cluster testcluster2 --storage-config=fastdisk --replica-storage-config=slowdisk

That example will create a cluster and specify a storage configuration of *fastdisk* to be used for the primary database storage, while the replica storage will use the storage configuration *slowdisk*.

    pgo backup testcluster --storage-config=offsitestorage

That example will create a backup and use the *offsitestorage* storage configuration for persisting the backup.

## Using Storage Configurations for Disaster Recovery
A simple mechanism for partial disaster recovery can be obtained by leveraging network storage, Kubernetes storage classes, and the storage configuration options within the Operator.

For example, if you define a Kubernetes storage class that refers to a storage backend that is running within your disaster recovery site, and then use that storage class as
a storage configuration for your backups, you essentially have moved your backup files automatically to your disaster recovery site thanks to network storage.

<commit_msg>correct documentation<commit_after>
---
title: "PGO YAML"
date: {docdate}
draft: false
weight: 31
---

# pgo.yaml Configuration
The *pgo.yaml* file contains many different configuration settings as described in this section of the documentation.

The *pgo.yaml* file is broken into major sections as described below:
## Cluster 

| Setting |Definition  |
|---|---|
|BasicAuth        | if set to *true* will enable Basic Authentication
|PrimaryNodeLabel        |newly created primary deployments will specify this node label if specified, unless you override it using the --node-label command line flag, if not set, no node label is specifed
|ReplicaNodeLabel        |newly created replica deployments will specify this node label if specified, unless you override it using the --node-label command line flag, if not set, no node label is specifed
|CCPImageTag        |newly created containers will be based on this image version (e.g. centos7-10.4-1.8.3), unless you override it using the --ccp-image-tag command line flag
|Port        | the PostgreSQL port to use for new containers (e.g. 5432)
|LogStatement        | postgresql.conf log_statement value (required field) (works with crunchy-postgres >= 2.2.0)
|LogMinDurationStatement        | postgresql.conf log_min_duration_statement value (required field) (works with crunchy-postgres >= 2.2.0)
|User        | the PostgreSQL normal user name
|Strategy        | sets the deployment strategy to be used for deploying a cluster, currently there is only strategy *1*
|Replicas        | the number of cluster replicas to create for newly created clusters
|Metrics        | boolean, if set to true will cause each new cluster to include crunchy-collect as a sidecar container for metrics collection, if set to false (default), users can still add metrics on a cluster-by-cluster basis using the pgo command flag --metrics
|Badger        | boolean, if set to true will cause each new cluster to include crunchy-pgbadger as a sidecar container for static log analysis, if set to false (default), users can still add pgbadger on a cluster-by-cluster basis using the pgo create cluster command flag --pgbadger
|Policies        | optional, list of policies to apply to a newly created cluster, comma separated, must be valid policies in the catalog
|PasswordAgeDays        | optional, if set, will set the VALID UNTIL date on passwords to this many days in the future when creating users or setting passwords, defaults to 60 days
|PasswordLength        | optional, if set, will determine the password length used when creating passwords, defaults to 8
|ArchiveMode        | optional, if set to true will enable archive logging for all clusters created, default is false.
|ArchiveTimeout        | optional, if set, will determine the archive timeout setting used when ArchiveMode is true, defaults to 60 seconds
|ServiceType        | optional, if set, will determine the service type used when creating primary or replica services, defaults to ClusterIP if not set, can be overridden by the user on the command line as well
|Backrest        | optional, if set, will cause clusters to have the pgbackrest volume PVC provisioned during cluster creation
|Autofail        | optional, if set, will cause clusters to be checked for auto failover in the event of a non-Ready status
|AutofailReplaceReplica        | optional, default is false, if set, will determine whether a replica is created as part of a failover to replace the promoted replica, the AutofailReplaceReplica setting in *pgo.yaml* is overrode with this command line flag if specified by a user.

## Storage
| Setting|Definition  |
|---|---|
|PrimaryStorage    |required, the value of the storage configuration to use for the primary PostgreSQL deployment
|XlogStorage    |optional, the value of the storage configuration to use for the pgwal (archive) volume for the Postgres container /pgwal volume, if not set, the PrimaryStorage setting is used
|BackupStorage    |required, the value of the storage configuration to use for backups, including the storage for pgbackrest repo volumes
|ReplicaStorage    |required, the value of the storage configuration to use for the replica PostgreSQL deployments
|ReplicaStorage    |required, the value of the storage configuration to use for the replica PostgreSQL deployments
|StorageClass        |for a dynamic storage type, you can specify the storage class used for storage provisioning(e.g. standard, gold, fast)
|AccessMode        |the access mode for new PVCs (e.g. ReadWriteMany, ReadWriteOnce, ReadOnlyMany). See below for descriptions of these.
|Size        |the size to use when creating new PVCs (e.g. 100M, 1Gi)
|Storage.storage1.StorageType        |supported values are either *dynamic*,  *create*,  if not supplied, *create* is used
|Fsgroup        | optional, if set, will cause a *SecurityContext* and *fsGroup* attributes to be added to generated Pod and Deployment definitions
|SupplementalGroups        | optional, if set, will cause a SecurityContext to be added to generated Pod and Deployment definitions
|MatchLabels        | optional, if set, will cause the PVC to add a *matchlabels* selector in order to match a PV, only useful when the StorageType is *create*, when specified a label of *key=value* is added to the PVC as a match criteria

## Storage Configuration Examples
In *pgo.yaml*, you will need to configure your storage configurations
depending on which storage you are wanting to use for
Operator provisioning of Persistent Volume Claims.  The examples
below are provided as a sample.  In all the examples you are
free to change the *Size* to meet your requirements of Persistent
Volume Claim size.

### HostPath Example

HostPath is provided for simple testing and use
cases where you only intend to run on a single
Linux host for your Kubernetes cluster.

```
  hostpathstorage:
    AccessMode:  ReadWriteMany
    Size:  1G
    StorageType:  create
```

### NFS Example

In the following NFS example, notice that the
*SupplementalGroups* setting is set, this can
be whatever GID you have your NFS mount set
to, typically we set this *nfsnobody* as below.
NFS file systems offer a *ReadWriteMany* access
mode.

```
  nfsstorage:
    AccessMode:  ReadWriteMany
    Size:  1G
    StorageType:  create
    SupplementalGroups:  65534
```

### Storage Class Example

In the following example, the important attribute to
set for a typical Storage Class is the  *Fsgroup* setting.
This value is almost always set to *26* which represents
the Postgres user ID that the Crunchy Postgres container
runs as.  Most Storage Class providers offer *ReadWriteOnce*
access modes, but refer to your provider documentation
for other access modes it might support.

```
  storageos:
    AccessMode:  ReadWriteOnce
    Size:  1G
    StorageType:  dynamic
    StorageClass:  fast
    Fsgroup:  26
```


## Container Resources
| Setting |Definition  |
|---|---|
|DefaultContainerResource    |optional, the value of the container resources configuration to use for all database containers, if not set, no resource limits or requests are added on the database container
|DefaultLoadResource    |optional, the value of the container resources configuration to use for pgo-load containers, if not set, no resource limits or requests are added on the database container
|DefaultLspvcResource    |optional, the value of the container resources configuration to use for pgo-lspvc containers, if not set, no resource limits or requests are added on the database container
|DefaultRmdataResource    |optional, the value of the container resources configuration to use for pgo-rmdata containers, if not set, no resource limits or requests are added on the database container
|DefaultBackupResource    |optional, the value of the container resources configuration to use for crunchy-backup containers, if not set, no resource limits or requests are added on the database container
|DefaultPgbouncerResource    |optional, the value of the container resources configuration to use for crunchy-pgbouncer containers, if not set, no resource limits or requests are added on the database container
|DefaultPgpoolResource    |optional, the value of the container resources configuration to use for crunchy-pgpool containers, if not set, no resource limits or requests are added on the database container
|RequestsMemory        | request size of memory in bytes
|RequestsCPU        | request size of CPU cores
|LimitsMemory        | request size of memory in bytes
|LimitsCPU        | request size of CPU cores

## Miscellaneous (Pgo)
| Setting |Definition  |
|---|---|
|PreferredFailoverNode        | optional, a label selector (e.g. hosttype=offsite) that if set, will be used to pick the failover target which is running on a host that matches this label if multiple targets are equal in replication status
|LSPVCTemplate        | the PVC lspvc template file that lists PVC contents
|LoadTemplate        | the load template file used for load jobs
|COImagePrefix        | image tag prefix to use for the Operator containers
|COImageTag        | image tag to use for the Operator containers
|Audit        | boolean, if set to true will cause each apiserver call to be logged with an *audit* marking

## Storage Configuration Details

You can define n-number of Storage configurations within the *pgo.yaml* file. Those Storage configurations follow these conventions -

 * they must have lowercase name (e.g. storage1)
 * they must be unique names (e.g. mydrstorage, faststorage, slowstorage)

These Storage configurations are referenced in the BackupStorage, ReplicaStorage, and PrimaryStorage configuration values. However, there are command line
options in the *pgo* client that will let a user override these default global values to offer you the user a way to specify very targeted storage configurations when needed (e.g. disaster recovery storage for certain backups).

You can set the storage AccessMode values to the following:

* *ReadWriteMany* - mounts the volume as read-write by many nodes
* *ReadWriteOnce* - mounts the PVC as read-write by a single node
* *ReadOnlyMany* - mounts the PVC as read-only by many nodes

These Storage configurations are validated when the *pgo-apiserver* starts, if a
non-valid configuration is found, the apiserver will abort.  These Storage values are only read at *apiserver* start time.

The following StorageType values are possible -

 * *dynamic* - this will allow for dynamic provisioning of storage using a StorageClass.
 * *create* - This setting allows for the creation of a new PVC for each PostgreSQL cluster using a naming convention of *clustername*.  When set, the *Size*, *AccessMode* settings are used in constructing the new PVC.

The operator will create new PVCs using this naming convention: *dbname* where *dbname* is the database name you have specified.  For example, if you run:

    pgo create cluster example1

It will result in a PVC being created named *example1* and in the case of a backup job, the pvc is named *example1-backup*

Note, when Storage Type is *create*, you can specify a storage configuration setting of *MatchLabels*, when set, this will cause a *selector* of *key=value* to be added into the PVC, this will let you target specific PV(s) to be matched for this cluster. Note, if a PV does not match the claim request, then the cluster will not start.  Users
that want to use this feature have to place labels on their PV resources as part of PG cluster creation before creating the PG cluster.  For example, users would add a label like this to their PV before they create the PG cluster:

    kubectl label pv somepv myzone=somezone

If you do not specify *MatchLabels* in the storage configuration, then no match filter is added and any available PV will be used to satisfy the PVC request.  This option does not apply to *dynamic* storage types.

Example PV creation scripts are provided that add labels to a set of PVs and can be used for testing:  `$COROOT/pv/create-pv-nfs-labels.sh`
 in that example, a label of **crunchyzone=red** is set on a set of PVs to test with.

The *pgo.yaml* includes a storage config named **nfsstoragered** that when used will demonstrate the label matching.  This feature allows you to support
n-number of NFS storage configurations and supports spreading a PG cluster across different NFS storage configurations.

## Container Resources Details
In the *pgo.yaml* configuration file you have the option to configure a default container resources configuration that when set will add CPU and memory resource limits and requests values into each database container when the container is created.

You can also override the default value using the `--resources-config` command flag when creating a new cluster:

    pgo create cluster testcluster --resources-config=large

Note, if you try to allocate more resources than your
host or Kube cluster has available then you will see your
pods wait in a *Pending* status. The output from a `kubectl describe pod`
command will show output like this in this event:
Events:

      Type     Reason            Age               From               Message
      ----     ------            ----              ----               -------
      Warning  FailedScheduling  49s (x8 over 1m)  default-scheduler  No nodes are available that match all of the predicates: Insufficient memory (1).

## Overriding Storage Configuration Defaults
 

    pgo create cluster testcluster --storage-config=bigdisk

 

That example will create a cluster and specify a storage configuration of *bigdisk* to be used for the primary database storage. The replica storage will default to the value of ReplicaStorage as specified in *pgo.yaml*.

    pgo create cluster testcluster2 --storage-config=fastdisk --replica-storage-config=slowdisk

That example will create a cluster and specify a storage configuration of *fastdisk* to be used for the primary database storage, while the replica storage will use the storage configuration *slowdisk*.

    pgo backup testcluster --storage-config=offsitestorage

That example will create a backup and use the *offsitestorage* storage configuration for persisting the backup.

## Using Storage Configurations for Disaster Recovery
A simple mechanism for partial disaster recovery can be obtained by leveraging network storage, Kubernetes storage classes, and the storage configuration options within the Operator.

For example, if you define a Kubernetes storage class that refers to a storage backend that is running within your disaster recovery site, and then use that storage class as
a storage configuration for your backups, you essentially have moved your backup files automatically to your disaster recovery site thanks to network storage.


{'fraction_non_alphanumeric': 0.04838172615876399, 'fraction_numerical': 0.003496270644645711, 'mean_word_length': 4.031496062992126, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '410703', 'n_tokens_mistral': 7353, 'n_tokens_neox': 7115, 'n_words': 4215}