<commit_before># INSTALLATION INSTRUCTIONS for RHEL 8.x, CentOS8/Stream
-------------------------

### -2/ RHEL8/CentOS8/CentOS_Stream/Fedora33 - status
-------------------------
!!! notice
    Tested fully working without SELinux by [@SteveClement](https://twitter.com/SteveClement) on 20210401
    TODO: Fix SELinux permissions, *pull-requests welcome*.

{!generic/manual-install-notes.md!}

!!! notice
    If the next line is `[!generic/community.md!]()` [click here](https://misp.github.io/MISP/INSTALL.rhel8/).

{!generic/community.md!}

### 0/ Overview and Assumptions

{!generic/rhelVScentos.md!}

!!! warning
    The core MISP team cannot easily verify if this guide is working or not. Please help us in keeping it up to date and accurate.
    Thus we also have difficulties in supporting RHEL issues but will do a best effort on a similar yet slightly different setup.

!!! notice
    Maintenance for CentOS 8 will end on: December 31st, 2021 [Source[0]](https://wiki.centos.org/About/Product) [Source[1]](https://linuxlifecycle.com/)
    CentOS 8 [NetInstallURL](http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS)

{!generic/manual-install-notes.md!}

This document details the steps to install MISP on Red Hat Enterprise Linux 8.x (RHEL 8.x) and CentOS 8.x.
This is a joint RHEL/CentOS install guide. The authors tried to make it contextually evident what applies to which flavor.

The following assumptions with regard to this installation have been made.

- A valid support agreement allowing the system to register to the Red Hat Customer Portal and receive updates
- The ability to enable additional RPM repositories, specifically the EPEL and Software Collections (SCL) repos
- This system will have direct or proxy access to the Internet for updates. Or connected to a Red Hat Satellite Server
- This document will bootstrap a MISP instance running over HTTPS. A full test of all features have yet to be done. [The following GitHub issue](https://github.com/MISP/MISP/issues/4084) details some shortcomings.

{!generic/globalVariables.md!}

!!! note
    For fresh installs the following tips might be handy.<br />
    Allow ssh to pass the firewall on the CLI
    ```bash
    firewall-cmd --zone=public --add-port=22/tcp --permanent
    firewall-cmd --reload
    ```
    <br />
    To quickly make sure if NetworkManager handles your network interface on boot, check in the following location:
    ```
    /etc/sysconfig/network-scripts/ifcfg-*
    ```

### 1/ OS Install and additional repositories

## 1.1/ Complete a minimal RHEL/CentOS installation, configure IP address to connect automatically.

## 1.2/ Configure system hostname (if not done during install)
```bash
sudo hostnamectl set-hostname misp.local # Your choice, in a production environment, it's best to use a FQDN
```

## 1.3/ **[RHEL]** Register the system for updates with Red Hat Subscription Manager
```bash
# <snippet-begin 0_RHEL_register.sh>
registerRHEL () {
  sudo subscription-manager register --auto-attach # register your system to an account and attach to a current subscription
}
# <snippet-end 0_RHEL_register.sh>
```

## 1.4/ **[RHEL]** Enable the optional repos (obsolete in v8)
```bash
# <snippet-begin 0_RHEL8_SCL.sh>
enableOptionalRHEL8 () {
  sudo subscription-manager refresh

  # The following is needed for -devel repos and ONLY for misp-modules, ignore if not needed
  sudo subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms
  # Software Collections is available for Red Hat Enterprise Linux 7 and previous supported releases. Starting with Red Hat Enterprise Linux 8, the content traditionally consumed via Software Collections is now part of Application Streams. Please see the Application Streams Life Cycle documentation for that release. Source: https://access.redhat.com/support/policy/updates/rhscl
}
# <snippet-end 0_RHEL8_SCL.sh>
```

## 1.5a/ Install the deltarpm package to help reduce download size when installing updates (optional)
```bash
sudo dnf install drpm -y
```

## 1.5.b/ Install vim (optional)
```bash
# Because (neo)vim is just so practical
sudo dnf install neovim -y
# For RHEL, it's vim and after enabling epel neovim is available too
```

## 1.5.c/ Install ntpdate (optional)
```bash
# In case you time is wrong, this will fix it.
sudo dnf install ntpdate -y
sudo ntpdate pool.ntp.org
```

## 1.5/ Update the system and reboot
```bash
# <snippet-begin 0_yum-update.sh>
yumUpdate () {
  sudo dnf update -y
}
# <snippet-end 0_yum-update.sh>
```

## 1.6/ Install the EPEL and remi repo
```bash
# <snippet-begin 0_EPEL_REMI.sh>
enableEPEL_REMI_8 () {
  sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y
  sudo dnf install http://rpms.remirepo.net/enterprise/remi-release-8.rpm -y
  sudo dnf install dnf-utils -y
  sudo dnf module enable php:remi-7.4 -y
  [[ ${DISTRI} == "centos8stream" ]] && sudo dnf config-manager --set-enabled powertools
  [[ ${DISTRI} == "centos8" ]] && sudo dnf config-manager --set-enabled powertools
}

enableREMI_f33 () {
  sudo dnf install http://rpms.remirepo.net/fedora/remi-release-33.rpm
  sudo dnf install dnf-utils -y
  sudo dnf module enable php:remi-7.4 -y
}
# <snippet-end 0_EPEL_REMI.sh>
```

### 2/ Dependencies

!!! note
    This guide installs PHP 7.4 from Remi's repo

!!! warning
    [PHP 5.6 and 7.0 aren't supported since December 2018](https://secure.php.net/supported-versions.php). Please update accordingly. In the future only PHP7 will be supported.

## 2.01/ Install some base system dependencies
```bash
# <snippet-begin 0_yumInstallCoreDeps8.sh>
yumInstallCoreDeps8 () {
  # Install the dependencies:
  PHP_BASE="/etc/"
  PHP_INI="/etc/php.ini"
  sudo dnf install @httpd -y
  sudo dnf install gcc git zip unzip \
                   httpd \
                   mod_ssl \
                   moreutils \
                   redis \
                   mariadb \
                   mariadb-server \
                   python3-devel python3-pip python3-virtualenv \
                   python3-policycoreutils \
                   policycoreutils-python-utils \
                   langpacks-en glibc-all-langpacks \
                   libxslt-devel zlib-devel ssdeep-devel -y
  sudo alternatives --set python /usr/bin/python3

  # Enable and start redis
  sudo systemctl enable --now redis.service

  # Install PHP 7.4 from Remi's repo, see https://rpms.remirepo.net/enterprise/8/php74/x86_64/repoview/
  sudo dnf install php php-fpm php-devel \
                   php-mysqlnd \
                   php-mbstring \
                   php-xml \
                   php-bcmath \
                   php-opcache \
                   php-zip \
                   php-pear \
                   php-brotli \
                   php-intl \
                   php-gd -y

  # cake has php baked in, thus we link to it if necessary.
  [[ ! -e "/usr/bin/php" ]] && sudo ln -s /usr/bin/php74 /usr/bin/php

  sudo systemctl enable --now php-fpm.service
}
# <snippet-end 0_yumInstallCoreDeps8.sh>
```

```bash
# <snippet-begin 0_yumInstallHaveged.sh>
installEntropyRHEL () {
  # GPG needs lots of entropy, haveged provides entropy
  # /!\ Only do this if you're not running rngd to provide randomness and your kernel randomness is not sufficient.
  sudo dnf install haveged -y
  sudo systemctl enable --now haveged.service
}
# <snippet-end 0_yumInstallHaveged.sh>
```

### 3/ MISP code
## 3.01/ Download MISP code using git in /var/www/ directory

```bash
# <snippet-begin 1_mispCoreInstall_RHEL8.sh>
compileLiefRHEL8 () {
  cd $PATH_TO_MISP/app/files/scripts
  $SUDO_WWW git clone --branch master --single-branch https://github.com/lief-project/LIEF.git lief
  # lief might need manual compilation
  sudo dnf groupinstall "Development Tools" -y

  cd $PATH_TO_MISP/app/files/scripts/lief
  $SUDO_WWW git config core.filemode false
  $SUDO_WWW mkdir build
  cd build
  $SUDO_WWW ${CMAKE_BIN} \
    -DLIEF_PYTHON_API=on \
    -DPYTHON_VERSION=3.6 \
    -DPYTHON_EXECUTABLE=$PATH_TO_MISP/venv/bin/python \
    -DLIEF_DOC=off \
    -DCMAKE_BUILD_TYPE=Release \
  ..
  $SUDO_WWW make -j3 pyLIEF

  if [ $? == 2 ]; then
    # In case you get "internal compiler error: Killed (program cc1plus)"
    # You ran out of memory.
    # Create some swap
    TEMP_DIR=$(mktemp -d)
    TEMP_SWAP=${TEMP_DIR}/swap.img
    sudo dd if=/dev/zero of=${TEMP_SWAP} bs=1024k count=4000
    sudo mkswap ${TEMP_SWAP}
    sudo swapon ${TEMP_SWAP}
    # And compile again
    ${SUDO_WWW} make -j3 pyLIEF
    sudo swapoff ${TEMP_SWAP}
    sudo rm -r ${TEMP_DIR}
  fi

  # The following adds a PYTHONPATH to where the pyLIEF module has been compiled
  echo /var/www/MISP/app/files/scripts/lief/build/api/python |$SUDO_WWW tee /var/www/MISP/venv/lib/python3.6/site-packages/lief.pth
  [[ "${DISTRI}" == "fedora33" ]] && (echo /var/www/MISP/app/files/scripts/lief/build/api/python |$SUDO_WWW tee /var/www/MISP/venv/lib/python3.9/site-packages/lief.pth)
$SUDO_WWW $PATH_TO_MISP/venv/bin/pip install -U python-magic plyara
}

installCoreRHEL8 () {
  # Download MISP using git in the $PATH_TO_MISP directory.
  sudo mkdir -p $(dirname $PATH_TO_MISP)
  sudo chown $WWW_USER:$WWW_USER $(dirname $PATH_TO_MISP)
  cd $(dirname $PATH_TO_MISP)
  $SUDO_WWW git clone https://github.com/MISP/MISP.git
  cd $PATH_TO_MISP

  # Fetch submodules
  $SUDO_WWW git submodule update --init --recursive
  # Make git ignore filesystem permission differences for submodules
  $SUDO_WWW git submodule foreach --recursive git config core.filemode false
  # Make git ignore filesystem permission differences
  $SUDO_WWW git config core.filemode false

  # Create a python3 virtualenv
  [[ -e $(which virtualenv-3 2>/dev/null) ]] && $SUDO_WWW virtualenv-3 -p python3 $PATH_TO_MISP/venv
  [[ -e $(which virtualenv 2>/dev/null) ]] && $SUDO_WWW virtualenv -p python3 $PATH_TO_MISP/venv
  sudo mkdir /usr/share/httpd/.cache
  sudo chown $WWW_USER:$WWW_USER /usr/share/httpd/.cache
  $SUDO_WWW $PATH_TO_MISP/venv/bin/pip install -U pip setuptools

  cd $PATH_TO_MISP/app/files/scripts
  $SUDO_WWW git clone https://github.com/CybOXProject/python-cybox.git
  $SUDO_WWW git clone https://github.com/STIXProject/python-stix.git
  $SUDO_WWW git clone https://github.com/CybOXProject/mixbox.git

  cd $PATH_TO_MISP/app/files/scripts/python-cybox
  $SUDO_WWW git config core.filemode false
  # If you umask is has been changed from the default, it is a good idea to reset it to 0022 before installing python modules
  ([[ ${DISTRI} == 'fedora33' ]] || [[ ${DISTRI} == 'rhel8.3' ]]) && sudo dnf install cmake3 -y && CMAKE_BIN='cmake3'
  [[ ${DISTRI} == 'centos8stream' ]] && sudo dnf install cmake -y && CMAKE_BIN='cmake'
  [[ ${DISTRI} == 'centos8' ]] && sudo dnf install cmake -y && CMAKE_BIN='cmake'

  UMASK=$(umask)
  umask 0022

  cd $PATH_TO_MISP/app/files/scripts/python-cybox
  $SUDO_WWW git config core.filemode false
  $SUDO_WWW $PATH_TO_MISP/venv/bin/pip install .
<commit_msg>chg: [doc] Added Rocky Linux 8.4<commit_after># INSTALLATION INSTRUCTIONS for RHEL 8.x, CentOS8/Stream/Rocky8.4
-------------------------

### -2/ RHEL8/CentOS8/CentOS_Stream/Rocky8.4/Fedora33 - status
-------------------------
!!! notice
    Tested fully working without SELinux by [@SteveClement](https://twitter.com/SteveClement) on 20210401
    TODO: Fix SELinux permissions, *pull-requests welcome*.

{!generic/manual-install-notes.md!}

!!! notice
    If the next line is `[!generic/community.md!]()` [click here](https://misp.github.io/MISP/INSTALL.rhel8/).

{!generic/community.md!}

### 0/ Overview and Assumptions

{!generic/rhelVScentos.md!}

!!! warning
    The core MISP team cannot easily verify if this guide is working or not. Please help us in keeping it up to date and accurate.
    Thus we also have difficulties in supporting RHEL issues but will do a best effort on a similar yet slightly different setup.

!!! notice
    Maintenance for CentOS 8 will end on: December 31st, 2021 [Source[0]](https://wiki.centos.org/About/Product) [Source[1]](https://linuxlifecycle.com/)
    Consider using [Rocky Linux](https://rockylinux.org/)
    CentOS 8 [NetInstallURL](http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS)

{!generic/manual-install-notes.md!}

This document details the steps to install MISP on Red Hat Enterprise Linux 8.x (RHEL 8.x) and CentOS 8.x.
This is a joint RHEL/CentOS install guide. The authors tried to make it contextually evident what applies to which flavor.

The following assumptions with regard to this installation have been made.

- A valid support agreement allowing the system to register to the Red Hat Customer Portal and receive updates
- The ability to enable additional RPM repositories, specifically the EPEL and Software Collections (SCL) repos
- This system will have direct or proxy access to the Internet for updates. Or connected to a Red Hat Satellite Server
- This document will bootstrap a MISP instance running over HTTPS. A full test of all features have yet to be done. [The following GitHub issue](https://github.com/MISP/MISP/issues/4084) details some shortcomings.

{!generic/globalVariables.md!}

!!! note
    For fresh installs the following tips might be handy.<br />
    Allow ssh to pass the firewall on the CLI
    ```bash
    firewall-cmd --zone=public --add-port=22/tcp --permanent
    firewall-cmd --reload
    ```
    <br />
    To quickly make sure if NetworkManager handles your network interface on boot, check in the following location:
    ```
    /etc/sysconfig/network-scripts/ifcfg-*
    ```

### 1/ OS Install and additional repositories

## 1.1/ Complete a minimal RHEL/CentOS installation, configure IP address to connect automatically.

## 1.2/ Configure system hostname (if not done during install)
```bash
sudo hostnamectl set-hostname misp.local # Your choice, in a production environment, it's best to use a FQDN
```

## 1.3/ **[RHEL]** Register the system for updates with Red Hat Subscription Manager
```bash
# <snippet-begin 0_RHEL_register.sh>
registerRHEL () {
  sudo subscription-manager register --auto-attach # register your system to an account and attach to a current subscription
}
# <snippet-end 0_RHEL_register.sh>
```

## 1.4/ **[RHEL]** Enable the optional repos (obsolete in v8)
```bash
# <snippet-begin 0_RHEL8_SCL.sh>
enableOptionalRHEL8 () {
  sudo subscription-manager refresh

  # The following is needed for -devel repos and ONLY for misp-modules, ignore if not needed
  sudo subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms
  # Software Collections is available for Red Hat Enterprise Linux 7 and previous supported releases. Starting with Red Hat Enterprise Linux 8, the content traditionally consumed via Software Collections is now part of Application Streams. Please see the Application Streams Life Cycle documentation for that release. Source: https://access.redhat.com/support/policy/updates/rhscl
}
# <snippet-end 0_RHEL8_SCL.sh>
```

## 1.5a/ Install the deltarpm package to help reduce download size when installing updates (optional)
```bash
sudo dnf install drpm -y
```

## 1.5.b/ Install vim (optional)
```bash
# Because (neo)vim is just so practical
sudo dnf install neovim -y
# For RHEL, it's vim and after enabling epel neovim is available too
```

## 1.5.c/ Install ntpdate (optional)
```bash
# In case you time is wrong, this will fix it.
sudo dnf install ntpdate -y
sudo ntpdate pool.ntp.org
```

## 1.5/ Update the system and reboot
```bash
# <snippet-begin 0_yum-update.sh>
yumUpdate () {
  sudo dnf update -y
}
# <snippet-end 0_yum-update.sh>
```

## 1.6/ Install the EPEL and remi repo
```bash
# <snippet-begin 0_EPEL_REMI.sh>
enableEPEL_REMI_8 () {
  sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y
  sudo dnf install http://rpms.remirepo.net/enterprise/remi-release-8.rpm -y
  sudo dnf install dnf-utils -y
  sudo dnf module enable php:remi-7.4 -y
  ([[ ${DISTRI} == "centos8stream" ]] || [[ ${DISTRI} == "centos8" ]] || [[ ${DISTRI} == "rocky8.4" ]]) && sudo dnf config-manager --set-enabled powertools
}

enableREMI_f33 () {
  sudo dnf install http://rpms.remirepo.net/fedora/remi-release-33.rpm
  sudo dnf install dnf-utils -y
  sudo dnf module enable php:remi-7.4 -y
}
# <snippet-end 0_EPEL_REMI.sh>
```

### 2/ Dependencies

!!! note
    This guide installs PHP 7.4 from Remi's repo

!!! warning
    [PHP 5.6 and 7.0 aren't supported since December 2018](https://secure.php.net/supported-versions.php). Please update accordingly. In the future only PHP7 will be supported.

## 2.01/ Install some base system dependencies
```bash
# <snippet-begin 0_yumInstallCoreDeps8.sh>
yumInstallCoreDeps8 () {
  # Install the dependencies:
  PHP_BASE="/etc/"
  PHP_INI="/etc/php.ini"
  sudo dnf install @httpd -y
  sudo dnf install gcc git zip unzip \
                   httpd \
                   mod_ssl \
                   moreutils \
                   redis \
                   mariadb \
                   mariadb-server \
                   python3-devel python3-pip python3-virtualenv \
                   python3-policycoreutils \
                   policycoreutils-python-utils \
                   langpacks-en glibc-all-langpacks \
                   libxslt-devel zlib-devel ssdeep-devel -y
  sudo alternatives --set python /usr/bin/python3

  # Enable and start redis
  sudo systemctl enable --now redis.service

  # Install PHP 7.4 from Remi's repo, see https://rpms.remirepo.net/enterprise/8/php74/x86_64/repoview/
  sudo dnf install php php-fpm php-devel \
                   php-mysqlnd \
                   php-mbstring \
                   php-xml \
                   php-bcmath \
                   php-opcache \
                   php-zip \
                   php-pear \
                   php-brotli \
                   php-intl \
                   php-gd -y

  # cake has php baked in, thus we link to it if necessary.
  [[ ! -e "/usr/bin/php" ]] && sudo ln -s /usr/bin/php74 /usr/bin/php

  sudo systemctl enable --now php-fpm.service
}
# <snippet-end 0_yumInstallCoreDeps8.sh>
```

```bash
# <snippet-begin 0_yumInstallHaveged.sh>
installEntropyRHEL () {
  # GPG needs lots of entropy, haveged provides entropy
  # /!\ Only do this if you're not running rngd to provide randomness and your kernel randomness is not sufficient.
  sudo dnf install haveged -y
  sudo systemctl enable --now haveged.service
}
# <snippet-end 0_yumInstallHaveged.sh>
```

### 3/ MISP code
## 3.01/ Download MISP code using git in /var/www/ directory

```bash
# <snippet-begin 1_mispCoreInstall_RHEL8.sh>
compileLiefRHEL8 () {
  cd $PATH_TO_MISP/app/files/scripts
  $SUDO_WWW git clone --branch master --single-branch https://github.com/lief-project/LIEF.git lief
  # lief might need manual compilation
  sudo dnf groupinstall "Development Tools" -y

  cd $PATH_TO_MISP/app/files/scripts/lief
  $SUDO_WWW git config core.filemode false
  $SUDO_WWW mkdir build
  cd build
  $SUDO_WWW ${CMAKE_BIN} \
    -DLIEF_PYTHON_API=on \
    -DPYTHON_VERSION=3.6 \
    -DPYTHON_EXECUTABLE=$PATH_TO_MISP/venv/bin/python \
    -DLIEF_DOC=off \
    -DCMAKE_BUILD_TYPE=Release \
  ..
  $SUDO_WWW make -j3 pyLIEF

  if [ $? == 2 ]; then
    # In case you get "internal compiler error: Killed (program cc1plus)"
    # You ran out of memory.
    # Create some swap
    TEMP_DIR=$(mktemp -d)
    TEMP_SWAP=${TEMP_DIR}/swap.img
    sudo dd if=/dev/zero of=${TEMP_SWAP} bs=1024k count=4000
    sudo mkswap ${TEMP_SWAP}
    sudo swapon ${TEMP_SWAP}
    # And compile again
    ${SUDO_WWW} make -j3 pyLIEF
    sudo swapoff ${TEMP_SWAP}
    sudo rm -r ${TEMP_DIR}
  fi

  # The following adds a PYTHONPATH to where the pyLIEF module has been compiled
  echo /var/www/MISP/app/files/scripts/lief/build/api/python |$SUDO_WWW tee /var/www/MISP/venv/lib/python3.6/site-packages/lief.pth
  [[ "${DISTRI}" == "fedora33" ]] && (echo /var/www/MISP/app/files/scripts/lief/build/api/python |$SUDO_WWW tee /var/www/MISP/venv/lib/python3.9/site-packages/lief.pth)
$SUDO_WWW $PATH_TO_MISP/venv/bin/pip install -U python-magic plyara
}

installCoreRHEL8 () {
  # Download MISP using git in the $PATH_TO_MISP directory.
  sudo mkdir -p $(dirname $PATH_TO_MISP)
  sudo chown $WWW_USER:$WWW_USER $(dirname $PATH_TO_MISP)
  cd $(dirname $PATH_TO_MISP)
  $SUDO_WWW git clone https://github.com/MISP/MISP.git
  cd $PATH_TO_MISP

  # Fetch submodules
  $SUDO_WWW git submodule update --init --recursive
  # Make git ignore filesystem permission differences for submodules
  $SUDO_WWW git submodule foreach --recursive git config core.filemode false
  # Make git ignore filesystem permission differences
  $SUDO_WWW git config core.filemode false

  # Create a python3 virtualenv
  [[ -e $(which virtualenv-3 2>/dev/null) ]] && $SUDO_WWW virtualenv-3 -p python3 $PATH_TO_MISP/venv
  [[ -e $(which virtualenv 2>/dev/null) ]] && $SUDO_WWW virtualenv -p python3 $PATH_TO_MISP/venv
  sudo mkdir /usr/share/httpd/.cache
  sudo chown $WWW_USER:$WWW_USER /usr/share/httpd/.cache
  $SUDO_WWW $PATH_TO_MISP/venv/bin/pip install -U pip setuptools

  cd $PATH_TO_MISP/app/files/scripts
  $SUDO_WWW git clone https://github.com/CybOXProject/python-cybox.git
  $SUDO_WWW git clone https://github.com/STIXProject/python-stix.git
  $SUDO_WWW git clone https://github.com/CybOXProject/mixbox.git

  cd $PATH_TO_MISP/app/files/scripts/python-cybox
  $SUDO_WWW git config core.filemode false
  # If you umask is has been changed from the default, it is a good idea to reset it to 0022 before installing python modules
  ([[ ${DISTRI} == 'fedora33' ]] || [[ ${DISTRI} == 'rhel8.3' ]]) && sudo dnf install cmake3 -y && CMAKE_BIN='cmake3'
  ([[ ${DISTRI} == 'centos8stream' ]] || [[ ${DISTRI} == 'centos8' ]] || [[ ${DISTRI} == 'rocky8.4' ]]) && sudo dnf install cmake -y && CMAKE_BIN='cmake'

  UMASK=$(umask)
  umask 0022

  cd $PATH_TO_MISP/app/files/scripts/python-cybox
  $SUDO_WWW git config core.filemode false
  $SUDO_WWW $PATH_TO_MISP/venv/bin/pip install .

{'fraction_non_alphanumeric': 0.11378008279878077, 'fraction_numerical': 0.016923706837723488, 'mean_word_length': 3.161681181370693, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 37, 'https://': 29, 'lorem ipsum': 0, 'www.': 10, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6304776', 'n_tokens_mistral': 7954, 'n_tokens_neox': 7250, 'n_words': 2218}