<commit_before>alluxio.conf.directory:
  The directory containing files used to configure Alluxio.
alluxio.debug:
  Set to true to enable debug mode which has additional logging and info in the Web UI.
alluxio.home:
  Alluxio installation directory.
alluxio.logs.dir:
  The path to store log files.
alluxio.keyvalue.enabled:
  Whether the key-value service is enabled.
alluxio.keyvalue.partition.size.bytes.max:
  Maximum allowable size (in bytes) of a single key-value partition in a store. This value
  should be no larger than the block size (alluxio.user.block.size.bytes.default)
alluxio.metrics.conf.file:
  The file path of the metrics system configuration file. By default it is `metrics.properties`
  in the `conf` directory.
alluxio.network.host.resolution.&#8203;timeout.ms:
  During startup of the Master and Worker processes Alluxio needs to ensure that they are
  listening on externally resolvable and reachable host names.  To do this, Alluxio will
  automatically attempt to select an appropriate host name if one was not explicitly specified.
  This represents the maximum amount of time spent waiting to determine if a candidate host name
  is resolvable over the network.
alluxio.network.thrift.frame.&#8203;size.bytes.max:
  (Experimental) The largest allowable frame size used for Thrift RPC communication.
alluxio.site.conf.dir:
  Default search path for configuration files to read
alluxio.test.mode:
  Flag used only during tests to allow special behavior.
alluxio.underfs.address:
  Alluxio directory in the underlayer file system.
alluxio.underfs.gcs.owner.id.to.username.mapping:
  Optionally, specify a preset gcs owner id to Alluxio username static mapping in the
  format "id1=user1;id2=user2". The Google Cloud Storage IDs can be found at the console address
  https://console.cloud.google.com/storage/settings . Please use the "Owners" one.
alluxio.underfs.glusterfs.impl:
  Glusterfs hook with hadoop.
alluxio.underfs.glusterfs.mapred.&#8203;system.dir:
  Optionally, specify subdirectory under GlusterFS for intermediary MapReduce data.
alluxio.underfs.hdfs.configuration:
  Location of the hdfs configuration file.<commit_msg>underlayer file system -> under file system<commit_after>alluxio.conf.directory:
  The directory containing files used to configure Alluxio.
alluxio.debug:
  Set to true to enable debug mode which has additional logging and info in the Web UI.
alluxio.home:
  Alluxio installation directory.
alluxio.logs.dir:
  The path to store log files.
alluxio.keyvalue.enabled:
  Whether the key-value service is enabled.
alluxio.keyvalue.partition.size.bytes.max:
  Maximum allowable size (in bytes) of a single key-value partition in a store. This value
  should be no larger than the block size (alluxio.user.block.size.bytes.default)
alluxio.metrics.conf.file:
  The file path of the metrics system configuration file. By default it is `metrics.properties`
  in the `conf` directory.
alluxio.network.host.resolution.&#8203;timeout.ms:
  During startup of the Master and Worker processes Alluxio needs to ensure that they are
  listening on externally resolvable and reachable host names.  To do this, Alluxio will
  automatically attempt to select an appropriate host name if one was not explicitly specified.
  This represents the maximum amount of time spent waiting to determine if a candidate host name
  is resolvable over the network.
alluxio.network.thrift.frame.&#8203;size.bytes.max:
  (Experimental) The largest allowable frame size used for Thrift RPC communication.
alluxio.site.conf.dir:
  Default search path for configuration files to read
alluxio.test.mode:
  Flag used only during tests to allow special behavior.
alluxio.underfs.address:
  Alluxio directory in the under file system.
alluxio.underfs.gcs.owner.id.to.username.mapping:
  Optionally, specify a preset gcs owner id to Alluxio username static mapping in the
  format "id1=user1;id2=user2". The Google Cloud Storage IDs can be found at the console address
  https://console.cloud.google.com/storage/settings . Please use the "Owners" one.
alluxio.underfs.glusterfs.impl:
  Glusterfs hook with hadoop.
alluxio.underfs.glusterfs.mapred.&#8203;system.dir:
  Optionally, specify subdirectory under GlusterFS for intermediary MapReduce data.
alluxio.underfs.hdfs.configuration:
  Location of the hdfs configuration file.
{'fraction_non_alphanumeric': 0.06300298919291791, 'fraction_numerical': 0.0073580133363991725, 'mean_word_length': 4.591259640102828, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5405974', 'n_tokens_mistral': 1258, 'n_tokens_neox': 1235, 'n_words': 502}