<commit_before>= A lightweight UTF8-aware String class meant for use Ruby 1.8.x written in C

I'm only planning on making the methods I need UTF8-aware, if you need others I welcome pull requests :)

== Example

  str = "你好世界"
  str.length # 12
  str[0] # 228
  str.chars.to_a.inspect # ["\344", "\275", "\240", "\345", "\245", "\275", "\344", "\270", "\226", "\347", "\225", "\214"]

  utf8_str = "你好世界".as_utf8
  utf8_str.length # 4
  utf8[0] #  你
  utf8.chars.to_a.inspect # ["你", "好", "世", "界"]<commit_msg>missed a space<commit_after>= A lightweight UTF8-aware String class meant for use Ruby 1.8.x written in C

I'm only planning on making the methods I need UTF8-aware, if you need others I welcome pull requests :)

== Example

  str = "你好世界"
  str.length # 12
  str[0] # 228
  str.chars.to_a.inspect # ["\344", "\275", "\240", "\345", "\245", "\275", "\344", "\270", "\226", "\347", "\225", "\214"]

  utf8_str = "你好世界".as_utf8
  utf8_str.length # 4
  utf8[0] # 你
  utf8.chars.to_a.inspect # ["你", "好", "世", "界"]
{'fraction_non_alphanumeric': 0.1980392156862745, 'fraction_numerical': 0.10392156862745099, 'mean_word_length': 3.2899159663865545, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '165941', 'n_tokens_mistral': 470, 'n_tokens_neox': 395, 'n_words': 127}