<commit_before># Unicode for String Processing

* Proposal: [SE-0363](0363-unicode-for-string-processing.md)
* Authors: [<NAME>](https://github.com/natecook1000), [<NAME>](https://github.com/Azoy)
* Review Manager: [<NAME>](https://github.com/airspeedswift)
* Status: **Accepted**
* Review: https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520
* Implementation: [apple/swift-experimental-string-processing][repo]  

### Version History

- Version 1: Initial version
- Version 2: 
  - Improved option method API names
  - Added Unicode property APIs to match regex syntax
  - Added `CharacterClass.noneOf(_:)` and sequence-based `init`
  - Clarified default state of options
  - Added detail around switching semantic modes
  - Added detail about Unicode property matching in character mode
  - Revised details of custom character class matching
  - Removed `\O`/`.anyUnicodeScalar`

### Table of Contents

  - [Introduction](#introduction)
  - [Motivation](#motivation)
  - [Proposed solution](#proposed-solution)
  - [Detailed design](#detailed-design)
    - [Options](#options)
      - [Case insensitivity](#case-insensitivity)
      - [Single line mode (`.` matches newlines)](#single-line-mode--matches-newlines)
      - [Multiline mode](#multiline-mode)
      - [ASCII-only character classes](#ascii-only-character-classes)
      - [Unicode word boundaries](#unicode-word-boundaries)<commit_msg>Mark 0363 implemented (#1761)<commit_after># Unicode for String Processing

* Proposal: [SE-0363](0363-unicode-for-string-processing.md)
* Authors: [<NAME>](https://github.com/natecook1000), [<NAME>](https://github.com/Azoy)
* Review Manager: [<NAME>](https://github.com/airspeedswift)
* Status: **Implemented (Swift 5.7)**
* Review: https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520
* Implementation: [apple/swift-experimental-string-processing][repo]  

### Version History

- Version 1: Initial version
- Version 2: 
  - Improved option method API names
  - Added Unicode property APIs to match regex syntax
  - Added `CharacterClass.noneOf(_:)` and sequence-based `init`
  - Clarified default state of options
  - Added detail around switching semantic modes
  - Added detail about Unicode property matching in character mode
  - Revised details of custom character class matching
  - Removed `\O`/`.anyUnicodeScalar`

### Table of Contents

  - [Introduction](#introduction)
  - [Motivation](#motivation)
  - [Proposed solution](#proposed-solution)
  - [Detailed design](#detailed-design)
    - [Options](#options)
      - [Case insensitivity](#case-insensitivity)
      - [Single line mode (`.` matches newlines)](#single-line-mode--matches-newlines)
      - [Multiline mode](#multiline-mode)
      - [ASCII-only character classes](#ascii-only-character-classes)
      - [Unicode word boundaries](#unicode-word-boundaries)
{'fraction_non_alphanumeric': 0.14091384722706662, 'fraction_numerical': 0.019532612486920127, 'mean_word_length': 4.281767955801105, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '926901', 'n_tokens_mistral': 943, 'n_tokens_neox': 818, 'n_words': 202}