<commit_before>
# qsplit
Quoted string splitting for Golang.

Qsplit splits a byte-slice into its constituent non-whitespace chunks,
keeping quoted chunks together, in the manner of the shell (mostly).

```
"foo bar baz"      -> [ b'foo', b'bar', b'baz'}
"   foo \tbar baz" -> [ b'foo', b'bar', b'baz'}
"'foo bar' baz"    -> [ b'foo bar', b`baz'}
"a b'cd e'f"       -> [ b'a', b"b'cd", b"e'f"}
```
The library is tuned for speed, and the definition of quote characters
is configurable.

See the [package doc](https://pkg.go.dev/github.com/firepear/qsplit/v2?tab=doc) for more
information.

## News

- 2020-05-17: v2.5.0: Refactor of `LocationsOnce`; 1.1% speedup of
  `Locations`. `Once` now calls `LocationsOnce`
- 2020-05-16: v2.4.0: Refactor for speed improvements (min 1.5%;
  scales with input size). Default quote set is now single and double
  quote
- 2020-05-13: v2.3.1: Fix for import path after module transition
- 2020-05-13: v2.3.0: `qsplit` is now a Go module

See the [Release
notes](https://github.com/firepear/qsplit/blob/master/RELEASE_NOTES)
for all updates.

## Use

`go get github.com/firepear/qsplit/v2` or just import it!
<commit_msg>update docs<commit_after>
# qsplit
Quoted string splitting for Golang.

Qsplit turns a byte-slice into its constituent non-whitespace chunks
of bytes, keeping quoted chunks together, in the manner of the
shell. Mostly.

```
"foo bar baz"      -> [ b`foo`, b`bar`, b`baz` ]
"   foo \tbar baz" -> [ b`foo`, b`bar`, b`baz` ]
"'foo bar' baz"    -> [ b`foo bar`, b`baz` ]
"a b'cd e'f"       -> [ b`a`, b`b'cd`, b`e'f` ]
```
The library is tuned for speed, and the definition of quote characters
is configurable.

See the [package doc](https://pkg.go.dev/github.com/firepear/qsplit/v2?tab=doc) for more
information.

## News

- 2020-05-17: v2.5.0: Refactor of `LocationsOnce`; 1.1% speedup of
  `Locations`. `Once` now calls `LocationsOnce`
- 2020-05-16: v2.4.0: Refactor for speed improvements (min 1.5%;
  scales with input size). Default quote set is now single and double
  quote
- 2020-05-13: v2.3.1: Fix for import path after module transition
- 2020-05-13: v2.3.0: `qsplit` is now a Go module

See the [Release
notes](https://github.com/firepear/qsplit/blob/master/RELEASE_NOTES)
for all updates.

## Use

`go get github.com/firepear/qsplit/v2` or just import it!

{'fraction_non_alphanumeric': 0.14605773373545886, 'fraction_numerical': 0.043084877208099955, 'mean_word_length': 3.672032193158954, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7090609', 'n_tokens_mistral': 925, 'n_tokens_neox': 847, 'n_words': 281}