<commit_before>homepage: https://github.com/AndrasKovacs/flatparse#readme
changelog-type: ''
hash: ab5e7a3255f626f8cbdf95afdacfc76ca2cbae628f0b34880f5041840f3e9b11
test-bench-deps: {}
maintainer: <EMAIL>
synopsis: High-performance parsing from strict bytestrings
changelog: ''
basic-deps:
  bytestring: -any
  flatparse: -any
  base: '>=4.7 && <5'
  megaparsec: -any
  parsec: -any
  gauge: -any
  containers: -any
  attoparsec: -any
  template-haskell: -any
all-versions:
- 0.1.0.0
author: <NAME>
latest: 0.1.0.0
description-type: markdown
description: "# flatparse\r\n\r\n![CI](https://github.com/AndrasKovacs/flatparse/actions/workflows/haskell.yml/badge.svg)\r\n\r\n`flatparse`
  is a high-performance parsing library, focusing on __programming languages__ and
  __human-readable data formats__. The \"flat\" in the name\r\nrefers to the `ByteString`
  parsing input, which has pinned contiguous data, and also to the library internals,
  which avoids indirections and heap allocations\r\nwhenever possible.\r\n\r\n## Features
  and non-features\r\n\r\n* __Excellent performance__. On microbenchmarks, `flatparse`
  is at least 10 times faster than `attoparsec` or `megaparsec`. On larger examples
  with heavier use of    source positions and spans and/or indentation parsing, the
  performance difference grows to 20-30 times. Compile times and exectuable sizes
  are also significantly better with `flatparse` than with `megaparsec` or `attoparsec`.
  `flatparse` interals make liberal use of unboxed tuples and GHC primops. As a result,
  pure validators (parsers returning `()`) in `flatparse` are not difficult to implement
  with zero heap allocation.\r\n* __No incremental parsing__, and __only strict `ByteString`__
  is supported as input. However, it can be still useful to convert from `Text`, `String`
  or other types to `ByteString`, and then use `flatparse` for parsing, since `flatparse`
  performance usually more than makes up for the conversion costs.\r\n* __Only little-endian
  64 bit systems are currently supported__. This may change in the future. Getting
  good performance requires architecture-specific optimizations; I've only considered
  the most common setting at this point.\r\n* __Support for fast source location handling,
  indentation parsing and informative error messages__. `flatparse` provides a low-level
  interface to these. Batteries are _not included_, but it should be possible for
  users to build custom solutions, which are more sophisticated, but still as fast
  as possible. In my experience, the included batteries in other libraries often come
  with major unavoidable overheads, and often we still have to extend existing machinery
  in order to scale to production features.\r\n* The __backtracking model__ of `flatparse`
  is different to parsec libraries, and is more close to the [nom](https://github.com/Geal/nom)
  library in Rust. The idea is that _parser failure_ is distinguished from _parsing
  error_. The former is used for control flow, and we can backtrack from it. The latter
  is used for unrecoverable errors, and by default it's propagated to the top. `flatparse`
  does not track whether parsers have consumed inputs. In my experience, what we really
  care about is the failure/error distinction, and in `parsec` or `megaparsec` the
  consumed/non-consumed separation is often muddled and discarded in larger parser
  implementations. By default, basic `flatparse` parsers can fail but can not throw
  errors, with the exception of the specifically error-throwing operations. Hence,
  `flatparse` users have to be mindful about grammar, and explicitly insert errors
  where it is known that the input can't be valid.\r\n\r\n`flatparse` comes in two
  flavors: [`FlatParse.Basic`](src/FlatParse/Basic.hs) and [`FlatParse.Stateful`](src/FlatParse/Stateful.hs).
  Both support a custom error type and a custom reader environment.\r\n\r\n* [`FlatParse.Basic`](src/FlatParse/Basic.hs)
  only supports the above features. If you don't need indentation parsing, this is
  sufficient.\r\n* [`FlatParse.Stateful`](src/FlatParse/Stateful.hs) additionally
  supports a built-in `Int` worth of internal state. This can support a wide range
  of indentation parsing features. There is a slight overhead in performance and code
  size compared to `Basic`. However, in small parsers and microbenchmarks the difference
  between `Basic` and `Stateful` is often reduced to near zero by GHC and LLVM optimization.
  The difference is more marked if we use native code backend instead of LLVM.\r\n\r\nThe
  reason for baking a reader into the parsers, is that if we need it, it's convenient,
  and if we don't, then GHC very reliably optimizes unused environments away. In contrast,
  GHC optimizes much less reliably if we try to wrap the existing `Reader` from `transformers`
  around our parsers.\r\n\r\n## Tutorial\r\n\r\nInformative tutorials are work in
  progress. See [`src/FlatParse/Examples`](src/FlatParse/Examples) for a lexer/parser
  example with acceptably good error messages.\r\n\r\n### Some benchmarks\r\n\r\nExecution
  times below. See source code in [bench](bench). Compiled with GHC 8.8.4 `-O2 -fllvm`.\r\n\r\n|
  \     benchmark              |  runtime   |\r\n|-----------------------------|-------------\r\n|
  s-exp/fpbasic               |  3.365 ms  |\r\n| s-exp/fpstateful            |  3.421
  ms  |\r\n| s-exp/attoparsec            |  42.84 ms  |\r\n| s-exp/megaparsec            |
  \ 57.54 ms  |\r\n| s-exp/parsec                |  179.7 ms  |\r\n| long keyword/fpbasic
  \       |  216.4 μs  |\r\n| long keyword/fpstateful     |  299.0 μs  |\r\n| long
  keyword/attoparsec     |  5.297 ms  |\r\n| long keyword/megaparsec     |  3.646
  ms  |\r\n| long keyword/parsec         |  49.18 ms  |\r\n| numeral csv/fpbasic         |
  \ 743.5 μs  |\r\n| numeral csv/fpstateful      |  848.5 μs  |\r\n| numeral csv/attoparsec
  \     |  20.64 ms  |\r\n| numeral csv/megaparsec      |  10.12 ms  |\r\n| numeral
  csv/parsec          |  78.52 ms  |\r\n\r\nObject file sizes for each module containing
  the `s-exp`, `long keyword` and `numeral csv` benchmarks.\r\n\r\n| library    |
  object file size (bytes) |\r\n| -------    | ------------------------ |\r\n| fpbasic
  \   |  26456                   |\r\n| fpstateful |  30008                   |\r\n|
  attoparsec |  83288                   |\r\n| megaparsec |  188696                  |\r\n|
  parsec     |  75880                   |\r\n"
license-name: MIT
<commit_msg>Update from Hackage at 2021-03-15T21:30:46Z<commit_after>homepage: https://github.com/AndrasKovacs/flatparse#readme
changelog-type: ''
hash: 15b4e3d5c01ce201bd99fc2d89b357e98a78fd109a6c2b0592c2dd4bb17e06b5
test-bench-deps:
  bytestring: -any
  flatparse: -any
  base: '>=4.7 && <5'
  megaparsec: -any
  parsec: -any
  gauge: -any
  attoparsec: -any
maintainer: <EMAIL>
synopsis: High-performance parsing from strict bytestrings
changelog: ''
basic-deps:
  bytestring: -any
  base: '>=4.7 && <5'
  containers: -any
  template-haskell: -any
all-versions:
- 0.1.0.0
- 0.1.0.1
author: <NAME>
latest: 0.1.0.1
description-type: markdown
description: "# flatparse\r\n\r\n![CI](https://github.com/AndrasKovacs/flatparse/actions/workflows/haskell.yml/badge.svg)\r\n\r\n`flatparse`
  is a high-performance parsing library, focusing on __programming languages__ and
  __human-readable data formats__. The \"flat\" in the name\r\nrefers to the `ByteString`
  parsing input, which has pinned contiguous data, and also to the library internals,
  which avoids indirections and heap allocations\r\nwhenever possible.\r\n\r\n## Features
  and non-features\r\n\r\n* __Excellent performance__. On microbenchmarks, `flatparse`
  is around 10 times faster than `attoparsec` or `megaparsec`. On larger examples
  with heavier use of source positions and spans and/or indentation parsing, the performance
  difference grows to 20-30 times. Compile times and exectuable sizes are also significantly
  better with `flatparse` than with `megaparsec` or `attoparsec`. `flatparse` interals
  make liberal use of unboxed tuples and GHC primops. As a result, pure validators
  (parsers returning `()`) in `flatparse` are not difficult to implement with zero
  heap allocation.\r\n* __No incremental parsing__, and __only strict `ByteString`__
  is supported as input. However, it can be still useful to convert from `Text`, `String`
  or other types to `ByteString`, and then use `flatparse` for parsing, since `flatparse`
  performance usually more than makes up for the conversion costs.\r\n* __Only little-endian
  64 bit systems are currently supported__. This may change in the future. Getting
  good performance requires architecture-specific optimizations; I've only considered
  the most common setting at this point.\r\n* __Support for fast source location handling,
  indentation parsing and informative error messages__. `flatparse` provides a low-level
  interface to these. Batteries are _not included_, but it should be possible for
  users to build custom solutions, which are more sophisticated, but still as fast
  as possible. In my experience, the included batteries in other libraries often come
  with major unavoidable overheads, and often we still have to extend existing machinery
  in order to scale to production features.\r\n* The __backtracking model__ of `flatparse`
  is different to parsec libraries, and is more close to the [nom](https://github.com/Geal/nom)
  library in Rust. The idea is that _parser failure_ is distinguished from _parsing
  error_. The former is used for control flow, and we can backtrack from it. The latter
  is used for unrecoverable errors, and by default it's propagated to the top. `flatparse`
  does not track whether parsers have consumed inputs. In my experience, what we really
  care about is the failure/error distinction, and in `parsec` or `megaparsec` the
  consumed/non-consumed separation is often muddled and discarded in larger parser
  implementations. By default, basic `flatparse` parsers can fail but can not throw
  errors, with the exception of the specifically error-throwing operations. Hence,
  `flatparse` users have to be mindful about grammar, and explicitly insert errors
  where it is known that the input can't be valid.\r\n\r\n`flatparse` comes in two
  flavors: [`FlatParse.Basic`](src/FlatParse/Basic.hs) and [`FlatParse.Stateful`](src/FlatParse/Stateful.hs).
  Both support a custom error type and a custom reader environment.\r\n\r\n* [`FlatParse.Basic`](src/FlatParse/Basic.hs)
  only supports the above features. If you don't need indentation parsing, this is
  sufficient.\r\n* [`FlatParse.Stateful`](src/FlatParse/Stateful.hs) additionally
  supports a built-in `Int` worth of internal state. This can support a wide range
  of indentation parsing features. There is a slight overhead in performance and code
  size compared to `Basic`. However, in small parsers and microbenchmarks the difference
  between `Basic` and `Stateful` is often reduced to near zero by GHC and LLVM optimization.
  The difference is more marked if we use native code backend instead of LLVM.\r\n\r\nThe
  reason for baking a reader into the parsers, is that if we need it, it's convenient,
  and if we don't, then GHC very reliably optimizes unused environments away. In contrast,
  GHC optimizes much less reliably if we try to wrap the existing `Reader` from `transformers`
  around our parsers.\r\n\r\n## Tutorial\r\n\r\nInformative tutorials are work in
  progress. See [`src/FlatParse/Examples`](src/FlatParse/Examples) for a lexer/parser
  example with acceptably good error messages.\r\n\r\n### Some benchmarks\r\n\r\nExecution
  times below. See source code in [bench](bench). Compiled with GHC 8.8.4 `-O2 -fllvm`.\r\n\r\n|
  \     benchmark              |  runtime   |\r\n|-----------------------------|-------------\r\n|
  s-exp/fpbasic               |  3.365 ms  |\r\n| s-exp/fpstateful            |  3.421
  ms  |\r\n| s-exp/attoparsec            |  42.84 ms  |\r\n| s-exp/megaparsec            |
  \ 57.54 ms  |\r\n| s-exp/parsec                |  179.7 ms  |\r\n| long keyword/fpbasic
  \       |  216.4 μs  |\r\n| long keyword/fpstateful     |  299.0 μs  |\r\n| long
  keyword/attoparsec     |  5.297 ms  |\r\n| long keyword/megaparsec     |  3.646
  ms  |\r\n| long keyword/parsec         |  49.18 ms  |\r\n| numeral csv/fpbasic         |
  \ 743.5 μs  |\r\n| numeral csv/fpstateful      |  848.5 μs  |\r\n| numeral csv/attoparsec
  \     |  20.64 ms  |\r\n| numeral csv/megaparsec      |  10.12 ms  |\r\n| numeral
  csv/parsec          |  78.52 ms  |\r\n\r\nObject file sizes for each module containing
  the `s-exp`, `long keyword` and `numeral csv` benchmarks.\r\n\r\n| library    |
  object file size (bytes) |\r\n| -------    | ------------------------ |\r\n| fpbasic
  \   |  26456                   |\r\n| fpstateful |  30008                   |\r\n|
  attoparsec |  83288                   |\r\n| megaparsec |  188696                  |\r\n|
  parsec     |  75880                   |\r\n"
license-name: MIT

{'fraction_non_alphanumeric': 0.10188794618971184, 'fraction_numerical': 0.023847741343728503, 'mean_word_length': 3.1222432262129804, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 5, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1045125', 'n_tokens_mistral': 4328, 'n_tokens_neox': 4024, 'n_words': 1517}