<commit_before>homepage: https://github.com/chessai/freq
changelog-type: ''
hash: 18e97888f7f9990972b55198841635f63d8948a796b5c34a640950a92b9aed77
test-bench-deps:
  bytestring: -any
  base: ! '>=4.7 && <5'
  hedgehog: -any
  gauge: -any
  containers: -any
  freq: -any
maintainer: <EMAIL>
synopsis: Are you ready to get freaky?
changelog: ''
basic-deps:
  bytestring: -any
  base: ! '>=4.9 && <5.0'
  containers: -any
  freq: -any
  primitive: ! '>=0.6.1'
all-versions:
- '0.0.0'
- '0.1.0.0'
- '0.1.0.1'
author: <NAME>
latest: '0.1.0.1'
description-type: markdown
description: ! "# freq\n\n## About\nThis is a simple cryptanalytic frequency analysis
  tool that uses [english character digrams](https://en.wikipedia.org/wiki/Bigram)
  as a probabilistic model for scoring ByteStrings according to their randomness (0..1,
  0 being the most random, 1 being the least random).\n\n## Uses\nI currently use
  this to validate domain names, and so the training data available consists of about
  6.5 Megabytes of Public Domain 19th and 20th century English novels. You can feed
  any training data you wish to 'freq' to achieve different results.\n\n## Improvements\nTo
  improve further on the accuracy of this approach, I will experiment with a generalised
  dynamic n-gram approach, which will sacrifice slightly in performance for a potentially
  large gain in accuracy. \n\n## Further\nSee my implementation of the [Linear Hadamard
  Spectral Test](https://github.com/chessai/lhst) for a different approach to a similar
  problem with much higher variance in the potentially random data, where a trained
  approach might be less accurate.\n"
license-name: MIT<commit_msg>Update from Hackage at 2018-05-24T14:11:44Z<commit_after>homepage: https://github.com/chessai/freq
changelog-type: ''
hash: ca409f19c7a99abdf933db51c48753e09c07a2b5fa089d61ceb1a4834e8dcb95
test-bench-deps:
  bytestring: -any
  base: ! '>=4.7 && <5'
  hedgehog: -any
  gauge: -any
  containers: -any
  freq: -any
maintainer: <EMAIL>
synopsis: Are you ready to get freaky?
changelog: ''
basic-deps:
  bytestring: -any
  base: ! '>=4.9 && <5.0'
  containers: -any
  freq: -any
  primitive: ! '>=0.6.1'
all-versions:
- '0.0.0'
- '0.1.0.0'
- '0.1.0.1'
- '0.1.0.2'
author: <NAME>
latest: '0.1.0.2'
description-type: markdown
description: ! "# freq\n\n## About\nThis is a simple cryptanalytic frequency analysis
  tool that uses [english character digrams](https://en.wikipedia.org/wiki/Bigram)
  as a probabilistic model for scoring ByteStrings according to their randomness (0..1,
  0 being the most random, 1 being the least random).\n\n## Uses\nI currently use
  this to validate domain names, and so the training data available consists of about
  6.5 Megabytes of Public Domain 19th and 20th century English novels. You can feed
  any training data you wish to 'freq' to achieve different results.\n\n## Improvements\nTo
  improve further on the accuracy of this approach, I will experiment with a generalised
  dynamic n-gram approach, which will sacrifice slightly in performance for a potentially
  large gain in accuracy. \n\n## Further\nSee my implementation of the [Linear Hadamard
  Spectral Test](https://github.com/chessai/lhst) for a different approach to a similar
  problem with much higher variance in the potentially random data, where a trained
  approach might be less accurate.\n"
license-name: MIT
{'fraction_non_alphanumeric': 0.10395747194329592, 'fraction_numerical': 0.05138806851742469, 'mean_word_length': 4.077961019490255, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 7, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4972336', 'n_tokens_mistral': 1209, 'n_tokens_neox': 1128, 'n_words': 400}