<commit_before>= Wordlist

* http://wordlist.rubyforge.org/
* http://github.com/sophsec/wordlist/
* Postmodern (postmodern.mod3 at gmail.com)

== DESCRIPTION:

A Ruby library for generating and working with wordlists. Wordlist allows
one to efficiently generate unique wordlists from arbitrary text or
other sources, such as website content. Wordlist can also quickly enumerate
through words within an existing wordlist, applying mutation rules to each
word.

== FEATURES:

* Uses a bucket system of CRC32 hashes for efficient filtering of duplicate
  words.
* Provides Wordlist::Mutations for defining mutations to apply to words.
* Supports building wordlists from arbitrary text.
* Supports custom builders:
  * Wordlist::Builders::Website: Build word lists from web-site content.
* Supports custom wordlist formats:
  * Wordlist::Formats::FlatFile: Enumerates through the words in a flat-file
    wordlist.

== EXAMPLES:

* Build a wordlist from arbitrary text:

    Wordlist::Builder.build('list.txt') do |builder|
      builder.parse(some_text)
    end

* Build a wordlist from another file:

    Wordlist::Builder.build('list.txt') do |builder|
      builder.parse_file('some/file.txt')
    end

* Build a wordlist from content off a website:

    require 'wordlist/builders/website'

    Wordlist::Builders::Website.build('list.txt', :host => 'www.example.com')

* Enumerate through each word in a flat-file wordlist:

    require 'wordlist/flat_file'

    list = Wordlist::FlatFile.new('list.txt')
    list.each_word do |word|
      puts word
    end

* Enumerate through each unique word in a flat-file wordlist:

    list.each_unique do |word|
      puts word
    end

* Define mutation rules, and enumerate through each unique mutation of each
  unique word in the word-list:

    list.mutate 'o', '0'
    list.mutate 'a', 0x41
    list.mutate /[hax]/i, lambda { |match| match.swapcase }

    list.each_mutation do |word|
      puts word
    end

== REQUIREMENTS:

* {spidr}[http://spidr.rubyforge.org] >= 0.1.9

== INSTALL:
<commit_msg>Updated an example in the README.<commit_after>= Wordlist

* http://wordlist.rubyforge.org/
* http://github.com/sophsec/wordlist/
* Postmodern (postmodern.mod3 at gmail.com)

== DESCRIPTION:

A Ruby library for generating and working with wordlists. Wordlist allows
one to efficiently generate unique wordlists from arbitrary text or
other sources, such as website content. Wordlist can also quickly enumerate
through words within an existing wordlist, applying mutation rules to each
word.

== FEATURES:

* Uses a bucket system of CRC32 hashes for efficient filtering of duplicate
  words.
* Provides Wordlist::Mutations for defining mutations to apply to words.
* Supports building wordlists from arbitrary text.
* Supports custom builders:
  * Wordlist::Builders::Website: Build word lists from web-site content.
* Supports custom wordlist formats:
  * Wordlist::Formats::FlatFile: Enumerates through the words in a flat-file
    wordlist.

== EXAMPLES:

* Build a wordlist from arbitrary text:

    Wordlist::Builder.build('list.txt') do |builder|
      builder.parse(some_text)
    end

* Build a wordlist from another file:

    Wordlist::Builder.build('list.txt') do |builder|
      builder.parse_file('some/file.txt')
    end

* Build a wordlist from content off a website:

    require 'wordlist/builders/website'

    Wordlist::Builders::Website.build('list.txt', :host => 'www.example.com')

* Enumerate through each word in a flat-file wordlist:

    require 'wordlist/flat_file'

    list = Wordlist::FlatFile.new('list.txt')
    list.each_word do |word|
      puts word
    end

* Enumerate through each unique word in a flat-file wordlist:

    list.each_unique do |word|
      puts word
    end

* Define mutation rules, and enumerate through each unique mutation of each
  unique word in the word-list:

    list.mutate 'o', '0'
    list.mutate 'a', 0x41
    list.mutate(/[hax]/i) { |match| match.swapcase }

    list.each_mutation do |word|
      puts word
    end

== REQUIREMENTS:

* {spidr}[http://spidr.rubyforge.org] >= 0.1.9

== INSTALL:

{'fraction_non_alphanumeric': 0.09584042811967891, 'fraction_numerical': 0.004864996351252737, 'mean_word_length': 3.753757225433526, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '440344', 'n_tokens_mistral': 1304, 'n_tokens_neox': 1243, 'n_words': 425}