<commit_before># EWAH Bitsets in Ruby

Enhanced Word Aligned Hybrid (EWAH) encoding is a compression strategy for bitmaps. The compressed bitmaps can be scanned over and operated on in place which results in extremely fast processing times even while maintaining high compression rates.

This library wraps the original C++ implementation written by one of the original paper authors <NAME> which is hosted here:

https://github.com/lemire/EWAHBoolArray

Examples are in the spec but the general idea goes something like this:

    require 'ewahbitset'
    
    bitset = EwahBitset.new
    0.upto(10) do |i|
      bitset.set(i * 10)
    end
    
    positions = []
    bitset.each do |position|
      positions << position
    end
    <commit_msg>put install instructions<commit_after># EWAH Bitsets in Ruby

Enhanced Word Aligned Hybrid (EWAH) encoding is a compression strategy for bitmaps. The compressed bitmaps can be scanned over and operated on in place which results in extremely fast processing times even while maintaining high compression rates.

This library wraps the original C++ implementation written by one of the original paper authors <NAME> which is hosted here:

https://github.com/lemire/EWAHBoolArray

The gem is built and tested against ruby 1.8.7. If anyone wants to modify it to also run in ruby 1.9.x I'd gladly accept the pull requests.

To install simply run

    gem install ewah-bitset

Examples are in the spec but the general idea goes something like this:

    require 'ewahbitset'
    
    bitset = EwahBitset.new
    0.upto(10) do |i|
      bitset.set(i * 10)
    end
    
    positions = []
    bitset.each do |position|
      positions << position
    end
    
{'fraction_non_alphanumeric': 0.053003533568904596, 'fraction_numerical': 0.0088339222614841, 'mean_word_length': 3.3901808785529717, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4054293', 'n_tokens_mistral': 486, 'n_tokens_neox': 462, 'n_words': 222}