<commit_before># Database-agnostic keyword searching routine
#
# Keywords have the format keyword:value
# Keywords can also be negated with -, as in -keyword:value
# Values are comma-separated, while keywords are space-separated
# See https://github.com/bruce/keyword_search for more information
#
# Subclasses must set the initial_relation, search_proc and sortable_fields class variables
#
#   initial_relation - the ActiveRecord::Relation that contains all
#                      records to be searched, usually ClassName.unscoped
#
#   search_proc - a proc passed to keyword_search's `search` method
#                 it receives keyword_search's `with` object as argument
#                 this proc must define the `keyword` blocks for keyword_search
#                 the relation to be scoped is contained in the @items instance variable
#
#   The `to_string_array` helper can help with parsing strings from the query
#
#   sortable_fields_map - a Hash that maps the lowercase names of fields
#                         which can be used to sort the results to symbols
#                         for their respective database columns
#                         keys are lowercase strings that should be allowed
#                         in options[:order_by]
#                         values are the corresponding database column names
#                         that will be passed to the order() method
#
# Callers of subclass routines provides a query argument and an options hash
#
# Required arguments:
#
#   query - a string that follows the keyword format above
#
# Options hash:
#
#     Ordering:
#
#       :order_by - list of fields to sort by, with optional sort directions
#                   can be a String, Array of Strings or Array of Hashes
#                   default: {:created_at => :asc}
#
#     Pagination:
#
#       :per_page - the maximum number of results per page - default: nil (disabled)
#       :page     - the page to return - default: 1
#
# This routine's output contains:
#
#   outputs[:items] - an ActiveRecord::Relation that matches the query terms and options
#
# You can use the following expression to obtain the<commit_msg>More explanation about the last change<commit_after># Database-agnostic keyword searching routine
#
# Keywords have the format keyword:value
# Keywords can also be negated with -, as in -keyword:value
# Values are comma-separated, while keywords are space-separated
# See https://github.com/bruce/keyword_search for more information
#
# Subclasses must set the initial_relation, search_proc and sortable_fields class variables
#
#   initial_relation - the ActiveRecord::Relation that contains all
#                      records to be searched, usually ClassName.unscoped
#
#   search_proc - a proc passed to keyword_search's `search` method
#                 it receives keyword_search's `with` object as argument
#                 this proc must define the `keyword` blocks for keyword_search
#                 the relation to be scoped is contained in the @items instance variable
#
#   The `to_string_array` helper can help with parsing strings from the query
#
#   sortable_fields_map - a Hash that maps the lowercase names of fields
#                         which can be used to sort the results to symbols
#                         for their respective database columns
#                         keys are lowercase strings that should be allowed
#                         in options[:order_by]
#                         values are the corresponding database column names
#                         that will be passed to the order() method
#                         columns from other tables can be specified either
#                         through Arel attributes (Class.arel_table[:column])
#                         or through literal strings
#
# Callers of subclass routines provides a query argument and an options hash
#
# Required arguments:
#
#   query - a string that follows the keyword format above
#
# Options hash:
#
#     Ordering:
#
#       :order_by - list of fields to sort by, with optional sort directions
#                   can be a String, Array of Strings or Array of Hashes
#                   default: {:created_at => :asc}
#
#     Pagination:
#
#       :per_page - the maximum number of results per page - default: nil (disabled)
#       :page     - the page to return - default: 1
#
# This routine's output contains:
#
#   outputs[:items] - an ActiveRecord::Relation that matches the query terms and options
#
# You can use the following expression to obtain the
{'fraction_non_alphanumeric': 0.05767543859649123, 'fraction_numerical': 0.0004385964912280702, 'mean_word_length': 2.4293233082706767, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2561123', 'n_tokens_mistral': 1105, 'n_tokens_neox': 1059, 'n_words': 506}