<commit_before>=begin

Some notes on PageTerms
-----------------------

See PageExtension::Index for most the code dealing with PageTerms.

PageTerms holds searching information for a page. Every Page has exactly one
PageTerms object and vice-versa. 

PageTerms has three uses:

(1) Sphinx full-text searching

In order to index for sphinx, we need to be able to run a sql query without ruby
logic. The page_terms table holds a copy of all the stuff we want to index for a
particular page. It turns out that this is 100 times faster to keep a copy of
this stuff in one table than trying to do complex queries on a sphinx reindex.

(2) Page permissions

PageTerms is also used to filter by access restrictions in non-sphinx page
queries. We could just use sphinx for all queries, but the problem with this is
that there are many situations in which the sphinx index could be out of date. 

The page_terms table is a MyISAM table that has a fulltext index on
(access_ids, tag_ids). It must be MyISAM for fulltext index. It is important
to note that the fulltext index is a compound index and will ONLY work if BOTH
columns are matched against in the query. This is very important!

(3) Permissions for stuff that inherits page permissions

Sometimes you want to search directly on a thing, without having to go through
Page to get at the thing. For example, search directly for assets or tasks. But
you also only want to return results for pages that the user has access to. In
these cases, these tables hold a reference to page_terms and then the query just
joins in the page_terms table and does a match against (access_ids, tag_ids) like
in a normal page query.

=end

class PageTerms < ActiveRecord::Base
  belongs_to :page

  define_index do
    begin
      ## text fields ##

      # general fields
      indexes :title,     :sortable => true
      indexes :page_type, :sortable => true
      indexes :tags
      indexes :body
      indexes :comments

      # denormalized names
      indexes :created_by_login, :sortable => true
      indexes :updated_by_login, :sortable => true
      indexes :group_name,       :sortable => true

      ## attributes ##

      # timedates
      has :page_created_at
      has :page_updated_at
      has :starts_at
      has :ends_at
      has :views

      # ids
      has :created_by_id
      has :updated_by_id<commit_msg>fixed column name of views_count in page_terms.rb<commit_after>=begin

Some notes on PageTerms
-----------------------

See PageExtension::Index for most the code dealing with PageTerms.

PageTerms holds searching information for a page. Every Page has exactly one
PageTerms object and vice-versa. 

PageTerms has three uses:

(1) Sphinx full-text searching

In order to index for sphinx, we need to be able to run a sql query without ruby
logic. The page_terms table holds a copy of all the stuff we want to index for a
particular page. It turns out that this is 100 times faster to keep a copy of
this stuff in one table than trying to do complex queries on a sphinx reindex.

(2) Page permissions

PageTerms is also used to filter by access restrictions in non-sphinx page
queries. We could just use sphinx for all queries, but the problem with this is
that there are many situations in which the sphinx index could be out of date. 

The page_terms table is a MyISAM table that has a fulltext index on
(access_ids, tag_ids). It must be MyISAM for fulltext index. It is important
to note that the fulltext index is a compound index and will ONLY work if BOTH
columns are matched against in the query. This is very important!

(3) Permissions for stuff that inherits page permissions

Sometimes you want to search directly on a thing, without having to go through
Page to get at the thing. For example, search directly for assets or tasks. But
you also only want to return results for pages that the user has access to. In
these cases, these tables hold a reference to page_terms and then the query just
joins in the page_terms table and does a match against (access_ids, tag_ids) like
in a normal page query.

=end

class PageTerms < ActiveRecord::Base
  belongs_to :page

  define_index do
    begin
      ## text fields ##

      # general fields
      indexes :title,     :sortable => true
      indexes :page_type, :sortable => true
      indexes :tags
      indexes :body
      indexes :comments

      # denormalized names
      indexes :created_by_login, :sortable => true
      indexes :updated_by_login, :sortable => true
      indexes :group_name,       :sortable => true

      ## attributes ##

      # timedates
      has :page_created_at
      has :page_updated_at
      has :starts_at
      has :ends_at
      has :views_count

      # ids
      has :created_by_id
      has :updated_by_id
{'fraction_non_alphanumeric': 0.04968553459119497, 'fraction_numerical': 0.0025157232704402514, 'mean_word_length': 3.199823943661972, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4192750', 'n_tokens_mistral': 1340, 'n_tokens_neox': 1286, 'n_words': 693}