<commit_before>=begin

Caching IDs

The idea here is that every user in a social networking universe
has a lot of relationships to other entities that might be expensive
to discover. For example, a list of all your peers or a list of all
groups you have direct or indirect access to. So, we cache it, in
the form of serialized arrays of integers corresponding to the ids of the
foreign relationships.

If you are paying attention, you will realize this is stupid. A couple reasons
why it is not:

  (1) by using compressed integers for serialization, we can actually store
      a lot of ids without taking up too much space.
  (2) it is much faster to deserialize a big array of integers than it is
      to join in another table or make an extra query.

In many cases, all we *want* are the ids, since this is sufficient
to test membership and to display name and avatar (if ever get around
to creating a memcached for users and groups that stores [id,name,avatar_id]).

Also, we make a lot of queries of the form "group_id IN (1,2,3,4)".
This is fast, according to the mysql manual:

   If all values are constants, they are evaluated according to
   the type of expr and sorted. The search for the item then is
   done using a binary search. This means IN is very quick if
   the IN value list consists entirely of constants.

This suggests that if we stored the ids caches pre-sorted, it would be
slightly faster.

As a handy bit of fun, if any of these ids caches changes, we increment
the user's version. This can be then used to easily expire caches which
use these values.

Columns
--------

  version -- increments when any of the id caches are changed

  id caches -- there are many columns to cache our relationships,
    because they are used very frequently and take time to calculate.
    The names of the cache attributes end with "_cache".

=end

module UserExtension
  module Cache
    def self.included(base)
      base.extend ClassMethods
    end

    #
    # friendly access, in a more railsy form
    #

    def group_ids();           direct_group_id_cache;    end
    def all_group_ids();       all_group_id_cache;       end
    def admin_for_group_ids(); admin_for_group_id_cache; end
    def peer_ids();            peer_id_cache;            end
    def friend_ids();          friend_id_cache;          end
    def foe_ids();             foe_id_cache;             end
    def tag_ids();             tag_id_cache;             end

    # Be careful with this method: it is called any time a Membership
    # object is created or destroyed, and it is also called any time
    # read_attribute(*_group_id_cache) is nil. Therefore, we better
    # set the id caches to non-nil in this method unless we want to
    # recurse forever.
    def update_membership_cache(membership=nil)
      clear_access_cache
      direct, all, admin_for = get_group_ids
      peer = get_peer_ids(direct)
      update_attributes :version => version+1,
        :direct_group_id_cache => direct,
        :all_group_id_cache    => all,
        :admin_for_group_id_cache    => admin_for,
        :peer_id_cache         => peer
    end

    #
    # When our membership changes, we need to clear the peer cache of all
    # the users who might have their peer info change. To do so, this method
    # must be called in two places:
    #   1) after a new membership is created
    #   2) before a membership is destroyed
    #
    def clear_peer_cache_of_my_peers(membership=nil)
      if peer_id_cache.any?
        self.class.connection.execute(%Q[
          UPDATE users SET peer_id_cache = NULL
          WHERE id IN (#{peer_id_cache.join(',')})
        ])
      end
    end

    def increment_group_version(membership)
      membership.group.increment!(:version)
    end

    #
    # This should be called if a change in relationships has potentially
    # invalidated the cache. For example, adding or removing a commmittee.
    #
    def clear_cache
       self.class.connection.execute(%Q[
         UPDATE users SET
         tag_id_cache = NULL, direct_group_id_cache = NULL, foe_id_cache = NULL,
         peer_id_cache = NULL, friend_id_cache = NULL, all_group_id_cache = NULL,
         admin_for_group_id_cache = NULL
         WHERE id = #{self.id}
       ])
       self.write_attribute(:tag_id_cache, nil)
       self.write_attribute(:foe_id_cache, nil)
       self.write_attribute(:peer_id_cache, nil)
       self.write_attribute(:friend_id_cache, nil)
       self.write_attribute(:direct_group_id_cache, nil)
       self.write_attribute(:all_group_id_cache, nil)
       self.write_attribute(:admin_for_group_id_cache, nil)
       self.clear_access_cache
    end

    # called whenever an empty self.friend_id_cache is accessed
    # or directly when a new contact is added
    def update_contacts_cache()
      friend,foe = get_contact_ids
      update_attributes :version => version+1,
        :friend_id_cache => friend,
        :foe_id_cache    => foe
    end

    # include direct memberships, committees, and networks
    def get_group_ids
      if self.id
        direct = Group.connection.select_values(%Q[
          SELECT groups.id FROM groups
          INNER JOIN memberships ON groups.id = memberships.group_id
          WHERE (memberships.user_id = #{self.id})
        ])
      else
        direct = []
      end
      if direct.any?
        committee = Group.connection.select_values(%Q[
          SELECT groups.id FROM groups
          WHERE groups.parent_id IN (#{direct.join(',')})
          AND groups.type = 'Committee'
        ])
        network = Group.connection.select_values(%Q[<commit_msg>write_attribute is private - so don't call it on self<commit_after>=begin

Caching IDs

The idea here is that every user in a social networking universe
has a lot of relationships to other entities that might be expensive
to discover. For example, a list of all your peers or a list of all
groups you have direct or indirect access to. So, we cache it, in
the form of serialized arrays of integers corresponding to the ids of the
foreign relationships.

If you are paying attention, you will realize this is stupid. A couple reasons
why it is not:

  (1) by using compressed integers for serialization, we can actually store
      a lot of ids without taking up too much space.
  (2) it is much faster to deserialize a big array of integers than it is
      to join in another table or make an extra query.

In many cases, all we *want* are the ids, since this is sufficient
to test membership and to display name and avatar (if ever get around
to creating a memcached for users and groups that stores [id,name,avatar_id]).

Also, we make a lot of queries of the form "group_id IN (1,2,3,4)".
This is fast, according to the mysql manual:

   If all values are constants, they are evaluated according to
   the type of expr and sorted. The search for the item then is
   done using a binary search. This means IN is very quick if
   the IN value list consists entirely of constants.

This suggests that if we stored the ids caches pre-sorted, it would be
slightly faster.

As a handy bit of fun, if any of these ids caches changes, we increment
the user's version. This can be then used to easily expire caches which
use these values.

Columns
--------

  version -- increments when any of the id caches are changed

  id caches -- there are many columns to cache our relationships,
    because they are used very frequently and take time to calculate.
    The names of the cache attributes end with "_cache".

=end

module UserExtension
  module Cache
    def self.included(base)
      base.extend ClassMethods
    end

    #
    # friendly access, in a more railsy form
    #

    def group_ids();           direct_group_id_cache;    end
    def all_group_ids();       all_group_id_cache;       end
    def admin_for_group_ids(); admin_for_group_id_cache; end
    def peer_ids();            peer_id_cache;            end
    def friend_ids();          friend_id_cache;          end
    def foe_ids();             foe_id_cache;             end
    def tag_ids();             tag_id_cache;             end

    # Be careful with this method: it is called any time a Membership
    # object is created or destroyed, and it is also called any time
    # read_attribute(*_group_id_cache) is nil. Therefore, we better
    # set the id caches to non-nil in this method unless we want to
    # recurse forever.
    def update_membership_cache(membership=nil)
      clear_access_cache
      direct, all, admin_for = get_group_ids
      peer = get_peer_ids(direct)
      update_attributes :version => version+1,
        :direct_group_id_cache => direct,
        :all_group_id_cache    => all,
        :admin_for_group_id_cache    => admin_for,
        :peer_id_cache         => peer
    end

    #
    # When our membership changes, we need to clear the peer cache of all
    # the users who might have their peer info change. To do so, this method
    # must be called in two places:
    #   1) after a new membership is created
    #   2) before a membership is destroyed
    #
    def clear_peer_cache_of_my_peers(membership=nil)
      if peer_id_cache.any?
        self.class.connection.execute(%Q[
          UPDATE users SET peer_id_cache = NULL
          WHERE id IN (#{peer_id_cache.join(',')})
        ])
      end
    end

    def increment_group_version(membership)
      membership.group.increment!(:version)
    end

    #
    # This should be called if a change in relationships has potentially
    # invalidated the cache. For example, adding or removing a commmittee.
    #
    def clear_cache
       self.class.connection.execute(%Q[
         UPDATE users SET
         tag_id_cache = NULL, direct_group_id_cache = NULL, foe_id_cache = NULL,
         peer_id_cache = NULL, friend_id_cache = NULL, all_group_id_cache = NULL,
         admin_for_group_id_cache = NULL
         WHERE id = #{self.id}
       ])
       write_attribute(:tag_id_cache, nil)
       write_attribute(:foe_id_cache, nil)
       write_attribute(:peer_id_cache, nil)
       write_attribute(:friend_id_cache, nil)
       write_attribute(:direct_group_id_cache, nil)
       write_attribute(:all_group_id_cache, nil)
       write_attribute(:admin_for_group_id_cache, nil)
       self.clear_access_cache
    end

    # called whenever an empty self.friend_id_cache is accessed
    # or directly when a new contact is added
    def update_contacts_cache()
      friend,foe = get_contact_ids
      update_attributes :version => version+1,
        :friend_id_cache => friend,
        :foe_id_cache    => foe
    end

    # include direct memberships, committees, and networks
    def get_group_ids
      if self.id
        direct = Group.connection.select_values(%Q[
          SELECT groups.id FROM groups
          INNER JOIN memberships ON groups.id = memberships.group_id
          WHERE (memberships.user_id = #{self.id})
        ])
      else
        direct = []
      end
      if direct.any?
        committee = Group.connection.select_values(%Q[
          SELECT groups.id FROM groups
          WHERE groups.parent_id IN (#{direct.join(',')})
          AND groups.type = 'Committee'
        ])
        network = Group.connection.select_values(%Q[
{'fraction_non_alphanumeric': 0.06094848296111754, 'fraction_numerical': 0.001779517750689563, 'mean_word_length': 2.6973684210526314, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5088804', 'n_tokens_mistral': 3185, 'n_tokens_neox': 3106, 'n_words': 1269}