<commit_before>What the bot does:
------------------

1. Keeps track of quit/joins over time.
2. Answers questions like how many regulars you've lost over a given time-span.
3. Answers questions about who is and isn't a regular member of the community.
4. Keep track of who interacts with who, using highlights
5. Signal/Noise Detection
 * Watch for lots of users highlighting one user in a short timeframe, and flag the single user.
 * Emotion detection from text. http://arxiv.org/pdf/1205.4944.pdf
6. Automatable temp-ban timers
 * Use builtin time library to parse user-inputted ban duration.
 * Possibly only check for ban durations along certain timespans (every minute versus every hour) depending on what kinds of bans are in the unban-queue. 
 
 7. Report feature (for users to report misbehavior), including message. 
Make sure it works in PM. Allow ops to opt-in to recieving PMs from the bot. 
Only regulars can use report feature. Optional feature to let user specify that 
their report will only be shown if others report too, so they aren't singled out 
for hostility. Anonymous reports by default, usernames can be revealed with the 
approval of n mods where n is more than one. (Configurable.)
 
 8. When a regular is banned from channel, PM the op that banned them so they can put a statement on record as to why.
 
 9. Lurker detection, making sure to filter for accidental pings and comments on nicks/hostmasks and the like. 
(Watch for the thing where people will accidentally autocomplete the wrong name.) 
Values of user engagement between 'lurker' and 'regular', continuous sliding scale.
 
 10. Auditbot should be able to detect when it is being used to flood channel.
 
 11. If emotion detection is good enough, warn users that are being overly aggressive. 
Possibly an autogenerated report as well, assuming the systems involved are reliable 
enough for such reports to not be spam. (If we do automated reports, we should 
have seperate ways for ops to turn off recieving user-generated reports and auto-generated reports.)
 
 12. Auditbot should be able to detect controversial or 'banned' discussions when 
they're brought up and figure out who consistently starts conversations about such topics, 
even going so far as to file autogenerated reports.
 
 13. Personalized auto-reports for ops (that are only seen by that op) that are generated whenever certain words or phrases are used. 
N+1. Get statistics about what time of day people use the channel by doing GeoIP lookups on hostmasks, and then checking what times they're active

Regulars and normalization:
---------------------------

One of the functions of auditbot is to keep track of who is and isn't a normal 
feature of the channel. For example, when a large number of trolls invade an IRC 
channel, it can be difficult to distinguish trolls from 'lurkers' that don't talk much. 
Auditbot solves this problem by letting operators see who's new, who's been around 
for a while but hasn't talked, and who channel 'regulars' are. 

Before proceeding further it would be best to talk about what a 'regular' *is* from the perspective auditbot. A regular is:

1. A regular is somebody who is normalized in channel, that nobody would say "Oh hi never seen you before" to.

2. Regularity is not a measure of goodness or conduct, trolls can be regulars.

3. Regulars generally have conversations with other channel participants, distinguishing them from a lurker.

Obviously auditbot cannot reliably determine the first for itself, so it uses 
crude metrics to whitelist people instead. These metrics are a deliberately low 
bar so as not to invite arguments over who is 'regular' in channel and who is not. 
Further, operators can grant users regular status if the metrics used by auditbot have missed somebody.

Commands:
---------

Each auditbot command is a keyword with a space seperated list of arguments. 
An asterick by an argument means that it is context sensitive. Arguments in 
square brackets have multiple possible <WORD> and the brackets enumerate the 
possibilities. Keywords are special words you can pass instead of an argument 
that do things like list every user that is new to the channel, you use a keyword 
by prepending it with the asterick character, to avoid confusion with a nick or channel.

Auditbot recognizes the following commands:

activity - show auditbots evaluation of activity for a user in a given channel

SYNTAX: activity <channel>* [<users>, keyword(s)]

KEYWORDS: 

"regulars": Show all users that auditbot considers to be normalized within channel and active.

"lurkers": Show all users that auditbot considers to be normalized within channel and inactive.

"new": Show all users that auditbot has not whitelisted through the other two methods.

parted - show users who have parted channel

SYNTAX: parted <channel>* keyword(s)

KEYWORDS:

"all": Show every user that auditbot has ever recorded parting channel.

"since" <date>: Show every user that has parted since date.



A table of time marks:

y - Years.

M - Months.

d - Days.

h - Hours.

m - Minutes.

"since" takes one time argument, which is a number followed by the time mark.

"

Auditbot Architecture:
---------

Auditbot has three layers to its functioning:

IRC Server Connection/command send layer -> Auditbot control -> Auditbot Components

The IRC Server Connection is managed by server ojbects.

Auditbot control is the mainloop of the program, implementing the bare basics of the bot that are used to bootstrap the advanced features and managing the components.

Auditbot components are the meat of the program, implementing features such as autogenerated reports and user profiling.

Server Objects:
---------------

A connection between auditbot and a server is mediated by a *server object*. The 
server object maintains the connection and facilitates message passing between auditbot and the server. 
The server object has methods for auditbot to send commands and messages to the server, and recieve messages and responses back.

A server object keeps the hostname of the server it's connected to as internal state.

Server objects have the following methods:

.raw_send(line):

Take a line of text and sends it to the server as a raw command.

.quit():

Send a QUIT command to server, ending the connection.

.nick(nickname):

Change the nickname of the client.

.user(username, mode, realname):

Register with the server as a given user.

.mode(modifier, mode)

Change the clients usermode, with modifier being the addition or subtraction of a mode and mode being which to add or subtract.

.join(channels):

Send a JOIN command to the server, with channels as the target.

.part(channels):

Send a PART command to the server, with channels as the channels to part from.

.raw_message(target, text):

The low-level privmsg send method. Sends a line of text to target without embellishment or splitlines.

.message(target, text):

Send message to target, splitting lines if text is longer than 510 characters.

.notice(target, text):

Send a notice to target, messages to which a client is not supposed to reply.

.ison(nicks):

Takes a list of nicks and returns the set which are online. Can only display nicks online up to 510 characters in a line, minus the amount the server sends to prepend the list of online nicks.

Auditbot Programming API:
-------------------------

The key question when designing an API to be used with auditbot is what sorts of input the components of auditbot should expect to deal with.

Inputs and interaction:

All IRC traffic comes in through the same tube, where it can be split into three categories:
1. Messages addressed to a channel which auditbot is a part of.
2. Messages addressed to auditbot directly from a specific user.
3. Messages from the server that do not otherwise fall into the first two categories.
Of messages from a channel and messages from a specific user, both have the potential to be a command, if it is not a command then it is passed to the systems facilities for dealing with passive messages.
Messages from the server are never commands.
(The most cases are covered by assuming that one user is an op in multiple channels that auditbot is in, and another user is not an op or otherwise privileged in any channels auditbot is in.)



<commit_msg>I don't know what I actually changed in the project file it was a while ago.<commit_after>What the bot does:
------------------

1. Keeps track of quit/joins over time.
2. Answers questions like how many regulars you've lost over a given time-span.
3. Answers questions about who is and isn't a regular member of the community.
4. Keep track of who interacts with who, using highlights
5. Signal/Noise Detection
 * Watch for lots of users highlighting one user in a short timeframe, and flag the single user.
 * Emotion detection from text. http://arxiv.org/pdf/1205.4944.pdf
6. Automatable temp-ban timers
 * Use builtin time library to parse user-inputted ban duration.
 * Possibly only check for ban durations along certain timespans (every minute versus every hour) depending on what kinds of bans are in the unban-queue. 
 
 7. Report feature (for users to report misbehavior), including message. 
Make sure it works in PM. Allow ops to opt-in to recieving PMs from the bot. 
Only regulars can use report feature. Optional feature to let user specify that 
their report will only be shown if others report too, so they aren't singled out 
for hostility. Anonymous reports by default, usernames can be revealed with the 
approval of n mods where n is more than one. (Configurable.)
 
 8. When a regular is banned from channel, PM the op that banned them so they can put a statement on record as to why.
 
 9. Lurker detection, making sure to filter for accidental pings and comments on nicks/hostmasks and the like. 
(Watch for the thing where people will accidentally autocomplete the wrong name.) 
Values of user engagement between 'lurker' and 'regular', continuous sliding scale.
 
 10. Auditbot should be able to detect when it is being used to flood channel.
 
 11. If emotion detection is good enough, warn users that are being overly aggressive. 
Possibly an autogenerated report as well, assuming the systems involved are reliable 
enough for such reports to not be spam. (If we do automated reports, we should 
have seperate ways for ops to turn off recieving user-generated reports and auto-generated reports.)
 
 12. Auditbot should be able to detect controversial or 'banned' discussions when 
they're brought up and figure out who consistently starts conversations about such topics, 
even going so far as to file autogenerated reports.
 
 13. Personalized auto-reports for ops (that are only seen by that op) that are generated whenever certain words or phrases are used. 
N+1. Get statistics about what time of day people use the channel by doing GeoIP lookups on hostmasks, and then checking what times they're active

Regulars and normalization:
---------------------------

One of the functions of auditbot is to keep track of who is and isn't a normal 
feature of the channel. For example, when a large number of trolls invade an IRC 
channel, it can be difficult to distinguish trolls from 'lurkers' that don't talk much. 
Auditbot solves this problem by letting operators see who's new, who's been around 
for a while but hasn't talked, and who channel 'regulars' are. 

Before proceeding further it would be best to talk about what a 'regular' *is* from the perspective auditbot. A regular is:

1. A regular is somebody who is normalized in channel, that nobody would say "Oh hi never seen you before" to.

2. Regularity is not a measure of goodness or conduct, trolls can be regulars.

3. Regulars generally have conversations with other channel participants, distinguishing them from a lurker.

Obviously auditbot cannot reliably determine the first for itself, so it uses 
crude metrics to whitelist people instead. These metrics are a deliberately low 
bar so as not to invite arguments over who is 'regular' in channel and who is not. 
Further, operators can grant users regular status if the metrics used by auditbot have missed somebody.

To determine if users meet the metrics, auditbot keeps every time a user speaks 
on record and 'plots' it along a timescale. It's not enough to just speak to be 
considered a regular by auditbot, a user must have a conversation while they are
in channel. (Below we get further into exactly what constitutes a 'conversation'
for auditbots purposes.) Auditbot evaluates users on a short term and long term basis.
If you have a certain number of conversations within a week, you're a regular.
If you have a certain number of conversations within a month or a year, you're a regular.

When using the 'activity' command to look at the activity of a user measured 
like this, auditbot will give an operator the following statistics:

User| Week: 0/7 Month: 0/30 Year: 0/12

Where week is how many days of this week the user had a conversation.
Month is how many days of this month the user had a conversation.
Year is how many months of this year the user had a conversation.

Auditbots crude metrics are as follows:

1. 

Commands:
---------

Each auditbot command is a keyword with a space seperated list of arguments. 
An asterick by an argument means that it is context sensitive. Arguments in 
square brackets have multiple possible <WORD> and the brackets enumerate the 
possibilities. Keywords are special words you can pass instead of an argument 
that do things like list every user that is new to the channel, you use a keyword 
by prepending it with the asterick character, to avoid confusion with a nick or channel.

Auditbot recognizes the following commands:

activity - show auditbots evaluation of activity for a user in a given channel

SYNTAX: activity <channel>* [<users>, keyword(s)]

KEYWORDS: 

"regulars": Show all users that auditbot considers to be normalized within channel and active.

"lurkers": Show all users that auditbot considers to be normalized within channel and inactive.

"new": Show all users that auditbot has not whitelisted through the other two methods.

parted - show users who have parted channel

SYNTAX: parted <channel>* keyword(s)

KEYWORDS:

"all": Show every user that auditbot has ever recorded parting channel.

"since" <date>: Show every user that has parted since date.



A table of time marks:

y - Years.

M - Months.

d - Days.

h - Hours.

m - Minutes.

"since" takes one time argument, which is a number followed by the time mark.

"

Auditbot Architecture:
---------

Auditbot has three layers to its functioning:

IRC Server Connection/command send layer -> Auditbot control -> Auditbot Components

The IRC Server Connection is managed by server ojbects.

Auditbot control is the mainloop of the program, implementing the bare basics of the bot that are used to bootstrap the advanced features and managing the components.

Auditbot components are the meat of the program, implementing features such as autogenerated reports and user profiling.

Server Objects:
---------------

A connection between auditbot and a server is mediated by a *server object*. The 
server object maintains the connection and facilitates message passing between auditbot and the server. 
The server object has methods for auditbot to send commands and messages to the server, and recieve messages and responses back.

A server object keeps the hostname of the server it's connected to as internal state.

Server objects have the following methods:

.raw_send(line):

Take a line of text and sends it to the server as a raw command.

.quit():

Send a QUIT command to server, ending the connection.

.nick(nickname):

Change the nickname of the client.

.user(username, mode, realname):

Register with the server as a given user.

.mode(modifier, mode)

Change the clients usermode, with modifier being the addition or subtraction of a mode and mode being which to add or subtract.

.join(channels):

Send a JOIN command to the server, with channels as the target.

.part(channels):

Send a PART command to the server, with channels as the channels to part from.

.raw_message(target, text):

The low-level privmsg send method. Sends a line of text to target without embellishment or splitlines.

.message(target, text):

Send message to target, splitting lines if text is longer than 510 characters.

.notice(target, text):

Send a notice to target, messages to which a client is not supposed to reply.

.ison(nicks):

Takes a list of nicks and returns the set which are online. Can only display nicks online up to 510 characters in a line, minus the amount the server sends to prepend the list of online nicks.

Auditbot Programming API:
-------------------------

The key question when designing an API to be used with auditbot is what sorts of input the components of auditbot should expect to deal with.

Inputs and interaction:

All IRC traffic comes in through the same tube, where it can be split into three categories:
1. Messages addressed to a channel which auditbot is a part of.
2. Messages addressed to auditbot directly from a specific user.
3. Messages from the server that do not otherwise fall into the first two categories.
Of messages from a channel and messages from a specific user, both have the potential to be a command, if it is not a command then it is passed to the systems facilities for dealing with passive messages.
Messages from the server are never commands.
(The most cases are covered by assuming that one user is an op in multiple channels that auditbot is in, and another user is not an op or otherwise privileged in any channels auditbot is in.)




{'fraction_non_alphanumeric': 0.050203297944431895, 'fraction_numerical': 0.004800090354641969, 'mean_word_length': 3.912343966712899, 'pattern_counts': {'":': 8, '<': 13, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5589731', 'n_tokens_mistral': 4599, 'n_tokens_neox': 4324, 'n_words': 2812}