<commit_before>* issues with flstats

** todo list

*** TODO 4.	Document: [simul_setup], [flow_details],
    [class_details], [fl_read_one_bin].  Give examples of use; warn
    about memory consumption.
*** TODO 8. Set atoft[] from Tcl code.  (Need to change "alltags"
	in [fl_setft]; or delete!)
*** TODO 9. Protohasports...
*** TODO 12. Change atoft[] to allow spec of (SHIFT_RIGHT|IN_PLACE)
		(to decouple output base from shifting) and DOTTED as
		a flag; make fmt == char * ("%d", say)?
*** TODO 16. Change "low level" comments to general lattice.
*** TODO 17. make low level flows use HIGH class numbers.
	(flstats(maxclass), flstats(maxflow), ...) set by .c.
*** TODO 18. "df len" should be a flow type "df" 
    it should have as its parent, a flow type "len".
*** DONE 19. Allow a "name" to be given to a class [fl_set_class_name].
    (maybe default to underlying flows' type, if only one type in the
    class?)
*** DONE 20. catch a signal (SIGINFO, C-t) as a way of terminating a bin, printing out a line.
*** DONE 21. option for timestamp (rather than binno) time of printing lines?  
    also, number of seconds spanned by this report line
*** TODO 22. hosta/hostb type (sort numerically/lexicographically)
*** DONE 23. flag to disable printing "name val" pairs; 
*** DONE 24. flag to have separate "reporting interval" line; indent class/flow lines after
*** DONE 25. require *entire* long_option names
*** DONE 26. allow specification of column separator for reports
    "|", ";", "<TAB>", etc.
*** DONE 27. flag to print a header line
*** DONE 28. get rid of --kind, fix28/44, etc.
*** DONE 29. make parameterless options *short*
*** TODO 30. if app init code (loading flstats.tcl, that is) fails
    we end up at a tclsh prompt -- program should rather exit
    ([catch]?)
*** TODO 31. [fl_time_base] return current state

*** DONE 32. make fe_id a char*
*** TODO 33. set_time() does a while loop of do_timers()
    probably only need this for simulation?
*** DONE 34. in flstats.c, move binno into struct ri.
*** TODO 35. make ce_created, etc., struct timeval
    probably need for general scheme for time format (so can make
    these integers, if we want; etc.)
*** TODO 36. get rid of [fl_set_tcpd_file]?
    maybe leave it?  in case some future file type (so will remember
    how to deal with it).
*** DONE 37. sipg for ri not working?
    they are current binno
*** DONE 38. count runt, and other bad packets, in ri structure
    currently they are counted in class 0.
*** DONE 39. new output format scheme
    table of: [printed name; fe->fe_foo; {int,time,sipg}].  can do in
    .c, but can also easily (if a bit slower) do in .tcl.
*** TODO 40. IPv6
*** DONE 41. runt packets, etc., should maybe be counted earlier
    in, e.g., receive_tcpd...
*** DONE 42. "errpkts" includes perfectly good, but ignored packets
     ARP, say
*** DONE 43. there are *two* labels "type" on output line
*** DONE 44. allow --dontoutput {cl|fl|ri} taglist
    probably more useful than --output

then, run it twice, once to get "format" array, once to get code to to
fill array (array has union to hold the three possible types).

then, after array filled, run a second array which is [index (into
above array); label; {int, time, sipg}] to create output string.

to set up this second array, fl_{ri,flow,class}_format "printed_name
label {int,time,sipg} ...", then map "printed_name" into index values.

advantage: o(1) on output (setup takes the expense)

then, can produce some derived statistics (such as "errpkts", to hold
runt, et al.)

*** DONE 45. allow --omodify
    one would just list the fields whose output representation one
    wanted to change, like "ri first_recv::int".  --omodify ??
*** TODO 46. a flag to print out the list of available flow specifiers
    i.e., the "atoft" table

*** TODO 47. with --binsecs 1, seconds aren't consistently
    counted in one second intervals.
*** DONE 48. could have -c *and* -f
    just prepend class stats to flow stats (or, indent flows twice as
    much)
*** TODO 49. -indentation-string
*** TODO 50. -USR1 reports flows, -USR2 classes?
    or, maybe, USR1 does reporting; USR2 iterates among the
    alternatives. (could have USR2 report current setting to stderr?)
*** TODO 51. readme file for github
    github as a [[https://github.com/github/markup][repository]] listing the approved formats.  man(1)
    isn't 1.
*** TODO 52. some switches for common output requests
    such as, -timebase tr, print lastrevd times, et al., as seconds, etc.
** reporting times, bin numbers, etc.

*** reporting intervals

for every reporting interval, would *like* to report when that
reporting interval started and when it stopped.

when the first packet in the interval arrived, when the last, sipg
within the interval, sipg since time 0.

we can report start time as being the time of the last packet received
in the *previous* reporting interval.

but, for stopping: if binsecs > 0, easy.

if binsecs == 0, and we are reading a live trace (which, actually, we
don't know, btw), then either the trace has ended (so, last packet
time), or we got a signal.

binsecs == 0, got a signal: we end up needing to wait for any
outstanding read to finish.  then, do we take the *next* packet to
arrive at that time?  or, the last packet we read in?  or..?

the problem is that the next and N following packets may have already
been received.  or, that next packet may take 30 seconds to arrive
after.

but, i guess in any case, the time of that packet is, in fact, the end
of the *reporting* interval.

for reporting interval:

| binno    | sequential id of reporting intervals               | just a counter                                                         |
| stime    | time this interval started                         | etime of *previous* interval                                           |
| etime    | time this interval ended                           | timestamp of last packet (if binsecs, round to integer)                |
| duration | how long this interval lasted                      | can be computed from etime-stime                                       |
| fptime   | time first packet arrived this interval            | can be derived by MINing class/flow reports (if everything classified) |
| lptime   | time last packet arrived this interval             | can be derived by MAXing class/flow reports (ditto)                    |
| npkts    | number of packets seen this interval               | can be derived by summing class/flow reports                           |
| nbytes   | number of bytes seen this interval                 | can be derived by summing class/flow reports                           |
| isipg    | smoothed inter packet gap this interval            |                                                                        |
| tsipg    | smoothed inter packet gap since beginning of trace |                                                                        |

for each class/flow

| kind    | kind of class/flow                         | for flow, "net/src"                                |
| id      | id of this particular class/flow           | for class, i guess "net/src" (look at child flows) |
| ifptime | time first packet arrived in this interval |                                                    |
| tfptime | time first packet arrived since b.o.t.     | can be derived from history of reports        |
| lptime  | time last packet arrived in this interval  |                                                    |
| npkts   | number of packets seen this interval       |                                                    |
| nbytes  | number of bytes seen this interval         |                                                    |
| isipg   | smoothed inter packet gap this interval    |                                                    |
| tsipg   | smoothed inter packet gap since b.o.t.     |                                                    |


<commit_msg>*** empty log message ***<commit_after>* issues with flstats

** todo list

*** TODO 4.	Document: [simul_setup], [flow_details],
    [class_details], [fl_read_one_bin].  Give examples of use; warn
    about memory consumption.
*** TODO 8. Set atoft[] from Tcl code.  (Need to change "alltags"
	in [fl_setft]; or delete!)
*** TODO 9. Protohasports...
*** TODO 12. Change atoft[] to allow spec of (SHIFT_RIGHT|IN_PLACE)
		(to decouple output base from shifting) and DOTTED as
		a flag; make fmt == char * ("%d", say)?
*** TODO 16. Change "low level" comments to general lattice.
*** TODO 17. make low level flows use HIGH class numbers.
	(flstats(maxclass), flstats(maxflow), ...) set by .c.
*** TODO 18. "df len" should be a flow type "df" 
    it should have as its parent, a flow type "len".
*** DONE 19. Allow a "name" to be given to a class [fl_set_class_name].
    (maybe default to underlying flows' type, if only one type in the
    class?)
*** DONE 20. catch a signal (SIGINFO, C-t) as a way of terminating a bin, printing out a line.
*** DONE 21. option for timestamp (rather than binno) time of printing lines?  
    also, number of seconds spanned by this report line
*** TODO 22. hosta/hostb type (sort numerically/lexicographically)
*** DONE 23. flag to disable printing "name val" pairs; 
*** DONE 24. flag to have separate "reporting interval" line; indent class/flow lines after
*** DONE 25. require *entire* long_option names
*** DONE 26. allow specification of column separator for reports
    "|", ";", "<TAB>", etc.
*** DONE 27. flag to print a header line
*** DONE 28. get rid of --kind, fix28/44, etc.
*** DONE 29. make parameterless options *short*
*** TODO 30. if app init code (loading flstats.tcl, that is) fails
    we end up at a tclsh prompt -- program should rather exit
    ([catch]?)
*** TODO 31. [fl_time_base] return current state

*** DONE 32. make fe_id a char*
*** TODO 33. set_time() does a while loop of do_timers()
    probably only need this for simulation?
*** DONE 34. in flstats.c, move binno into struct ri.
*** TODO 35. make ce_created, etc., struct timeval
    probably need for general scheme for time format (so can make
    these integers, if we want; etc.)
*** TODO 36. get rid of [fl_set_tcpd_file]?
    maybe leave it?  in case some future file type (so will remember
    how to deal with it).
*** DONE 37. sipg for ri not working?
    they are current binno
*** DONE 38. count runt, and other bad packets, in ri structure
    currently they are counted in class 0.
*** DONE 39. new output format scheme
    table of: [printed name; fe->fe_foo; {int,time,sipg}].  can do in
    .c, but can also easily (if a bit slower) do in .tcl.
*** TODO 40. IPv6
*** DONE 41. runt packets, etc., should maybe be counted earlier
    in, e.g., receive_tcpd...
*** DONE 42. "errpkts" includes perfectly good, but ignored packets
     ARP, say
*** DONE 43. there are *two* labels "type" on output line
*** DONE 44. allow --dontoutput {cl|fl|ri} taglist
    probably more useful than --output

then, run it twice, once to get "format" array, once to get code to to
fill array (array has union to hold the three possible types).

then, after array filled, run a second array which is [index (into
above array); label; {int, time, sipg}] to create output string.

to set up this second array, fl_{ri,flow,class}_format "printed_name
label {int,time,sipg} ...", then map "printed_name" into index values.

advantage: o(1) on output (setup takes the expense)

then, can produce some derived statistics (such as "errpkts", to hold
runt, et al.)

*** DONE 45. allow --omodify
    one would just list the fields whose output representation one
    wanted to change, like "ri first_recv::int".  --omodify ??
*** TODO 46. a flag to print out the list of available flow specifiers
    i.e., the "atoft" table

*** TODO 47. with --binsecs 1, seconds aren't consistently
    counted in one second intervals.
*** DONE 48. could have -c *and* -f
    just prepend class stats to flow stats (or, indent flows twice as
    much)
*** TODO 49. -indentation-string
*** TODO 50. -USR1 reports flows, -USR2 classes?
    or, maybe, USR1 does reporting; USR2 iterates among the
    alternatives. (could have USR2 report current setting to stderr?)
*** TODO 51. readme file for github
    github as a [[https://github.com/github/markup][repository]] listing the approved formats.  man(1)
    isn't 1.
*** TODO 52. some switches for common output requests
    such as, -timebase tr, print lastrevd times, et al., as seconds, etc.

** reporting times, bin numbers, etc.

*** reporting intervals

for every reporting interval, would *like* to report when that
reporting interval started and when it stopped.

when the first packet in the interval arrived, when the last, sipg
within the interval, sipg since time 0.

we can report start time as being the time of the last packet received
in the *previous* reporting interval.

but, for stopping: if binsecs > 0, easy.

if binsecs == 0, and we are reading a live trace (which, actually, we
don't know, btw), then either the trace has ended (so, last packet
time), or we got a signal.

binsecs == 0, got a signal: we end up needing to wait for any
outstanding read to finish.  then, do we take the *next* packet to
arrive at that time?  or, the last packet we read in?  or..?

the problem is that the next and N following packets may have already
been received.  or, that next packet may take 30 seconds to arrive
after.

but, i guess in any case, the time of that packet is, in fact, the end
of the *reporting* interval.

for reporting interval:

| binno    | sequential id of reporting intervals               | just a counter                                                         |
| stime    | time this interval started                         | etime of *previous* interval                                           |
| etime    | time this interval ended                           | timestamp of last packet (if binsecs, round to integer)                |
| duration | how long this interval lasted                      | can be computed from etime-stime                                       |
| fptime   | time first packet arrived this interval            | can be derived by MINing class/flow reports (if everything classified) |
| lptime   | time last packet arrived this interval             | can be derived by MAXing class/flow reports (ditto)                    |
| npkts    | number of packets seen this interval               | can be derived by summing class/flow reports                           |
| nbytes   | number of bytes seen this interval                 | can be derived by summing class/flow reports                           |
| isipg    | smoothed inter packet gap this interval            |                                                                        |
| tsipg    | smoothed inter packet gap since beginning of trace |                                                                        |

for each class/flow

| kind    | kind of class/flow                         | for flow, "net/src"                                |
| id      | id of this particular class/flow           | for class, i guess "net/src" (look at child flows) |
| ifptime | time first packet arrived in this interval |                                                    |
| tfptime | time first packet arrived since b.o.t.     | can be derived from history of reports        |
| lptime  | time last packet arrived in this interval  |                                                    |
| npkts   | number of packets seen this interval       |                                                    |
| nbytes  | number of bytes seen this interval         |                                                    |
| isipg   | smoothed inter packet gap this interval    |                                                    |
| tsipg   | smoothed inter packet gap since b.o.t.     |                                                    |



{'fraction_non_alphanumeric': 0.08468321971355307, 'fraction_numerical': 0.012508599662267809, 'mean_word_length': 2.2289983844911148, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 45, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2163920', 'n_tokens_mistral': 4446, 'n_tokens_neox': 4160, 'n_words': 2122}