<commit_before>$Id$


DONE a lot of those emit_set_*_field routines are basically the same and can
be replaced by one that specifies which variable containing source for
setting that field type needs to be substituted.

be able to import from an array, including only specific variables

be able to get fields from a row in array get format

ALREADY DONE - IT's INHERENT be able to import key-value pairs in array 
get/set format 

Add a way for someone doing a CTable definition to specify additional
files to include.

Add a way to write methods in C and add additional C code to the stuff
being generated, from the CTable definition.

DONE make the *_init routine have a static internal copy of an initialization
row and a static saying if it's been generated or not.  generate it
the first time.  from then on, don't generate, but always do a structure
copy of the pregenerated initialized row to the new row being initialized.

DONE when issuing a bad something parsing something, maybe include the bad
value itself.

create a direct interface to postgresql "copy in".  Make it optional.

    http://www.postgresql.org/docs/8.1/static/libpq-exec.html
    http://www.postgresql.org/docs/8.1/static/libpq-copy.html
    http://www.postgresql.org/docs/8.1/static/sql-copy.html

    We'd escape all of the strings that could contain escape characters,
    i.e. strings, but not integers, IP addresses, MAC addresses, etc.

    You'd pass us a connection handle and you'd have already issued the
    "copy in" so the channel would be in copy_in state, and we'll
    call PQputCopyData repeatedly and then PQputCopyEnd and send you
    back the result / error, etc.

    You'd probably tell us how many rows to batch in a single put
    (they can be separated by a newline) or max bytes or something
    like that.

    Also you'd probably also want to do the reverse, copy in.

create a direct interface to reading and writing CSV or tab-separated or 
whatever.

make all the types able to return text, not just to generate tcl object's.
from that, make a function that returns "CONST char *" for all the types.

make a way to find out the types of fields and whether quoting may be
required (i.e. not for ints, floats, macs, inets, needed for strings)

since we will want Tcl_Objs with the names of the fields pretty often,
at startup time generate them and increment their reference counts so
they'll be kept around forever.<commit_msg>Mark prior stuff done.  Add new stuff.<commit_after>$Id$


DONE a lot of those emit_set_*_field routines are basically the same and can
be replaced by one that specifies which variable containing source for
setting that field type needs to be substituted.

be able to import from an array, including only specific variables

be able to get fields from a row in array get format

ALREADY DONE - IT's INHERENT be able to import key-value pairs in array 
get/set format 

Add a way for someone doing a CTable definition to specify additional
files to include.

Add a way to write methods in C and add additional C code to the stuff
being generated, from the CTable definition.

DONE make the *_init routine have a static internal copy of an initialization
row and a static saying if it's been generated or not.  generate it
the first time.  from then on, don't generate, but always do a structure
copy of the pregenerated initialized row to the new row being initialized.

DONE when issuing a bad something parsing something, maybe include the bad
value itself.

create a direct interface to postgresql "copy in".  Make it optional.

    http://www.postgresql.org/docs/8.1/static/libpq-exec.html
    http://www.postgresql.org/docs/8.1/static/libpq-copy.html
    http://www.postgresql.org/docs/8.1/static/sql-copy.html

    We'd escape all of the strings that could contain escape characters,
    i.e. strings, but not integers, IP addresses, MAC addresses, etc.

    You'd pass us a connection handle and you'd have already issued the
    "copy in" so the channel would be in copy_in state, and we'll
    call PQputCopyData repeatedly and then PQputCopyEnd and send you
    back the result / error, etc.

    You'd probably tell us how many rows to batch in a single put
    (they can be separated by a newline) or max bytes or something
    like that.

    Also you'd probably also want to do the reverse, copy in.

create a direct interface to reading and writing CSV or tab-separated or 
whatever.

make all the types able to return text, not just to generate tcl object's.
from that, make a function that returns "CONST char *" for all the types.

make a way to find out the types of fields and whether quoting may be
required (i.e. not for ints, floats, macs, inets, needed for strings)

DONE since we will want Tcl_Objs with the names of the fields pretty often,
at startup time generate them and increment their reference counts so
they'll be kept around forever.

factor out the get routine as the set routine was factored out earlier and
then use it from the big GET case statement.  This will facilitate using
this stuff from C as well as having alternative ways to get stuff, like,
for example, a routine that gets stuff in "array set" format and a routine
that gets stuff for CSV output and stuff like that.


{'fraction_non_alphanumeric': 0.0439581351094196, 'fraction_numerical': 0.002283539486203616, 'mean_word_length': 3.718132854578097, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '472982', 'n_tokens_mistral': 1429, 'n_tokens_neox': 1360, 'n_words': 827}