<commit_before>Notes for Json ingestion talk.

Topics to discuss

* How sites install XALT on login and compute nodes?
** Distributed install of XALT on all nodes/?
   This is the way that TACC does it
** Single Global location accessible for all nodes?
*

* How to sites install XALT for ingestion?
** TACC uses a VM to store using syslog  transport style
** A single login node used to copy file based transport   
** This talk will assume TACC style layout but it might
   be different at your site.
* At some sites it is more work to re-install XALT
  on eompute and login nodes
** TACC requires a maintenance to install on compute nodes
   This may change in future where they might split the compute nodes
   into block where they can pause a block at a time and install
* Making changes to the ingestion install is easy (at least for TACC)
** Is it different at your site?  
* executable filtering during ingestion
** Some sites would like to be able to control ingestion and not modify their install of XALT 
** There is a block in your Site Config.py file
*** This block of python code is converted to a flex routine to provide fast regex parsing.
pre_ingest_patterns = [
#   percent   path pattern
    [0.0,     r'.*\/foobar'],
    [0.01,    r'.*\/BAZ'],
]
*** This pattern in your sites config is combined with the xalt system config file:
    As shown via the xalt_configuration_report program:
*----------------------*
 Array: ingestPatternA
*----------------------*
================== /path/to/site/config.py ==================
   0: 0.0000 => .*\/foobar
   1: 0.0100 => .*\/BAZ
================== src/tmpl/xalt_config.py ==================
   2: 1.0000 => .*

*** This parsing is converted to a C then shared library (libpreIngest.so).
*** This C library is connected to the python ingestion programs via the ctypes
from   ctypes import *   # used to interact with C shared libraries

libpath      = os.path.join(dirNm, "../lib64/libpreIngest.so")
libpreIngest = CDLL(libpath)
pre_ingest_filter = libpreIngest.pre_ingest_filter
pre_ingest_filter.argtypes = [c_char_p]
pre_ingest_filter.restype  = c_double
...
exec_prob = pre_ingest_filter(exec_path.encode())


   
* New pkg filter added to ingestion
** It is the same pkg filter added to xalt_record_pkg
** This just means that you can remove packages during ingestion phase
*** This means you can prevent packages from being added to the DB
    in two places: Either at creation phase (but have to reinstall
    XALT everywhere)
** This means if you find a package you want to ignore
   You can do at the ingestion install of XALT and not wait for a
   maintenance
   
* Debugging json ingestion
** A user on the XALT mailing list was asking for help
** It wasn't clear why their *.json files where not being ingested.
* The --debug (-D) option to xalt_file_to_db.py and xalt_syslog_to_db.py was born
<commit_msg>Nore notes for 5/19 presentation<commit_after>Notes for Json ingestion talk.

Topics to discuss

* How sites install XALT on login and compute nodes?
** Distributed install of XALT on all nodes/?
   This is the way that TACC does it
** Single Global location accessible for all nodes?
*

* How to sites install XALT for ingestion?
** TACC uses a VM to store using syslog  transport style
** A single login node used to copy file based transport   
** This talk will assume TACC style layout but it might
   be different at your site.
* At some sites it is more work to re-install XALT
  on eompute and login nodes
** TACC requires a maintenance to install on compute nodes
   This may change in future where they might split the compute nodes
   into block where they can pause a block at a time and install
* Making changes to the ingestion install is easy (at least for TACC)
** Is it different at your site?  
* executable filtering during ingestion
** Some sites would like to be able to control ingestion and not modify their install of XALT 
** There is a block in your Site Config.py file
*** This block of python code is converted to a flex routine to provide fast regex parsing.
pre_ingest_patterns = [
#   percent   path pattern
    [0.0,     r'.*\/foobar'],
    [0.01,    r'.*\/BAZ'],
]
*** This pattern in your sites config is combined with the xalt system config file:
    As shown via the xalt_configuration_report program:
*----------------------*
 Array: ingestPatternA
*----------------------*
================== /path/to/site/config.py ==================
   0: 0.0000 => .*\/foobar
   1: 0.0100 => .*\/BAZ
================== src/tmpl/xalt_config.py ==================
   2: 1.0000 => .*

*** This parsing is converted to a C then shared library (libpreIngest.so).
*** This C library is connected to the python ingestion programs via the ctypes
from   ctypes import *   # used to interact with C shared libraries

libpath      = os.path.join(dirNm, "../lib64/libpreIngest.so")
libpreIngest = CDLL(libpath)
pre_ingest_filter = libpreIngest.pre_ingest_filter
pre_ingest_filter.argtypes = [c_char_p]
pre_ingest_filter.restype  = c_double
...
exec_prob = pre_ingest_filter(exec_path.encode())


   
* New pkg filter added to ingestion
** It is the same pkg filter added to xalt_record_pkg
** This just means that you can remove packages during ingestion phase
*** This means you can prevent packages from being added to the DB
    in two places: Either at creation phase (but have to reinstall
    XALT everywhere)
** This means if you find a package you want to ignore
   You can do at the ingestion install of XALT and not wait for a
   maintenance
   
* Debugging json ingestion
** A user on the XALT mailing list was asking for help
** It wasn't clear why their *.json files where not being ingested.
* The --debug (-D) option to xalt_file_to_db.py and xalt_syslog_to_db.py was born
* Issue #46 shows a detailed discussion of a particular site having issues.
* This issue pointed out another way that XALT had to protect itself from site installations
** I have to guarantee that no XALT program would be traced. 
   Otherwise an endless loop (:-<) AKA bad.
** A site had in their site Config.py: 
path_patterns = [
  ['KEEP',  r'\/opt\/envHPC\/.*'],
  [ 'SKIP',  r'.*'],
  ]

where XALT was stored in /opt/envHPC/xalt/...
** XALT uses two configuration files
*** One from the system src/tmpl/xalt_config.py
*** Your site Config.py file
*** The two config files control how XALT filters
** src/tmpl/xalt_config.py has head_path_patterns (and your config file does not)
   It forces XALT to skip (ignore) all XALT executables 
   It also ignores "logger" a standard unix command to write to syslog.
   XALT now uses library calls instead of system() calls to write to syslog.
   (It only uses logger during testing)

* This site was writing *.json files to ~user/.xalt.d
  XALT now warns users of xalt_file_to_db.py that using that directory
  has a race condition and should not be used.
  It is much faster to write to a shared global location.
* This site had problems walking the directory tree as it was a parallel file system
  To better track what XALT is doing both xalt_file_to_db.py and
  xalt_syslog_to_db.py support a -D (debug) option to print out what
  XALT is doing to ingest your Json records.
* File ingestion:  It either looks for *.json files in a global location or in ~user/.xalt.d
  Searching each user can be slow especially if you have lots of
  users.
** XALT searches first for link.*.json files and reports the number then each one:
link: /home/user/.xalt.d/link.rios.2022_05_04_13_27_30_6203.user.f84ccbcc-fa7c-47cf-957f-d086abc93550.json
  --> Trying to open file
  --> Trying to load json
  --> Sending record to xalt.link_to_db()
  --> Trying to connect to database
  --> Starting TRANSACTION
  --> Searching for build_uuid in db
  --> Trying to insert link record into db
  --> Success: link recorded
  --> Trying to insert objects into db
  --> Trying to insert functions into db
  --> Done
  
** XALT searches for run.*.json files next:  
run: /home/user/.xalt.d/run.rios.2022_05_04_13_27_30_6957.mclay.zzz.7c9269f9-2bcc-4cb9-a8f4-526f2b36ba62.json
  --> Trying to open file
  --> Trying to load json
  --> Sending record to xalt.run_to_db()
  --> Trying to connect to database
  --> Starting TRANSACTION
  --> Searching for run_uuid in db
  --> Trying to insert run record into db
  --> Success: stored full xalt_run record
  --> Trying to insert objects into db
  --> Trying to insert env vars into db
  --> Done
** Finally XALT searches for pkg.*.json: 
  --> Found 16 pkg.*.json files

  --> Success: pkg entry "R:bar" stored
  --> Success: pkg entry "R:foo" stored
  --> Success: pkg entry "R:acme" stored
  --> Failed to record: pkgFilter blocks "R:base" 
  --> Success: pkg entry "python:json" stored
  --> Success: pkg entry "python:linecache" stored
  --> Success: pkg entry "python:struct" stored
  --> Success: pkg entry "python:base64" stored
  --> Success: pkg entry "python:tokenize" stored
  --> Success: pkg entry "python:traceback" stored
  --> Success: pkg entry "python:token" stored
  --> Success: pkg entry "python:json" stored
  --> Success: pkg entry "python:base64" stored
  --> Success: pkg entry "python:codecs" stored
  --> Success: pkg entry "python:string" stored
  --> Success: pkg entry "python:encodings" stored




* Similarly for ingesting json records through Syslog. 
  But there there are no file names given and the link, run and pkg
  records are mixed together
  --> Trying to connect to database
  --> Starting TRANSACTION
  --> Searching for build_uuid in db
  --> Trying to insert link record into db
  --> Success: link recorded
  --> Trying to insert objects into db
  --> Trying to insert functions into db
  --> Done

  --> Trying to connect to database
  --> Starting TRANSACTION
  --> Searching for run_uuid in db
  --> Trying to insert run record into db
  --> Success: stored full xalt_run record
  --> Trying to insert objects into db
  --> Trying to insert env vars into db
  --> Done

  --> Success: pkg entry "python:token" stored
  --> Success: pkg entry "python:tokenize" stored
  --> Success: pkg entry "python:linecache" stored

* For tracking down problems with ingestion this is useful
  But is not very practical for thousands of program links and
  executions

  Still it is very useful.
  

{'fraction_non_alphanumeric': 0.10023332685203189, 'fraction_numerical': 0.01283297686175384, 'mean_word_length': 3.388651877133106, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5391717', 'n_tokens_mistral': 3146, 'n_tokens_neox': 2889, 'n_words': 1381}