<commit_before>TODO for podload

* create an init.d file for running as daemon
* expose the seed feature via some commandline switch
* some kind of gui for editing the configs
** clojurescript? web based front-end?
* some way to dynamically add/remove configs from the queue
** i.e. based on
   #+BEGIN_SRC clojure
    (seq (.getQueue executor))
   #+END_SRC
* better command-line handling
** i.e.
   #+BEGIN_SRC clojure
    (let [[opts args banner] (cli/cli args
                                    ["-h"
                                     "--help" "pass config filename"
                                     :default false
                                     :flag true])]
    (when (:help opts)
      (println banner)))
   #+END_SRC
* use env properly
* add log file path as an optional (with default) item in the config file and process it
* integrate with smart blocks somehow more closely?<commit_msg>Another TODO item.<commit_after>TODO for podload

* create an init.d file for running as daemon
* add a "brutal mode" flag for brutally scraping things that aren't actually rss feeds.
** re-use my old py regexps, i.e.
   #+BEGIN_SRC python
     ['.*"(http:.+mp3)".*',
        '.*>(http:.+mp3)<.*',   
        '.*>(http:.+ogg)<.*',
        '.*"(http:.+ogg)".*',
        '.*>(http:.+flac)<.*',
        '.*"(http:.+flac)".*',
        '.*"(.+flac)".*',
        '.*"(.+m3u)".*']
   #+END_SRC
* expose the seed feature via some commandline switch
* some kind of gui for editing the configs
** clojurescript? web based front-end?
* some way to dynamically add/remove configs from the queue
** i.e. based on
   #+BEGIN_SRC clojure
    (seq (.getQueue executor))
   #+END_SRC
* better command-line handling
** i.e.
   #+BEGIN_SRC clojure
    (let [[opts args banner] (cli/cli args
                                    ["-h"
                                     "--help" "pass config filename"
                                     :default false
                                     :flag true])]
    (when (:help opts)
      (println banner)))
   #+END_SRC
* use env properly
* add log file path as an optional (with default) item in the config file and process it
* integrate with smart blocks somehow more closely?
{'fraction_non_alphanumeric': 0.12229241877256318, 'fraction_numerical': 0.0013537906137184115, 'mean_word_length': 1.9838492597577388, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2298803', 'n_tokens_mistral': 681, 'n_tokens_neox': 639, 'n_words': 238}