<commit_before>
*Note: This repo has been just created. It will get cleaned up
over the next few days. Please dont use it yet. 

Plugins for dgit
===============

This repo has a series of dgit addons that are expected to be more
computationally expensive/otherwise complex such as integrity checks,
anonymization, and may be eventually bitcoin integration.

::

   $ pip install dgit-extensions 

   $ dgit plugins 
   



Validator 
=========

This will evaluate the content against the validation rules specified
<commit_msg>Extend notes<commit_after>

*Note: This repo has been just created. It will get cleaned up
over the next few days. Please dont use it yet.* 

`dgit <https://github.com/pingali/dgit>`_ is a dataset management
tool. This repo has a series of dgit addons that are expected to be
more computationally expensive/otherwise complex such as integrity
checks, anonymization, and may be eventually bitcoin integration.

::

   $ pip install dgit-extensions 

   # The modules in bold are extensions from this repo 
   $ dgit plugins 
   ========
   metadata
   ========
   generic-metadata (v0) : generic-metadata Basic metadata tracker
      Supp: ['generic-metadata']
   
   ========
   validator
   ========
   basic-validator (v0) : basic-validator Basic validator of the content
      Supp: ['basic-validator']
   **simple-validator (v0) : simple-validator Simple validator extension
      Supp: ['simple-validator']**
   
   ========
   instrumentation
   ========
   platform (v0) : platform Execution platform information
      Supp: ['platform']
   content (v0) : content Basic content analysis
      Supp: ['content']
   executable (v0) : executable Executable analysis
      Supp: ['executable']
   
   ========
   backend
   ========
   s3 (v0) : s3 S3 backend
      Supp: ['s3']
   local (v0) : local Local filesystem backend
      Supp: ['local']
   
   ========
   repomanager
   ========
   git (v0) : git Git-based repomanager
      Supp: ['git']
      
   


Validator 
=========

This will evaluate the content against the validation rules specified

{'fraction_non_alphanumeric': 0.11787254432199329, 'fraction_numerical': 0.006229036895064686, 'mean_word_length': 3.2962962962962963, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6017012', 'n_tokens_mistral': 595, 'n_tokens_neox': 553, 'n_words': 230}