<commit_before># Algorithm for the file-system crawler.

## Input:
* Takes absolute directory to start the crawl.

## Output:
* Does the operation defined by doOperation function on each file found during
  crawl.

* Discrimination based on file-type  can be included for various file types in
  doOperation function.

## Algo:

> doOperation (currentFile):
>       based of file type apply discrimination and do the operation.
>
> allThreadsTerminated():
>       Checks each bit of isDone for the over-all termination status
>
> Crawl (Qindex):
>       while Qindex is outOfBound and (allThreadsTerminated() == True):
>               if Qindex outOfBound:
>                       continue
>               LOCK
>                       Iterate from Qindex till you find and index
>                         with value 0.
>                       Keep updating Qindex.
>               UNLOCK
>               currentCrawlDir <= key(Qindex)
>               Get files/directories in the currentCrawlDir
>               doOperation on files/subDirs
>               LOCK
>                       Enqueue Subdirs in queue of key value pair
>                        - (where key is absolute subdir path and value 0/1)
>                        - (0: not crawled , 1: crawled)
>               UNLOCK
>
> masterCrawl (Q):
>       init all threads maxThreads from index 0
>               Crawl (0)
>       
>
> initCrawl (rootCrawlDir, maxThreads = 10):
>       Initialize a global integer "isDone" to zero.
>        - where each bit corresponds to its status, whether done or not.
>       Initialize a Queue with rootCrawlDirectory with value as 0.
>       masterCrawl (Q)<commit_msg>Fixing indentation, trial 3.<commit_after># Algorithm for the file-system crawler.

## Input:
* Takes absolute directory to start the crawl.

## Output:
* Does the operation defined by doOperation function on each file found during
  crawl.

* Discrimination based on file-type  can be included for various file types in
  doOperation function.

## Algo:

> doOperation (currentFile):
>     * based of file type apply discrimination and do the operation.
>
> allThreadsTerminated():
>     * Checks each bit of isDone for the over-all termination status
>
> Crawl (Qindex):
>     * while Qindex is outOfBound and (allThreadsTerminated() == True):
>             * if Qindex outOfBound:
>                     * continue
>             * LOCK
>                     * Iterate from Qindex till you find and index
>                        with value 0.
>                     * Keep updating Qindex.
>             * UNLOCK
>             * currentCrawlDir <= key(Qindex)
>             * Get files/directories in the currentCrawlDir
>             * doOperation on files/subDirs
>             * LOCK
>                     * Enqueue Subdirs in queue of key value pair
>                      * - (where key is absolute subdir path and value 0/1)
>                      * - (0: not crawled , 1: crawled)
>             * UNLOCK
>
> masterCrawl (Q):
>     * init all threads maxThreads from index 0
>             * Crawl (0)
>       
>
> initCrawl (rootCrawlDir, maxThreads = 10):
>     * Initialize a global integer "isDone" to zero.
>      * - where each bit corresponds to its status, whether done or not.
>     * Initialize a Queue with rootCrawlDirectory with value as 0.
>     * masterCrawl (Q)
{'fraction_non_alphanumeric': 0.07428400954653938, 'fraction_numerical': 0.0062649164677804296, 'mean_word_length': 1.8980121002592913, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6667482', 'n_tokens_mistral': 937, 'n_tokens_neox': 880, 'n_words': 348}