<commit_before>Contributing to the Item Browser
================================

The Item browser has two interfaces, the web application and a console 
command to rebuild the item database.

Console command
---------------

```
php src/artisan cataclysm:rebuild
```


The code for this command is located in 
[src/app/commands/CataclysmRebuild.php](src/app/commands/CataclsymRebuild.php).

When you run the console command, the fire() method is executed.

The main players here are the repositories and the indexers.

The repositories can be found in 
[src/app/models/Repositories](src/app/models/Repositories), 
there are two Repositories:

* **LocalRepository** is the main database parser, it finds the json files 
in a given path and loads them into memory.
* **CacheRepository** is the optimized database layer, it uses a 
LocalRepository and then stores the data in the cache.

Then there are the indexers, found in 
[src/app/models/Repositories/Indexers](src/app/models/Repositories/Indexers), 
there is one indexer for each type of object.

Each indexer will have two event listeners:

**onNewObject** is called when the LocalRepository loads a new object, 
the indexer will decide if it's interested and then add the object to 
the database index.

The index is hashtable, to add to it, you call $repo->set() like this:

```
$repo->set("someName", "key", "value");
$repo->set("someName", "id", "data");
$repo->set("items", "toolset", 10);
$repo->set("items", "fire", 12);
```

The database index will then have the following content:

```
array(
        "someName"=>array(
            "key"=>"value",
            "id"=>"data"
        ),
        "items"=>array(
            "toolset"=>10,
            "fire"=>12
        ),
)
```

**onFinishedLoading** is called when all the data files are read, this is 
the last chance to reread everything and check some extra relationships.
<commit_msg>Update the hacking guide following recent commits.<commit_after>Contributing to the Item Browser
================================

The Item browser has two interfaces, the web application and a console 
command to rebuild the item database.

Console command
---------------

```
php src/artisan cataclysm:rebuild
```


The code for this command is located in 
[src/app/commands/CataclysmRebuild.php](src/app/commands/CataclsymRebuild.php).

When you run the console command, the fire() method is executed.

The main players here are the repositories and the indexers.

The repositories can be found in 
[src/app/models/Repositories](src/app/models/Repositories), 
there are two Repositories:

* **LocalRepository** is the main database parser, it finds the json files 
in a given path and loads them into memory.
* **CacheRepository** is the optimized database layer, it uses a 
LocalRepository and then stores the data in the cache.

Then there are the indexers, found in 
[src/app/models/Repositories/Indexers](src/app/models/Repositories/Indexers), 
there is one indexer for each type of object.

Each indexer will have two event listeners:

**onNewObject** is called when the LocalRepository loads a new object, 
the indexer will decide if it's interested and then add the object to 
the database index.

The index is hashtable, to add to it, you call $repo->set() like this:

```
$repo->set("someName.key", "value");
$repo->set("someName.id", "data");
$repo->set("items.toolset", 10);
$repo->set("items.fire", 12);
```

The database index will then have the following content:

```
array(
    "someName.key"=>"value",
    "someName.id"=>"data"
    "item.toolset"=>10,
    "item.fire"=>12
)
```

There's also a $repo->append() method which makes the value of the 
index an array and appends the value to the end, for example:

```
$repo->append("apps", "CDDA");
$repo->append("apps", "Item Browser");
```

This will result in an index like this:

```
array(
        "apps"=>array(
            "CDDA",
            "Item Browser"
        )
)
```

**onFinishedLoading** is called when all the data files are read, this is 
the last chance to reread everything and check some extra relationships.

{'fraction_non_alphanumeric': 0.12863070539419086, 'fraction_numerical': 0.003905296558457408, 'mean_word_length': 3.9552599758162033, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2513737', 'n_tokens_mistral': 1239, 'n_tokens_neox': 1179, 'n_words': 483}