<commit_before>//===== rAthena Documentation ================================
//= rAthena Map Cache Builder and Format Documentation
//===== By: ==================================================
//= DracoRPG
//===== Current Version: =====================================
//= 20070724
//===== Description: =========================================
//= A complete manual for rAthena's map cache generator as 
//= well as a reference on the map cache format used.
//============================================================

Preface:
-------------------------------------------------------------------------------

Since SVN revision ~10000, the map-server no longer knows how to read GRFs directly. It reads maps from a
"map cache" file that contains all and only the useful data about the maps. A map cache containing every official
kRO Sakray map currently supported by rAthena is provided as a default.
If you have custom maps or want to minimize the size of your map cache because your server does not load all of them
(multi-map-server or light test server), you can use the map cache builder to generate a new one fitting your needs.

Map cache builder manual:
-------------------------------------------------------------------------------

The source code for the map cache builder is located in src/tool/. It can be built using "make tools" if you use the Makefile
or using the "mapcache" project under Visual Studio. Named "mapcache", the executable will be in your rAthena main folder.
The map cache builder needs 3 file paths : one is a list of GRFs and/or data directory containing the maps, the second
is the list of maps to add to the map cache, and the last one is the path of the map cache to generate. Default values for
those paths are "tools/mapcache/grf_files.txt", "db/map_index.txt" and "db/map_cache.dat".

The list of GRFs and/or data directory must follow the format and indication of the default file: as many "grf:" entries as
you wish and optionally only one "data_dir:" entry with trailing backslash included. // comments are supported as usual.
In fact, any file with one map name per line can be used as a map list, that's why the map index list is used as a default:
we are sure it contains every map supported by the server. Anything after the map name is ignored, // comments are supported
and if the first word on the line is "map:" then the second word is used as the map name instead: that allows using
maps_athena.conf as your map list, which is handy if you want to generate a minimal map cache for each of your multiple
map-servers.
The map cache file path can point to an already existing file, as the builder adds a map only if it's not already cached.
This way, you can add custom maps to the base map cache without even needing kRO Sakray maps. If you wish to rebuild the
entire map cache, though, you can either provide a path to a non-existing file, or force the rebuild mode.

Here are the command-line arguments you can provide to the map cache builder to customize its behavior:
 -grf path/to/grf/list
   Allows to specify the file containing the list of GRFs and/or data directory
 -list path/to/map/list
   Allows to specify the file containing the list of maps to add to the map cache
 -cache path/to/map/cache
   Allows to specify the path to the generated map cache
 -rebuild
   Allows to force the rebuild mode (map cache will be overwritten even if it already exists)


Map cache format reference:
-------------------------------------------------------------------------------
<commit_msg>* Update to doc/map_cache.txt due to r16867<commit_after>//===== rAthena Documentation ================================
//= rAthena Map Cache Builder and Format Documentation
//===== By: ==================================================
//= DracoRPG
//===== Current Version: =====================================
//= 20070724
//===== Description: =========================================
//= A complete manual for rAthena's map cache generator as 
//= well as a reference on the map cache format used.
//============================================================

Preface:
-------------------------------------------------------------------------------

Since SVN revision ~10000, the map-server no longer knows how to read GRFs directly. It reads maps from a
"map cache" file that contains all and only the useful data about the maps. A map cache containing every official
kRO Sakray map currently supported by rAthena is provided as a default.
If you have custom maps or want to minimize the size of your map cache because your server does not load all of them
(multi-map-server or light test server), you can use the map cache builder to generate a new one fitting your needs.

Map cache builder manual:
-------------------------------------------------------------------------------

The source code for the map cache builder is located in src/tool/. It can be built using "make tools" if you use the Makefile
or using the "mapcache" project under Visual Studio. Named "mapcache", the executable will be in your rAthena main folder.
The map cache builder needs 3 file paths : one is a list of GRFs and/or data directory containing the maps, the second
is the list of maps to add to the map cache, and the last one is the path of the map cache to generate. Default values for
those paths are "tools/mapcache/grf_files.txt", "db/map_index.txt" and "db/(pre-)re/map_cache.dat".

As of r16867, the mapcache within SVN can be located in db/pre-re/ and db/re/. This is due to renewal and pre-renewal modes
having slightly different maps. When building your cache, you should ensure you're pointing the tool to the correct location.

The list of GRFs and/or data directory must follow the format and indication of the default file: as many "grf:" entries as
you wish and optionally only one "data_dir:" entry with trailing backslash included. // comments are supported as usual.
In fact, any file with one map name per line can be used as a map list, that's why the map index list is used as a default:
we are sure it contains every map supported by the server. Anything after the map name is ignored, // comments are supported
and if the first word on the line is "map:" then the second word is used as the map name instead: that allows using
maps_athena.conf as your map list, which is handy if you want to generate a minimal map cache for each of your multiple
map-servers.
The map cache file path can point to an already existing file, as the builder adds a map only if it's not already cached.
This way, you can add custom maps to the base map cache without even needing kRO Sakray maps. If you wish to rebuild the
entire map cache, though, you can either provide a path to a non-existing file, or force the rebuild mode.

Here are the command-line arguments you can provide to the map cache builder to customize its behavior:
 -grf path/to/grf/list
   Allows to specify the file containing the list of GRFs and/or data directory
 -list path/to/map/list
   Allows to specify the file containing the list of maps to add to the map cache
 -cache path/to/map/cache
   Allows to specify the path to the generated map cache
 -rebuild
   Allows to force the rebuild mode (map cache will be overwritten even if it already exists)


Map cache format reference:
-------------------------------------------------------------------------------

{'fraction_non_alphanumeric': 0.1690579317812669, 'fraction_numerical': 0.005143475906876015, 'mean_word_length': 3.086836283185841, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7174381', 'n_tokens_mistral': 1781, 'n_tokens_neox': 1696, 'n_words': 1085}