<commit_before>
@root common

@module chunk

    Chunk encoder/decoder
    
        Chunks encode a volumetric world in chunks of 16x16x256-unit sections.
        
        Zlib compression is used to reduce encoded data size.
        If the chunk contains any 16-bit block IDs a lookup table may be used to map 8-bit RLE
        values to the appropriate 16-bit block IDs to avoid redundant data if the number of unique
        block indices in a chunk is less-than 256 (usually) failing that the RLE will encode 16-bit
        block indices.
        
        Chunks also contain a biome/generator system for generation of new chunks as a chunkoid is explored.
        The chunkoid's containing-quadrant's starsystem type will further affect biome resource generation.
        
        Chunks may be of arbitrary size (well it has an upper limit but is VERY large)
        
        Biome flags:
            Wet
                - High precip
            Arid
                - Low precip (both flags off = normal precip)
            Polar
                - Low temperature (snow/ice at surface)
            Hot
                - High temperature
            Humid
                - high air humidity
            River
                - river generated in biome
            Fertile
                - plants can grow
            Trees<commit_msg>Lose RLE from plan for chunk format<commit_after>
@root common

@module chunk

    Chunk encoder/decoder
    
        Chunks encode a volumetric world in chunks of 16x16x256-unit sections.
        
        Zlib compression is used to reduce encoded data size.
        Lose the RLE stuff as I'm going to use zlib-compressed chunks.
        
        So far plan for a chunk is simply:
        [long32] biome designation (flagbits)
            ([blk:long16],[attr:long16])x256 (one vertical strip top-to-bottom Y 255..0)
                x16 one row (all strips along X -8..+7)
                x16 one chunk (all rows along Z -8..+7)
        
        So length is 4+((2+2)*256*16*16) = 262148 (or just over 256 KB)
        The chunks of a 16-chunk view distance would occupy about .197 GB
        16*16*pi (805) * 262148
        
        If chunks are stored as separate nodes (most logical choice) scale/translate/rotate
        are independent of other chunks so vertices may be reused since each chunk may use
        geometry centered on 0,0,0 then translated as necessary to chunk's acutal world
        position by the node.
        
        TODO: Investigate ways to reduce triangle load of loaded chunks.  Perhaps a viewpoint
        reference (player/viewe's location) point for line-of-sight culling of non-visible
        geometry by the chunk decoder?
        
        Chunks also contain a biome/generator system for generation of new chunks as a chunkoid is explored.
        The chunkoid's containing-quadrant's starsystem type will further affect biome resource generation.
        
        Chunks may be of arbitrary size (well it has an upper limit but is VERY large)
        
        Biome flags:
            Wet
                - High precip
            Arid
                - Low precip (both flags off = normal precip)
            Polar
                - Low temperature (snow/ice at surface)
            Hot
                - High temperature
            Humid
                - high air humidity
            River
                - river generated in biome
            Fertile
                - plants can grow
            Trees
{'fraction_non_alphanumeric': 0.040676024061873386, 'fraction_numerical': 0.024348324262389, 'mean_word_length': 1.6697247706422018, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7447421', 'n_tokens_mistral': 944, 'n_tokens_neox': 831, 'n_words': 407}