<commit_before>
            
   VCDIFF   
            


STANDARDS ==>                     #RFC 3284

GOAL ==>                          #Produces a "delta file", i.e. a diff between a source and a target
                                  #General focus on easy/fast decoding (but hard encoding), quite efficient and for any format:
                                  #  - ease of decoding
                                  #     - instead of ease of encoding
                                  #     - decoding is straight-forward, but not encoding
                                  #  - decoding speed
                                  #     - instead of encoding speed
                                  #     - instead of encoding efficiency
                                  #     - do not use bit operations (only bytes) so it is faster to decode (but less efficient)
                                  #  - encoding efficiency
                                  #     - instead of resource consumption
                                  #     - need to keep whole window in memory
                                  #  - generic
                                  #     - instead of specifying more about the implementation
                                  #     - allow flexibility (but more work) for encoders
                                  #  - any format (text/binary)
                                  #     - instead of specialized/more efficient for a format
                                  #  - portability


																						 /=+===============================+=\ 
																						/  :                               :  \  
							 														  )==:          ALGORITHMS           :==(   
																						\  :_______________________________:  /    
																						 \=+===============================+=/


ENCODING ALGORITHM ==>            #I/O:
                                  #  - input: source file + target file
                                  #  - output: delta file (start with empty string)
                                  #Instructions:
                                  #  - ADD SIZE, BYTE_ARR: 
                                  #     - append BYTE_ARR (of length SIZE)
                                  #  - RUN SIZE, BYTE: 
                                  #     - append BYTE SIZE times
                                  #  - COPY SIZE, ADDRESS: 
                                  #     - append SIZE bytes from ADDRESS
                                  #     - ADDRESS is byte index inside concatenation of source window + target window
                                  #        - can copy parts of target window as long as it will already have been decoded
                                  #Max SIZE: 255
                                  #Target|source windows: target|source chunks compared against each other

ENCODING IMPLEMENTATION ==>       #Implementation-dependant (not specified):
                                  #  - how to figure out instructions ("string matching")
                                  #  - windows (e.g. their sizes)
                                  #     - the bigger the window, the most efficient, but the more resource-hungry
                                  #     - source window should be a bit larger than target window to increase matching chances
                                  #  - custom code table
                                  #  - secondary compressor

DECODING ALGORITHM ==>            #I/O:
                                  #  - input: source file + delta file
                                  #  - output: target file


																						 /=+===============================+=\ 
																						/  :                               :  \  
							 														  )==:            FORMAT             :==(   
																						\  :_______________________________:  /    
																						 \=+===============================+=/


GOAL ==>                          #Only specifies list of instructions (e.g. RUN SIZE, BYTE) but:
                                  #  - efficiently:
                                  #     - secondary compressor
                                  #     - code table data for often-used instructions
                                  #     - efficient COPY addresses:
                                  #        - SAME_CACHE for often used addresses
                                  #        - NEAR_CACHE for addresses close to each other
                                  #        - "MODE 1" for addresses close to source segment current pointer
                                  #  - use windows

FORMAT ==>                        #Header:
																	#  - 'VCD' (magic number)
																	#  - Version BYTE: 0x00
																	#  - Header indicator BYTE_FLAG
																	#  - [Secondary compressor ID] BYTE
																	#  - [Length]
																	#  - [Code table data]
																	#Each window:
																	#  - Window indicator BYTE
																	#  - [Source segment size] NUM
																	#  - [Source segment position] NUM
																	#  - Delta encoding of target window:
																	#     - length
																	#     - delta encoding:
																	#        - length of target window NUM: length of whole window, after decompression
																	#        - delta indicator BYTE_FLAG
																	#        - length (for each of next three sections)
																	#        - data for ADDs and RUNs BYTE_ARR
																	#        - instructions and sizes BYTE_ARR
																	#        - addresses for COPYs BYTE_ARR
<commit_msg>no commit message<commit_after>
            
   VCDIFF   
            


STANDARDS ==>                     #RFC 3284

GOAL ==>                          #Produces a "delta file", i.e. a diff between a source and a target
                                  #General focus on easy/fast decoding (but hard encoding), quite efficient and for any format:
                                  #  - ease of decoding
                                  #     - instead of ease of encoding
                                  #     - decoding is straight-forward, but not encoding
                                  #  - decoding speed
                                  #     - instead of encoding speed
                                  #     - instead of encoding efficiency
                                  #     - do not use bit operations (only bytes) so it is faster to decode (but less efficient)
                                  #  - encoding efficiency
                                  #     - instead of resource consumption
                                  #     - need to keep whole window in memory
                                  #  - generic
                                  #     - instead of specifying more about the implementation
                                  #     - allow flexibility (but more work) for encoders
                                  #  - any format (text/binary)
                                  #     - instead of specialized/more efficient for a format
                                  #  - portability


																						 /=+===============================+=\ 
																						/  :                               :  \  
							 														  )==:          ALGORITHMS           :==(   
																						\  :_______________________________:  /    
																						 \=+===============================+=/


ENCODING ALGORITHM ==>            #I/O:
                                  #  - input: source file + target file
                                  #  - output: delta file (start with empty string)
                                  #Instructions:
                                  #  - ADD SIZE, BYTE_ARR: 
                                  #     - append BYTE_ARR (of length SIZE)
                                  #  - RUN SIZE, BYTE: 
                                  #     - append BYTE SIZE times
                                  #  - COPY SIZE, ADDRESS: 
                                  #     - append SIZE bytes from ADDRESS
                                  #     - ADDRESS is byte index inside concatenation of source window + target window
                                  #        - can copy parts of target window as long as it will already have been decoded
                                  #Max SIZE: 255
                                  #Target|source windows: target|source chunks compared against each other

ENCODING IMPLEMENTATION ==>       #Implementation-dependant (not specified):
                                  #  - how to figure out instructions ("string matching")
                                  #  - windows (e.g. their sizes)
                                  #     - the bigger the window, the most efficient, but the more resource-hungry
                                  #     - source window should be a bit larger than target window to increase matching chances
                                  #  - custom code table
                                  #  - secondary compressor

DECODING ALGORITHM ==>            #I/O:
                                  #  - input: source file + delta file
                                  #  - output: target file


																						 /=+===============================+=\ 
																						/  :                               :  \  
							 														  )==:            FORMAT             :==(   
																						\  :_______________________________:  /    
																						 \=+===============================+=/


GOAL ==>                          #Only specifies list of instructions (e.g. RUN SIZE, BYTE) but:
                                  #  - efficiently:
                                  #     - secondary compressor
                                  #     - code table data for often-used instructions
                                  #     - efficient COPY addresses:
                                  #        - SAME_CACHE for often used addresses
                                  #        - NEAR_CACHE for addresses close to each other
                                  #        - "MODE 1" for addresses close to source segment current pointer
                                  #  - use windows

FORMAT ==>                        #Header:
																	#  - 'VCD' (magic number)
																	#  - Version BYTE: 0x00
																	#  - Header indicator BYTE_FLAG
																	#  - [Secondary compressor ID] BYTE
																	#  - [Length]
																	#  - [Code table data]
																	#Each window:
																	#  - Window indicator BYTE
																	#  - [Source segment size] NUM
																	#  - [Source segment position] NUM
																	#  - Delta encoding of target window:
																	#     - length
																	#     - delta encoding:
																	#        - length of target window NUM: length of whole window, after decompression
																	#        - delta indicator BYTE_FLAG
																	#        - length (for each of next three sections)
																	#        - data for ADDs and RUNs BYTE_ARR
																	#        - instructions and sizes BYTE_ARR
																	#        - addresses for COPYs BYTE_ARR
                                  #Interleaving (non-standard):
                                  #  - interleaving last three sections, so that can start to decode right away when streaming instead of
                                  #    waiting for first two sections to be downloaded

{'fraction_non_alphanumeric': 0.12649087221095334, 'fraction_numerical': 0.0017849898580121705, 'mean_word_length': 0.8789634146341463, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 150, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2260037', 'n_tokens_mistral': 5175, 'n_tokens_neox': 4312, 'n_words': 778}