<commit_before>file = namelength (1 byte) | name | Properties | 0x00 | Inheritances | 0x00 | Parameters
  where
	Inheritance = parentnamelength (1 byte) | parentname | Bool extends (1 byte)
	Properties = propertynamelength (1 byte) | propertyname | casinglength | casing | flags (1 byte) | Type
	Types = Type*
	Type = ClassType or LambdaType or ParameterizedType or ListType or OptionalType
    ClassType = 0x01 | classnamelength (1 byte) | classname | shadow (1 byte int) | TypeCommon
	LambdaType = 0x02 | ReturnType | Types | 0x00 | TypeCommon
	ParameterizedType = 0x03 | labellength | label | (Type upper OR 0x00) | (Type lower OR 0x00) | shadow | typecommon
	ListType = 0x04 | Type | levels (1 byte)
	OptionalType = 0x05 | Type | levels (1 byte)
	ReturnType = (0x01 | Type) or (0x00)
	TypeCommon = arrayed | aliaslength | alias | specialtylength | specialty | optional

Memory Layout

     File          Property       Classtype          Lambda       Parameterized     ListType      OptionalType     ReturnType      Typecommon      Inheritance
 ------------    ------------    ------------     ------------    ------------    ------------    ------------    ------------    ------------    ------------
| namelength |  | namelength |  |    0x01    |   |    0x02    |  |    0x03    |  |    0x04    |  |    0x05    |  |    0x01    |  |   alias    |  | NameLength |
 ------------    ------------    ------------     ------------    ------------    ------------    ------------    ------------   |    ...     |   ------------
|    name    |  |    name    |  | namelength |   | ReturnType |  |  labellen  |  |    Type    |  |    Type    |  |    Type    |   ------------   |    Name    |
|    ...     |  |    ...     |   ------------    |    ...     |   ------------   |     ...    |  |     ...    |  |     ...    |  | specialtyl |  |    ...     |
|    ...     |  |    ...     |  |    name    |    ------------   |    label   |   ------------    ------------    ------------    ------------    ------------
 ------------    ------------   |    ....    |   |    Types   |  |     ...    |  |   levels   |  |   levels   |        OR        | specialty  |  | Bool extend|
|   0x00     |  |   casingl  |  |    ....    |   |     ...    |  |     ...    |   ------------    ------------    ------------   |    ...     |   ------------
 ------------    ------------    ------------     ------------    ------------   | Typecommon |  | Typecommon |  |    0x00    |   ------------
| Properties |  |   casing   |  | ParamTypes |   |    0x00    |  | Type upper |   ------------    ------------    ------------
|    ...     |   ------------    ------------     ------------    ------------
 ------------   |   flags    |  |    0x00    |   | Typecommon |  | Type lower |
|   0x00     |   ------------    ------------     ------------    ------------
 ------------   |    Type    |  | Typecommon |                   |   shadow   |
|Inheritances|   ------------    ------------                     ------------
|    ...     |                                                   | Typecommon |
 ------------                                                     ------------
|   0x00     |
 ------------
| Parameters |
|    ...     |
 ------------<commit_msg>New tablefilespec for table files containing annotations, refs #69<commit_after>file = version (1 byte) | namelength (1 byte) | name | Properties | 0x00 | Inheritances | 0x00 | Parameters | Annotations | 0x00
  where
	Inheritance = parentnamelength (1 byte) | parentname | Bool extends (1 byte)
	Properties = propertynamelength (1 byte) | propertyname | casinglength | casing | flags (1 byte) | Type | Annotations | 0x00
	Types = Type*
	Type = ClassType or LambdaType or ParameterizedType or ListType or OptionalType
    ClassType = 0x01 | classnamelength (1 byte) | classname | shadow (1 byte int) | TypeCommon
	LambdaType = 0x02 | ReturnType | Types | 0x00 | TypeCommon
	ParameterizedType = 0x03 | labellength | label | (Type upper OR 0x00) | (Type lower OR 0x00) | shadow | typecommon
	ListType = 0x04 | Type | levels (1 byte)
	OptionalType = 0x05 | Type | levels (1 byte)
	ReturnType = (0x01 | Type) or (0x00)
	TypeCommon = arrayed | aliaslength | alias | specialtylength | specialty | optional
	Annotation = NameLength (1 byte) | Name | AnnotationVals | 0x00
	AnnotationVals = (StrAnnotationVal | NumAnnotationVal | BoolAnnotationVal)*
	StrAnnotationVal = 0x01 | len (1 byte) | value
	NumAnnotationVal = 0x02 | value (4 bytes)
	BoolAnnotationVal = 0x03 | value (1 bytes)

Latest version: 4


Memory Layout

     File          Property       Classtype          Lambda      Parameterized     ListType      OptionalType     ReturnType      Typecommon      Inheritance
 ------------    ------------    ------------    ------------    ------------    ------------    ------------    ------------    ------------    ------------
|   version  |  | namelength |  |    0x01    |  |    0x02    |  |    0x03    |  |    0x04    |  |    0x05    |  |    0x01    |  |   alias    |  | NameLength |
 ------------    ------------    ------------    ------------    ------------    ------------    ------------    ------------   |    ...     |   ------------
| namelength |  |    name    |  | namelength |  | ReturnType |  |  labellen  |  |    Type    |  |    Type    |  |    Type    |   ------------   |    Name    |
 ------------   |    ...     |   ------------   |    ....    |   ------------   |     ...    |  |     ...    |  |     ...    |  | specialtyl |  |    ....    |
|    name    |   ------------   |    name    |   ------------   |    label   |   ------------    ------------    ------------    ------------    ------------
|    ...     |  |   casingl  |  |    ....    |  |    Types   |  |     ...    |  |   levels   |  |   levels   |        OR        | specialty  |  | Bool extend|
 ------------    ------------    ------------   |    ....    |   ------------    ------------    ------------    ------------   |    ....    |   ------------
|   0x00     |  |   casing   |  | ParamTypes |   ------------   | Type upper |  | Typecommon |  | Typecommon |  |    0x00    |   ------------
 ------------    ------------   |    ...     |  |    0x00    |  |     ...    |  |    ...     |  |    ...     |   ------------
| Properties |  |   flags    |   ------------    ------------    ------------    ------------    ------------
|    ...     |   ------------   |    0x00    |  | Typecommon |  | Type lower |
 ------------   |    Type    |   ------------   |    ...     |  |     ...    |
|   0x00     |  |    ...     |  | Typecommon |   ------------    ------------
 ------------    ------------   |    ...     |                  |   shadow   |
|Inheritances|  | Annotations|   ------------                    ------------
|    ...     |  |    ...     |                                  | Typecommon |
 ------------    ------------                                   |     ...    |
|   0x00     |  |    0x00    |                                   ------------
 ------------    ------------
| Parameters |
|    ...     |
 ------------
| Annotations|
|    ...     |
 ------------
|   0x00     |
 ------------



  Annotation     StrAnnotnVal    NumAnnotnVal    BoolAnnotVal
 ------------    ------------    ------------    ------------
| NameLength |  |    0x01    |  |    0x02    |  |    0x03    |
 ------------    ------------    ------------    ------------
|    Name    |  |     len    |  |   value    |  |   value    |
 ------------    ------------   |    ...     |   ------------
|    Vals    |  |   value    |   ------------
|    ....    |  |    ....    |
 ------------    ------------
|    0x00    |
 ------------
{'fraction_non_alphanumeric': 0.30767216548834775, 'fraction_numerical': 0.026970411102382824, 'mean_word_length': 0.6863134657836645, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1953958', 'n_tokens_mistral': 2184, 'n_tokens_neox': 2058, 'n_words': 391}