<commit_before># ttv

Timed Text Markup Language (TTML) Validation Tools

The `ttv` tool is used to validate the contents of a [TTML](http://www.w3.org/TR/ttaf1-dfxp/) document represented using XML as a concrete encoding, and where this document is validated against a specific validation *model*.

Validation is performed in four phases in the following order:

 1. Check resource presence, encoding, and decodability (into characters).
 2. Check well-formedness.
 3. Check schema validity.
 4. Check semantic validity.

If errors occur in some phase, then the following phases are skipped.

The first phase, resource checking, attempts to determine (sniff) the character encoding of the resource. To do this, it uses a combination of a BOM (byte order mark) prefix, if present, and an XML encoding declaration, if present. If neither are present (or discernable), then it treats the result as ASCII. Once the encoding has been sniffed, an attempt is made to decode the entire resource using that encoding. If no encoding can be determined, is determined but not supported (by the platform), is determined but not permitted (by TTML), or if the resource cannot be successfully decoded without error using the encoding, then this phase fails.

The encodings which `ttv` can successfully resolve include:

 * `US-ASCII`
 * `UTF-8`
 * `UTF-16LE`
 * `UTF-16BE`
 * `UTF-32LE`
 * `UTF-32BE`

If both BOM is present and XML declaration is present and the two encodings differ, then the BOM encoding is used.

The second phrase performs well-formedness testing by attempting to parse the resource as XML but with validation disabled. If the syntax of the resource does not adhere to the XML specification, then this phase fails.

The third phase performs schema validity testing using an XSD (XML Schema Definition) schema. By default, the published TTML XSD schema is used. However, provisions are being made to support alternative schemas, either built-in to the tool or provided by the user. During this phase, elements and attributes in foreign (non-TTML and non-XML) namespaces are pruned by a filtering process before attempting validation. As a consequence, the potential validity of foreign namespace vocabulary is not assessed. [A future version of this tool may provide a mechanism to validate foreign vocabulary.] If a schema validation error occurs, then this phase fails.

The fourth phase performs additional semantic (and some syntactic) testing on the resource that is beyond what is tested by the previous three phases. For example, some TTML style properties are specified by the TTML schema using the `xs:string` type, which does not test the form of that string to determine compliance with TTML. In this fourth phase, these values are further tested to determine if they comply with TTML syntactic and semantic rules. In addition, a variety of other constraints specified by TTML are tested during this phase.

The following `xs:string` schema typed TTML style properties are tested for syntactic correctness during this fourth phase:

 * `tts:color`
 * `tts:backgroundColor`
 * `tts:extent`
 * `tts:fontFamily`
 * `tts:fontSize`
 * `tts:lineHeight`
 * `tts:opacity`
 * `tts:origin`
 * `tts:padding`
 * `tts:textOutline`
 * `tts:zIndex`

In addition, the following TTML attributes' values are similarly tested:

 * `begin`
 * `dur`
 * `end`
 * `ttp:cellResolution`
 * `ttp:frameRateMultiplier`
 * `ttp:pixelAspectRatio`

A number of additional semantic constraints are tested in the fourth phase, which will (in the meausure of time), be summarizeed here.

A validation *model* includes the following information:<commit_msg>note which properties/attributes are TBD for semantic validity testing<commit_after># ttv

Timed Text Markup Language (TTML) Validation Tools

The `ttv` tool is used to validate the contents of a [TTML](http://www.w3.org/TR/ttaf1-dfxp/) document represented using XML as a concrete encoding, and where this document is validated against a specific validation *model*.

Validation is performed in four phases in the following order:

 1. Check resource presence, encoding, and decodability (into characters).
 2. Check well-formedness.
 3. Check schema validity.
 4. Check semantic validity.

If errors occur in some phase, then the following phases are skipped.

The first phase, resource checking, attempts to determine (sniff) the character encoding of the resource. To do this, it uses a combination of a BOM (byte order mark) prefix, if present, and an XML encoding declaration, if present. If neither are present (or discernable), then it treats the result as ASCII. Once the encoding has been sniffed, an attempt is made to decode the entire resource using that encoding. If no encoding can be determined, is determined but not supported (by the platform), is determined but not permitted (by TTML), or if the resource cannot be successfully decoded without error using the encoding, then this phase fails.

The encodings which `ttv` can successfully resolve include:

 * `US-ASCII`
 * `UTF-8`
 * `UTF-16LE`
 * `UTF-16BE`
 * `UTF-32LE`
 * `UTF-32BE`

If both BOM is present and XML declaration is present and the two encodings differ, then the BOM encoding is used.

The second phrase performs well-formedness testing by attempting to parse the resource as XML but with validation disabled. If the syntax of the resource does not adhere to the XML specification, then this phase fails.

The third phase performs schema validity testing using an XSD (XML Schema Definition) schema. By default, the published TTML XSD schema is used. However, provisions are being made to support alternative schemas, either built-in to the tool or provided by the user. During this phase, elements and attributes in foreign (non-TTML and non-XML) namespaces are pruned by a filtering process before attempting validation. As a consequence, the potential validity of foreign namespace vocabulary is not assessed. [A future version of this tool may provide a mechanism to validate foreign vocabulary.] If a schema validation error occurs, then this phase fails.

The fourth phase performs additional semantic (and some syntactic) testing on the resource that is beyond what is tested by the previous three phases. For example, some TTML style properties are specified by the TTML schema using the `xs:string` type, which does not test the form of that string to determine compliance with TTML. In this fourth phase, these values are further tested to determine if they comply with TTML syntactic and semantic rules. In addition, a variety of other constraints specified by TTML are tested during this phase.

The following `xs:string` schema typed TTML style properties are tested for syntactic correctness during this fourth phase (those not yet implemented are noted as *TBD*):

 * `tts:color`
 * `tts:backgroundColor`
 * `tts:extent` - *TBD*
 * `tts:fontFamily` - *TBD*
 * `tts:fontSize` - *TBD*
 * `tts:lineHeight` - *TBD*
 * `tts:opacity` - *TBD*
 * `tts:origin` - *TBD*
 * `tts:padding` - *TBD*
 * `tts:textOutline` - *TBD*
 * `tts:zIndex` - *TBD*

In addition, the following TTML attributes' values are similarly tested:

 * `begin` - *TBD*
 * `dur` - *TBD*
 * `end` - *TBD*
 * `ttp:cellResolution` - *TBD*
 * `ttp:frameRateMultiplier` - *TBD*
 * `ttp:pixelAspectRatio` - *TBD*

A number of additional semantic constraints are tested in the fourth phase, which will (in the meausure of time), be summarizeed here.

A validation *model* includes the following information:
{'fraction_non_alphanumeric': 0.06423240938166311, 'fraction_numerical': 0.003997867803837953, 'mean_word_length': 4.613313388182498, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '995660', 'n_tokens_mistral': 1988, 'n_tokens_neox': 1898, 'n_words': 1074}