<jupyter_start><jupyter_text>Export all lemmatised tokens in CSV* __Note__: This script is similar to [this one](../lemmatize-new-witness/NV_2_extraction_sans_lemmes.ipynb), which is rather meant for a file whose `` still have no `att.lemma|att.pos` attributes, or when one means to overwrite them.* __Note__: due to the fact that on GitHub the `@` sign is used to tag users, it is replaced by `att.` in XPath expressions.This script takes a valid TEI-XML file as input, with all `//w[att.n and att.lemma and att.pos]`. It will return a CSV table with the number, text, lemma and part-of-speach of each `` element, for assessment or manual correction.For instance, if the original tokens are as follows:```xmlLaCoutumedeNormandie```...then the resulting table will be this:| ID | TOKEN | LEMMES | POS ||------------|----------------|-------------|-----------|| 1 | La | LE | Da || 2 | Coutume | COUTUME | Nc || 3 | de | DE | S || 4 | Normandie | NORMANDIE | Np |This script deals with elements within each ``, for instance `` or `` elements. It exports the full modernised text, with abbreviations resolved. FUNCTION: get the tokens and make the CSV output file<jupyter_code>def export_tokens_to_csv(chemin_entree, chemin_sortie):
    
    """
    Function taking the text of all elements from the
    input TEI-XML file, along with the value of their
    @n, @lemma and @pos attributes, to export all into
    a CSV file with one column for each information
    and one line per token.
    
    :param chemin_entree: The local path to the TEI-XML file
        whose <w> elements need to extracted into a table.
    :param chemin_sortie: The local path for the output file.
    """

    import xml.etree.ElementTree as ET
    import csv
    
    # The column headers for the CSV-to-be.
    colonnes = ['ID', 'TOKEN', 'LEMMES', 'POS']
    
    # Declare the TEI namespace, without a prefix since it is the only one.
    ET.register_namespace('', "http://tei-c.org/ns/1.0")
    
    # Import and parse the input XML file.
    tree = ET.parse(chemin_entree)
    root = tree.getroot()
    
    # Open an output CSV file in "writing" mode and write column headers.
    with open(chemin_sortie, 'w') as csv_file:
        csv_contenu = csv.DictWriter(csv_file, fieldnames = colonnes, delimiter=";")
        csv_contenu.writeheader()
        
        # Loop on all <w> elements in reading order.
        for word in root.findall('.//{http://tei-c.org/ns/1.0}w'):
        
            # Get the @n attribute into var numero,
            # same thing for @lemma and @pos,
            # and create an empty var to compose the text.
            numero = str(word.get('n'))
            lemmes = str(word.get('lemma'))
            pos = str(word.get('pos'))
            texte = ""
        
            # If the current <w> element has no child at all,
            # then the text can be taken directly.
            if word.find('.') == None :
                texte = str(word.text)
            
            # Otherwise the text needs to be compiled.
            else:
            
                # Add all text before the first child, if any.
                if word.text:
                    texte += str(word.text)
                
                # Loop on all children.
                for item in word:
                
                    # If the child is <height> or <supplied>, add its text directly.
                    if item.tag == '{http://tei-c.org/ns/1.0}height' or item.tag == '{http://tei-c.org/ns/1.0}supplied':
                        texte += str(item.text)
                        # If there is text between this child and the next,
                        # add it also.
                        if item.tail:
                            texte += str(item.tail)
                        
                    # If the child is a <lb> (line beginning), just check
                    # for text between it and the next child.
                    elif item.tag == '{http://tei-c.org/ns/1.0}lb':
                        if item.tail:
                            texte += str(item.tail)
                        
                    # If the child is a <choice> element, take its second child
                    # (<reg>, <expan> or <cor>) and check for text after <choice>.
                    elif item.tag == '{http://tei-c.org/ns/1.0}choice':
                        texte += str(item[1].text)
                        if item.tail:
                            texte += str(item.tail)
                    
                    # <c> = character (marks initials).
                    elif item.tag == '{http://tei-c.org/ns/1.0}c':
                        texte += item.text
                        if item.tail:
                            texte += str(item.tail)
                      
                    # <hi> = highlight (italics, color, etc)
                    elif item.tag == '{http://tei-c.org/ns/1.0}hi':
                        texte += item.text
                        if item.tail:
                            texte += item.tail
                    
                    # <add> = the scriptor added text in the interline.
                    elif item.tag == '{http://tei-c.org/ns/1.0}add':
                        # All tests need to be redone within the <add> element.
                        if item.find('.') == None :
                            texte = str(item.text)
                            
                        else:
                            
                            if item.text:
                                texte += str(item.text)
                            
                            for subitem in item:
                                if subitem.tag == '{http://tei-c.org/ns/1.0}lb':
                                    if subitem.tail:
                                        texte += str(subitem.tail)
                                elif subitem.tag == '{http://tei-c.org/ns/1.0}choice':
                                    texte += str(subitem[1].text)
                                    if subitem.tail:
                                        texte += str(subitem.tail)
                
            # Then write values into their appropriate column.
            csv_contenu.writerow(
                {
                    "ID" : numero,
                    "TOKEN" : str(texte),
                    "LEMMES" : lemmes,
                    "POS" : pos
                }
            )<jupyter_output><empty_output><jupyter_text>Executing the script<jupyter_code># To execute the function on a given file,
# write the path to said file as the first argument,
# and a new one for the output CSV.

export_tokens_to_csv('/home/erminea/Documents/CONDE/editions/base-version/rouille_base.xml',
                 '/home/erminea/Documents/CONDE/Encodage/ROUILLE/rouille_tokens_decalles.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1049490538573508, 'fraction_numerical': 0.0042212518195050945, 'mean_word_length': 1.332315003394433, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '447661', 'license': 'CC0-1.0', 'n_tokens_mistral': 1909, 'n_tokens_neox': 1823, 'n_words': 601, 'path': 'corpus-construction/disambiguate-lemmatization-in-corrected-file/REV_2_script_export_csv_tokens.ipynb', 'repo_name': 'RIN-ConDE/tools', 'size': '6870'}