natural language toolkit xml corpus reader c 20012023 nltk project steven bird stevenbird1gmail com url https www nltk org for license information see license txt corpus reader for corpora whose documents are xml files note not named xml to avoid conflicting w standard xml package corpus reader for corpora whose documents are xml files note that the xmlcorpusreader constructor does not take an encoding argument because the unicode encoding is specified by the xml files themselves see the xml specs for more info make sure we have exactly one file no concatenating xml read the xml in using elementtree if requested wrap it return the elementtree element returns all of the words and punctuation symbols in the specified file that were in text nodes ie tags are ignored like the xml method fileid can only specify one file return the given file s text nodes as a list of words and punctuation symbols rtype liststr a corpus view that selects out specified elements from an xml file and provides a flat listlike interface for accessing them note xmlcorpusview is not used by xmlcorpusreader itself but may be used by subclasses of xmlcorpusreader every xml corpus view has a tag specification indicating what xml elements should be included in the view and each nonnested element that matches this specification corresponds to one item in the view tag specifications are regular expressions over tag paths where a tag path is a list of element tag names separated by indicating the ancestry of the element some examples foo a toplevel element whose tag is foo foobar an element whose tag is bar and whose parent is a toplevel element whose tag is foo foo an element whose tag is foo appearing anywhere in the xml tree foobar an wlement whose tag is foo or bar appearing anywhere in the xml tree the view items are generated from the selected xml elements via the method handleelt by default this method returns the element asis i e as an elementtree object but it can be overridden either via subclassing or via the elthandler constructor parameter if true then display debugging output to stdout when reading blocks the number of characters read at a time by this corpus reader create a new corpus view based on a specified xml file note that the xmlcorpusview constructor does not take an encoding argument because the unicode encoding is specified by the xml files themselves type tagspec str param tagspec a tag specification indicating what xml elements should be included in the view each nonnested element that matches this specification corresponds to one item in the view param elthandler a function used to transform each element to a value for the view if no handler is specified then self handleelt is called which returns the element as an elementtree object the signature of elthandler is elthandlerelt tagspec value the tag specification for this corpus view self tagcontext 0 no encoding found what should the default be convert an element into an appropriate value for inclusion in the view unless overridden by a subclass or by the elthandler constructor argument this method simply returns elt return the view value corresponding to elt type elt elementtree param elt the element that should be converted type context str param context a string composed of element tags separated by forward slashes indicating the xml context of the given element for example the string foobarbaz indicates that the element is a baz element whose parent is a bar element and whose grandparent is a toplevel foo element a regular expression that matches xml fragments that do not contain any unclosed tags a regular expression used to extract the tag name from a start tag end tag or emptyelt tag string a regular expression used to find all starttags endtags and emptyelt tags in an xml file this regexp is more lenient than the xml spec e g it allows spaces in some places where the spec does not include these so we can skip them these are the ones we actually care about read a string from the given stream that does not contain any unclosed tags in particular this function first reads a block from the stream of size self blocksize it then checks if that block contains an unclosed tag if it does then this function either backtracks to the last or reads another block read a block and add it to the fragment do we have a wellformed xml fragment do we have a fragment that will never be wellformed end of file if not then we must be in the middle of a tag if appropriate backtrack to the most recent character otherwise read another block i e return to the top of the loop read from stream until we find at least one element that matches tagspec and return the result of applying elthandler to each element found use a stack of strings to keep track of our context end of file process each tag in the xml fragment keep context uptodate is this one of the elts we re looking for sanity checks is this the end of an element keep context uptodate if we haven t found any elements yet then keep looping until we do if we ve found at least one element then try backtracking to the start of the element that we re inside of take back the last starttag and return what we ve gotten so far elts is nonempty update the tagcontext dict natural language toolkit xml corpus reader c 2001 2023 nltk project steven bird stevenbird1 gmail com url https www nltk org for license information see license txt corpus reader for corpora whose documents are xml files note not named xml to avoid conflicting w standard xml package corpus reader for corpora whose documents are xml files note that the xmlcorpusreader constructor does not take an encoding argument because the unicode encoding is specified by the xml files themselves see the xml specs for more info make sure we have exactly one file no concatenating xml read the xml in using elementtree if requested wrap it return the elementtree element returns all of the words and punctuation symbols in the specified file that were in text nodes ie tags are ignored like the xml method fileid can only specify one file return the given file s text nodes as a list of words and punctuation symbols rtype list str a corpus view that selects out specified elements from an xml file and provides a flat list like interface for accessing them note xmlcorpusview is not used by xmlcorpusreader itself but may be used by subclasses of xmlcorpusreader every xml corpus view has a tag specification indicating what xml elements should be included in the view and each non nested element that matches this specification corresponds to one item in the view tag specifications are regular expressions over tag paths where a tag path is a list of element tag names separated by indicating the ancestry of the element some examples foo a top level element whose tag is foo foo bar an element whose tag is bar and whose parent is a top level element whose tag is foo foo an element whose tag is foo appearing anywhere in the xml tree foo bar an wlement whose tag is foo or bar appearing anywhere in the xml tree the view items are generated from the selected xml elements via the method handle_elt by default this method returns the element as is i e as an elementtree object but it can be overridden either via subclassing or via the elt_handler constructor parameter if true then display debugging output to stdout when reading blocks the number of characters read at a time by this corpus reader create a new corpus view based on a specified xml file note that the xmlcorpusview constructor does not take an encoding argument because the unicode encoding is specified by the xml files themselves type tagspec str param tagspec a tag specification indicating what xml elements should be included in the view each non nested element that matches this specification corresponds to one item in the view param elt_handler a function used to transform each element to a value for the view if no handler is specified then self handle_elt is called which returns the element as an elementtree object the signature of elt_handler is elt_handler elt tagspec value the tag specification for this corpus view a dictionary mapping from file positions as returned by stream seek to xml contexts an xml context is a tuple of xml tag names indicating which tags have not yet been closed no encoding found what should the default be convert an element into an appropriate value for inclusion in the view unless overridden by a subclass or by the elt_handler constructor argument this method simply returns elt return the view value corresponding to elt type elt elementtree param elt the element that should be converted type context str param context a string composed of element tags separated by forward slashes indicating the xml context of the given element for example the string foo bar baz indicates that the element is a baz element whose parent is a bar element and whose grandparent is a top level foo element a regular expression that matches xml fragments that do not contain any un closed tags comment cdata raw character data doctype s s doctype decl tag or pi z a regular expression used to extract the tag name from a start tag end tag or empty elt tag string a regular expression used to find all start tags end tags and empty elt tags in an xml file this regexp is more lenient than the xml spec e g it allows spaces in some places where the spec does not include these so we can skip them p comment p cdata cdata p pi p doctype doctype s s these are the ones we actually care about p empty_elt_tag s s s p start_tag s s p end_tag s s read a string from the given stream that does not contain any un closed tags in particular this function first reads a block from the stream of size self _block_size it then checks if that block contains an un closed tag if it does then this function either backtracks to the last or reads another block read a block and add it to the fragment do we have a well formed xml fragment do we have a fragment that will never be well formed end of file if not then we must be in the middle of a tag if appropriate backtrack to the most recent character otherwise read another block i e return to the top of the loop read from stream until we find at least one element that matches tagspec and return the result of applying elt_handler to each element found use a stack of strings to keep track of our context check this could it ever happen where does the elt start what context depth end of file process each tag in the xml fragment keep context up to date is this one of the elts we re looking for sanity checks is this the end of an element keep context up to date if we haven t found any elements yet then keep looping until we do if we ve found at least one element then try backtracking to the start of the element that we re inside of take back the last start tag and return what we ve gotten so far elts is non empty update the _tag_context dict
import codecs
from xml.etree import ElementTree

from nltk.corpus.reader.api import CorpusReader
from nltk.corpus.reader.util import *
from nltk.data import SeekableUnicodeStreamReader
from nltk.internals import ElementWrapper
from nltk.tokenize import WordPunctTokenizer


class XMLCorpusReader(CorpusReader):
    

    def __init__(self, root, fileids, wrap_etree=False):
        self._wrap_etree = wrap_etree
        CorpusReader.__init__(self, root, fileids)

    def xml(self, fileid=None):
        
        if fileid is None and len(self._fileids) == 1:
            fileid = self._fileids[0]
        if not isinstance(fileid, str):
            raise TypeError("Expected a single file identifier string")
        
        with self.abspath(fileid).open() as fp:
            elt = ElementTree.parse(fp).getroot()
        
        if self._wrap_etree:
            elt = ElementWrapper(elt)
        
        return elt

    def words(self, fileid=None):
        

        elt = self.xml(fileid)
        encoding = self.encoding(fileid)
        word_tokenizer = WordPunctTokenizer()
        try:
            iterator = elt.getiterator()
        except:
            iterator = elt.iter()
        out = []

        for node in iterator:
            text = node.text
            if text is not None:
                if isinstance(text, bytes):
                    text = text.decode(encoding)
                toks = word_tokenizer.tokenize(text)
                out.extend(toks)
        return out


class XMLCorpusView(StreamBackedCorpusView):
    

    
    
    _DEBUG = False

    
    _BLOCK_SIZE = 1024

    def __init__(self, fileid, tagspec, elt_handler=None):
        
        if elt_handler:
            self.handle_elt = elt_handler

        self._tagspec = re.compile(tagspec + r"\Z")
        

        self._tag_context = {0: ()}
        

        encoding = self._detect_encoding(fileid)
        StreamBackedCorpusView.__init__(self, fileid, encoding=encoding)

    def _detect_encoding(self, fileid):
        if isinstance(fileid, PathPointer):
            try:
                infile = fileid.open()
                s = infile.readline()
            finally:
                infile.close()
        else:
            with open(fileid, "rb") as infile:
                s = infile.readline()
        if s.startswith(codecs.BOM_UTF16_BE):
            return "utf-16-be"
        if s.startswith(codecs.BOM_UTF16_LE):
            return "utf-16-le"
        if s.startswith(codecs.BOM_UTF32_BE):
            return "utf-32-be"
        if s.startswith(codecs.BOM_UTF32_LE):
            return "utf-32-le"
        if s.startswith(codecs.BOM_UTF8):
            return "utf-8"
        m = re.match(rb'\s*<\?xml\b.*\bencoding="([^"]+)"', s)
        if m:
            return m.group(1).decode()
        m = re.match(rb"\s*<\?xml\b.*\bencoding='([^']+)'", s)
        if m:
            return m.group(1).decode()
        
        return "utf-8"

    def handle_elt(self, elt, context):
        
        return elt

    
    
    _VALID_XML_RE = re.compile(
        r,
        re.DOTALL | re.VERBOSE,
    )

    
    
    _XML_TAG_NAME = re.compile(r"<\s*(?:/\s*)?([^\s>]+)")

    
    
    
    
    _XML_PIECE = re.compile(
        r,
        re.DOTALL | re.VERBOSE,
    )

    def _read_xml_fragment(self, stream):
        
        fragment = ""

        if isinstance(stream, SeekableUnicodeStreamReader):
            startpos = stream.tell()
        while True:
            
            xml_block = stream.read(self._BLOCK_SIZE)
            fragment += xml_block

            
            if self._VALID_XML_RE.match(fragment):
                return fragment

            
            if re.search("[<>]", fragment).group(0) == ">":
                pos = stream.tell() - (
                    len(fragment) - re.search("[<>]", fragment).end()
                )
                raise ValueError('Unexpected ">" near char %s' % pos)

            
            if not xml_block:
                raise ValueError("Unexpected end of file: tag not closed")

            
            
            
            last_open_bracket = fragment.rfind("<")
            if last_open_bracket > 0:
                if self._VALID_XML_RE.match(fragment[:last_open_bracket]):
                    if isinstance(stream, SeekableUnicodeStreamReader):
                        stream.seek(startpos)
                        stream.char_seek_forward(last_open_bracket)
                    else:
                        stream.seek(-(len(fragment) - last_open_bracket), 1)
                    return fragment[:last_open_bracket]

            
            

    def read_block(self, stream, tagspec=None, elt_handler=None):
        
        if tagspec is None:
            tagspec = self._tagspec
        if elt_handler is None:
            elt_handler = self.handle_elt

        
        context = list(self._tag_context.get(stream.tell()))
        assert context is not None  

        elts = []

        elt_start = None  
        elt_depth = None  
        elt_text = ""

        while elts == [] or elt_start is not None:
            if isinstance(stream, SeekableUnicodeStreamReader):
                startpos = stream.tell()
            xml_fragment = self._read_xml_fragment(stream)

            
            if not xml_fragment:
                if elt_start is None:
                    break
                else:
                    raise ValueError("Unexpected end of file")

            
            for piece in self._XML_PIECE.finditer(xml_fragment):
                if self._DEBUG:
                    print("{:>25} {}".format("/".join(context)[-20:], piece.group()))

                if piece.group("START_TAG"):
                    name = self._XML_TAG_NAME.match(piece.group()).group(1)
                    
                    context.append(name)
                    
                    if elt_start is None:
                        if re.match(tagspec, "/".join(context)):
                            elt_start = piece.start()
                            elt_depth = len(context)

                elif piece.group("END_TAG"):
                    name = self._XML_TAG_NAME.match(piece.group()).group(1)
                    
                    if not context:
                        raise ValueError("Unmatched tag </%s>" % name)
                    if name != context[-1]:
                        raise ValueError(f"Unmatched tag <{context[-1]}>...</{name}>")
                    
                    if elt_start is not None and elt_depth == len(context):
                        elt_text += xml_fragment[elt_start : piece.end()]
                        elts.append((elt_text, "/".join(context)))
                        elt_start = elt_depth = None
                        elt_text = ""
                    
                    context.pop()

                elif piece.group("EMPTY_ELT_TAG"):
                    name = self._XML_TAG_NAME.match(piece.group()).group(1)
                    if elt_start is None:
                        if re.match(tagspec, "/".join(context) + "/" + name):
                            elts.append((piece.group(), "/".join(context) + "/" + name))

            if elt_start is not None:
                
                
                if elts == []:
                    elt_text += xml_fragment[elt_start:]
                    elt_start = 0

                
                
                
                else:
                    
                    
                    if self._DEBUG:
                        print(" " * 36 + "(backtrack)")
                    if isinstance(stream, SeekableUnicodeStreamReader):
                        stream.seek(startpos)
                        stream.char_seek_forward(elt_start)
                    else:
                        stream.seek(-(len(xml_fragment) - elt_start), 1)
                    context = context[: elt_depth - 1]
                    elt_start = elt_depth = None
                    elt_text = ""

        
        pos = stream.tell()
        if pos in self._tag_context:
            assert tuple(context) == self._tag_context[pos]
        else:
            self._tag_context[pos] = tuple(context)

        return [
            elt_handler(
                ElementTree.fromstring(elt.encode("ascii", "xmlcharrefreplace")),
                context,
            )
            for (elt, context) in elts
        ]