natural language toolkit interface to maltparser dan garrette dhgarrettegmail com contributor liling tan mustufain osamamukhtar11 c 20012023 nltk project url https www nltk org for license information see license txt a module to find maltparser jar file and its dependencies checks that that the found directory contains all the necessary jar a module to find pretrained maltparser model a class for dependency parsing with maltparser the input is the paths to optionally a maltparser directory optionally the path to a pretrained maltparser mco model file optionally the tagger to use for pos tagging before parsing optionally additional java arguments example from nltk parse import malt with maltparser and maltmodel environment set mp malt maltparsermodelfilename engmalt linear1 7 mco doctest skip mp parseone i shot an elephant in my pajamas split tree doctest skip shot i elephant an in pajamas my without maltparser and maltmodel environment mp malt maltparser homeusermaltparser1 9 2 homeuserengmalt linear1 7 mco doctest skip mp parseone i shot an elephant in my pajamas split tree doctest skip shot i elephant an in pajamas my an interface for parsing with the malt parser param parserdirname the path to the maltparser directory that contains the maltparser1 x jar type parserdirname str param modelfilename the name of the pretrained model with mco file extension if provided training will not be required see http www maltparser orgmcomco html and see http www patful comchalknode185 type modelfilename str param tagger the tagger used to pos tag the raw string before formatting to conll format it should behave like nltk postag type tagger function param additionaljavaargs this is the additional java arguments that one can use when calling maltparser usually this is the heapsize limits e g additionaljavaargs xmx1024m see https goo glmpdbvq type additionaljavaargs list find all the necessary jar files for maltparser initialize additional java arguments initialize model set the workingdir parameters i e w from maltparser s option initialize pos tagger use maltparser to parse multiple pos tagged sentences takes multiple sentences where each sentence is a list of word tag tuples the sentences must have already been tokenized and tagged param sentences input sentences to parse type sentence listlisttuplestr str return iteriterdependencygraph the dependency graph representation of each sentence convert list of sentences to conll format generate command to run maltparser this is a maltparser quirk it needs to be run where the model file is otherwise it goes into an awkward missing jars or strange w workingdir problem must return iteritertree use maltparser to parse multiple sentences takes a list of sentences where each sentence is a list of words each sentence will be automatically tagged with this maltparser instance s tagger param sentences input sentences to parse type sentence listliststr return iterdependencygraph this function generates the maltparser command use at the terminal param inputfilename path to the input file type inputfilename str param outputfilename path to the output file type outputfilename str joins classpaths with if on windows and on linuxmac use adds the model file train maltparser from a list of dependencygraph objects param depgraphs list of dependencygraph objects for training input data type depgraphs dependencygraph write the conllstr to malttrain conll file in tmp trains the model with the malttrain conll removes the malttrain conll once training finishes train maltparser from a file param conllfile str for the filename of the training input data type conllfile str if conllfile is a zipfilepathpointer then we need to do some extra massaging generate command to run maltparser a demonstration function to show how nltk users can use the malt parser api from nltk import postag assert maltparser in os environ str please set maltparser in your global environment e g n export maltparser homeusermaltparser1 9 2 assert maltmodel in os environ str please set maltmodel in your global environment e g n export maltmodel homeuserengmalt linear1 7 mco dg1str str1 john nnp 2 subj n 2 sees vb 0 root n 3 a dt 4 spec n 4 dog nn 2 obj n 5 2 punct n dg2str str1 john nnp 2 subj n 2 walks vb 0 root n 3 2 punct n dg1 dependencygraphdg1str dg2 dependencygraphdg2str initialize a maltparser object mp maltparser trains a model mp traindg1 dg2 verbosefalse sent1 john sees mary sent2 john walks a dog parse a single sentence parsedsent1 mp parseonesent1 parsedsent2 mp parseonesent2 printparsedsent1 tree sees john mary printparsedsent2 tree walks john dog a parsing multiple sentences sentences sent1 sent2 parsedsents mp parsesentssentences printnextnextparsedsents tree sees john mary printnextnextparsedsents tree walks john dog a initialize a maltparser object with an english pretrained model parserdirname maltparser1 9 2 modelname engmalt linear1 7 mco mp maltparserparserdirnameparserdirname modelfilenamemodelname taggerpostag sent1 i shot an elephant in my pajamas split sent2 time flies like banana split parse a single sentence printmp parseonesent1 tree shot i elephant an in pajamas my parsing multiple sentences sentences sent1 sent2 parsedsents mp parsesentssentences printnextnextparsedsents tree shot i elephant an in pajamas my printnextnextparsedsents tree flies time like banana natural language toolkit interface to maltparser dan garrette dhgarrette gmail com contributor liling tan mustufain osamamukhtar11 c 2001 2023 nltk project url https www nltk org for license information see license txt fullstop comma qmark round brackets square brackets cardinal numbers articles pronouns possessive possessive time prepopsitions time prepopsitions time prepopsitions space prepopsitions space prepopsitions space prepopsitions space prepopsitions space prepopsitions space prepopsitions adjectives nouns formed from adjectives adverbs plural nouns gerunds past tense verbs nouns default a module to find maltparser jar file and its dependencies if a full path is given try to find path to maltparser directory in environment variables checks that that the found directory contains all the necessary jar a module to find pre trained maltparser model if a full path is given try to find path to malt model in environment variables a class for dependency parsing with maltparser the input is the paths to optionally a maltparser directory optionally the path to a pre trained maltparser mco model file optionally the tagger to use for pos tagging before parsing optionally additional java arguments example from nltk parse import malt with malt_parser and malt_model environment set mp malt maltparser model_filename engmalt linear 1 7 mco doctest skip mp parse_one i shot an elephant in my pajamas split tree doctest skip shot i elephant an in pajamas my without malt_parser and malt_model environment mp malt maltparser home user maltparser 1 9 2 home user engmalt linear 1 7 mco doctest skip mp parse_one i shot an elephant in my pajamas split tree doctest skip shot i elephant an in pajamas my an interface for parsing with the malt parser param parser_dirname the path to the maltparser directory that contains the maltparser 1 x jar type parser_dirname str param model_filename the name of the pre trained model with mco file extension if provided training will not be required see http www maltparser org mco mco html and see http www patful com chalk node 185 type model_filename str param tagger the tagger used to pos tag the raw string before formatting to conll format it should behave like nltk pos_tag type tagger function param additional_java_args this is the additional java arguments that one can use when calling maltparser usually this is the heapsize limits e g additional_java_args xmx1024m see https goo gl mpdbvq type additional_java_args list find all the necessary jar files for maltparser initialize additional java arguments initialize model set the working_dir parameters i e w from maltparser s option initialize pos tagger use maltparser to parse multiple pos tagged sentences takes multiple sentences where each sentence is a list of word tag tuples the sentences must have already been tokenized and tagged param sentences input sentences to parse type sentence list list tuple str str return iter iter dependencygraph the dependency graph representation of each sentence convert list of sentences to conll format generate command to run maltparser this is a maltparser quirk it needs to be run where the model file is otherwise it goes into an awkward missing jars or strange w working_dir problem remembers the current path change to modelfile path run command change back to current path must return iter iter tree use maltparser to parse multiple sentences takes a list of sentences where each sentence is a list of words each sentence will be automatically tagged with this maltparser instance s tagger param sentences input sentences to parse type sentence list list str return iter dependencygraph this function generates the maltparser command use at the terminal param inputfilename path to the input file type inputfilename str param outputfilename path to the output file type outputfilename str adds additional java arguments joins classpaths with if on windows and on linux mac use adds classpaths for jars adds the main function adds the model file when parsing when learning mode use to generate parses train maltparser from a list of dependencygraph objects param depgraphs list of dependencygraph objects for training input data type depgraphs dependencygraph write the conll_str to malt_train conll file in tmp trains the model with the malt_train conll removes the malt_train conll once training finishes train maltparser from a file param conll_file str for the filename of the training input data type conll_file str if conll_file is a zipfilepathpointer then we need to do some extra massaging generate command to run maltparser a demonstration function to show how nltk users can use the malt parser api from nltk import pos_tag assert malt_parser in os environ str please set malt_parser in your global environment e g n export malt_parser home user maltparser 1 9 2 assert malt_model in os environ str please set malt_model in your global environment e g n export malt_model home user engmalt linear 1 7 mco _dg1_str str 1 john _ nnp _ _ 2 subj _ _ n 2 sees _ vb _ _ 0 root _ _ n 3 a _ dt _ _ 4 spec _ _ n 4 dog _ nn _ _ 2 obj _ _ n 5 _ _ _ 2 punct _ _ n _dg2_str str 1 john _ nnp _ _ 2 subj _ _ n 2 walks _ vb _ _ 0 root _ _ n 3 _ _ _ 2 punct _ _ n dg1 dependencygraph _dg1_str dg2 dependencygraph _dg2_str initialize a maltparser object mp maltparser trains a model mp train dg1 dg2 verbose false sent1 john sees mary sent2 john walks a dog parse a single sentence parsed_sent1 mp parse_one sent1 parsed_sent2 mp parse_one sent2 print parsed_sent1 tree sees john mary print parsed_sent2 tree walks john dog a parsing multiple sentences sentences sent1 sent2 parsed_sents mp parse_sents sentences print next next parsed_sents tree sees john mary print next next parsed_sents tree walks john dog a initialize a maltparser object with an english pre trained model parser_dirname maltparser 1 9 2 model_name engmalt linear 1 7 mco mp maltparser parser_dirname parser_dirname model_filename model_name tagger pos_tag sent1 i shot an elephant in my pajamas split sent2 time flies like banana split parse a single sentence print mp parse_one sent1 tree shot i elephant an in pajamas my parsing multiple sentences sentences sent1 sent2 parsed_sents mp parse_sents sentences print next next parsed_sents tree shot i elephant an in pajamas my print next next parsed_sents tree flies time like banana
import inspect
import os
import subprocess
import sys
import tempfile

from nltk.data import ZipFilePathPointer
from nltk.internals import find_dir, find_file, find_jars_within_path
from nltk.parse.api import ParserI
from nltk.parse.dependencygraph import DependencyGraph
from nltk.parse.util import taggedsents_to_conll


def malt_regex_tagger():
    from nltk.tag import RegexpTagger

    _tagger = RegexpTagger(
        [
            (r"\.$", "."),
            (r"\,$", ","),
            (r"\?$", "?"),  
            (r"\($", "("),
            (r"\)$", ")"),  
            (r"\[$", "["),
            (r"\]$", "]"),  
            (r"^-?[0-9]+(\.[0-9]+)?$", "CD"),  
            (r"(The|the|A|a|An|an)$", "DT"),  
            (r"(He|he|She|she|It|it|I|me|Me|You|you)$", "PRP"),  
            (r"(His|his|Her|her|Its|its)$", "PRP$"),  
            (r"(my|Your|your|Yours|yours)$", "PRP$"),  
            (r"(on|On|in|In|at|At|since|Since)$", "IN"),  
            (r"(for|For|ago|Ago|before|Before)$", "IN"),  
            (r"(till|Till|until|Until)$", "IN"),  
            (r"(by|By|beside|Beside)$", "IN"),  
            (r"(under|Under|below|Below)$", "IN"),  
            (r"(over|Over|above|Above)$", "IN"),  
            (r"(across|Across|through|Through)$", "IN"),  
            (r"(into|Into|towards|Towards)$", "IN"),  
            (r"(onto|Onto|from|From)$", "IN"),  
            (r".*able$", "JJ"),  
            (r".*ness$", "NN"),  
            (r".*ly$", "RB"),  
            (r".*s$", "NNS"),  
            (r".*ing$", "VBG"),  
            (r".*ed$", "VBD"),  
            (r".*", "NN"),  
        ]
    )
    return _tagger.tag


def find_maltparser(parser_dirname):
    
    if os.path.exists(parser_dirname):  
        _malt_dir = parser_dirname
    else:  
        _malt_dir = find_dir(parser_dirname, env_vars=("MALT_PARSER",))
    
    malt_dependencies = ["", "", ""]
    _malt_jars = set(find_jars_within_path(_malt_dir))
    _jars = {os.path.split(jar)[1] for jar in _malt_jars}
    malt_dependencies = {"log4j.jar", "libsvm.jar", "liblinear-1.8.jar"}

    assert malt_dependencies.issubset(_jars)
    assert any(
        filter(lambda i: i.startswith("maltparser-") and i.endswith(".jar"), _jars)
    )
    return list(_malt_jars)


def find_malt_model(model_filename):
    
    if model_filename is None:
        return "malt_temp.mco"
    elif os.path.exists(model_filename):  
        return model_filename
    else:  
        return find_file(model_filename, env_vars=("MALT_MODEL",), verbose=False)


class MaltParser(ParserI):
    

    def __init__(
        self,
        parser_dirname="",
        model_filename=None,
        tagger=None,
        additional_java_args=None,
    ):
        

        
        self.malt_jars = find_maltparser(parser_dirname)
        
        self.additional_java_args = (
            additional_java_args if additional_java_args is not None else []
        )
        
        self.model = find_malt_model(model_filename)
        self._trained = self.model != "malt_temp.mco"
        
        self.working_dir = tempfile.gettempdir()
        
        self.tagger = tagger if tagger is not None else malt_regex_tagger()

    def parse_tagged_sents(self, sentences, verbose=False, top_relation_label="null"):
        
        if not self._trained:
            raise Exception("Parser has not been trained. Call train() first.")

        with tempfile.NamedTemporaryFile(
            prefix="malt_input.conll.", dir=self.working_dir, mode="w", delete=False
        ) as input_file:
            with tempfile.NamedTemporaryFile(
                prefix="malt_output.conll.",
                dir=self.working_dir,
                mode="w",
                delete=False,
            ) as output_file:
                
                for line in taggedsents_to_conll(sentences):
                    input_file.write(str(line))
                input_file.close()

                
                cmd = self.generate_malt_command(
                    input_file.name, output_file.name, mode="parse"
                )

                
                
                
                _current_path = os.getcwd()  
                try:  
                    os.chdir(os.path.split(self.model)[0])
                except:
                    pass
                ret = self._execute(cmd, verbose)  
                os.chdir(_current_path)  

                if ret != 0:
                    raise Exception(
                        "MaltParser parsing (%s) failed with exit "
                        "code %d" % (" ".join(cmd), ret)
                    )

                
                with open(output_file.name) as infile:
                    for tree_str in infile.read().split("\n\n"):
                        yield (
                            iter(
                                [
                                    DependencyGraph(
                                        tree_str, top_relation_label=top_relation_label
                                    )
                                ]
                            )
                        )

        os.remove(input_file.name)
        os.remove(output_file.name)

    def parse_sents(self, sentences, verbose=False, top_relation_label="null"):
        
        tagged_sentences = (self.tagger(sentence) for sentence in sentences)
        return self.parse_tagged_sents(
            tagged_sentences, verbose, top_relation_label=top_relation_label
        )

    def generate_malt_command(self, inputfilename, outputfilename=None, mode=None):
        

        cmd = ["java"]
        cmd += self.additional_java_args  
        
        classpaths_separator = ";" if sys.platform.startswith("win") else ":"
        cmd += [
            "-cp",
            classpaths_separator.join(self.malt_jars),
        ]  
        cmd += ["org.maltparser.Malt"]  

        
        if os.path.exists(self.model):  
            cmd += ["-c", os.path.split(self.model)[-1]]
        else:  
            cmd += ["-c", self.model]

        cmd += ["-i", inputfilename]
        if mode == "parse":
            cmd += ["-o", outputfilename]
        cmd += ["-m", mode]  
        return cmd

    @staticmethod
    def _execute(cmd, verbose=False):
        output = None if verbose else subprocess.PIPE
        p = subprocess.Popen(cmd, stdout=output, stderr=output)
        return p.wait()

    def train(self, depgraphs, verbose=False):
        

        
        with tempfile.NamedTemporaryFile(
            prefix="malt_train.conll.", dir=self.working_dir, mode="w", delete=False
        ) as input_file:
            input_str = "\n".join(dg.to_conll(10) for dg in depgraphs)
            input_file.write(str(input_str))
        
        self.train_from_file(input_file.name, verbose=verbose)
        
        os.remove(input_file.name)

    def train_from_file(self, conll_file, verbose=False):
        

        
        
        if isinstance(conll_file, ZipFilePathPointer):
            with tempfile.NamedTemporaryFile(
                prefix="malt_train.conll.", dir=self.working_dir, mode="w", delete=False
            ) as input_file:
                with conll_file.open() as conll_input_file:
                    conll_str = conll_input_file.read()
                    input_file.write(str(conll_str))
                return self.train_from_file(input_file.name, verbose=verbose)

        
        cmd = self.generate_malt_command(conll_file, mode="learn")
        ret = self._execute(cmd, verbose)
        if ret != 0:
            raise Exception(
                "MaltParser training (%s) failed with exit "
                "code %d" % (" ".join(cmd), ret)
            )
        self._trained = True


if __name__ == "__main__":
    

    import doctest

    doctest.testmod()