<jupyter_start><jupyter_text>Surface realization<jupyter_code>from surface import grammar
from surface import converter
from surface import utils
from collections import defaultdict
import ast<jupyter_output><empty_output><jupyter_text>First we initialize the training and the test file to a variable, the files can be downloaded from the SRST 19 page.<jupyter_code>TRAIN_FILE = "pt_bosque-ud-train.conllu"
TEST_FILE = "pt_bosque-Pred-Stanford.conllu"<jupyter_output><empty_output><jupyter_text>Then, we train the two static grammars (the first corresponds to the subgraphs from the ud trees, the second is the fallback grammar, where each rule is binary)Later, the dynamic grammars are generated from these ones.<jupyter_code>grammar.train_subgraphs(TRAIN_FILE, TEST_FILE)
grammar.train_edges(TRAIN_FILE, TEST_FILE)
SUBGRAPH_GRAMMAR_FILE = "train_subgraphs"
EDGE_GRAMMAR_FILE = "train_edges"<jupyter_output><empty_output><jupyter_text>We need to extract the graphs from the conll format (conversion from conll to isi), and the rules that use the lin feature.The rules are for incorporating the lin feature, so we can dynamically delete every rule the contradicts the linearity.<jupyter_code>rules, _ = converter.extract_rules(TEST_FILE)
graphs, _, id_graphs= converter.convert(TEST_FILE)
_, sentences, _ = converter.convert(TEST_FILE)
conll = grammar.get_conll_from_file(TEST_FILE)
id_to_parse = {}
stops = []<jupyter_output><empty_output><jupyter_text>We run through the sentences and call the alto parser to generate the derivation and map the ud representation to string.The alto can be downloaded from [bitbucket](https://bitbucket.org/tclup/alto/downloads/).<jupyter_code>for sen_id in range(0, len(rules)):
    print(sen_id)
    try:
        grammar_fn = open('dep_grammar_spec.irtg', 'w') 
        grammar.generate_grammar(SUBGRAPH_GRAMMAR_FILE, rules[sen_id], grammar_fn)
        grammar.generate_terminal_ids(conll[sen_id], grammar_fn)
        grammar_fn.close()
        utils.set_parse("ewt_ones", id_graphs[sen_id])
        !timeout 70 java -Xmx32G -cp alto-2.3.6-SNAPSHOT-all.jar de.up.ling.irtg.script.ParsingEvaluator ewt_ones -g dep_grammar_spec.irtg -I ud -O string=toString -o surface_eval_ewt
        text_parse, conll_parse = utils.get_parse("surface_eval_ewt", conll[sen_id])
        id_to_parse[sen_id] = (text_parse, conll_parse)
    except StopIteration:
        print("stop iteratioin")
        stops.append(sen_id)
        continue<jupyter_output><empty_output><jupyter_text>We then iterate through the sentences that took too long to parse with the original grammar, and switch to the binary grammar for faster results.<jupyter_code>for sen_id in stops:
    grammar_fn = open('dep_grammar_edges.irtg', 'w') 
    grammar.generate_grammar(EDGE_GRAMMAR_FILE, rules[sen_id], grammar_fn)
    grammar.generate_terminal_ids(conll[sen_id], grammar_fn)
    grammar_fn.close()
    set_parse("ewt_ones", id_graphs[sen_id])
    !java -Xmx32G -cp alto-2.3.6-SNAPSHOT-all.jar de.up.ling.irtg.script.ParsingEvaluator ewt_ones -g dep_grammar_edges.irtg -I ud -O string=toString -o surface_eval_ewt
    text_parse, conll_parse = get_parse("surface_eval_ewt", conll[sen_id])
    id_to_parse[sen_id] = (text_parse, conll_parse)
with open("pt_bosque-Pred-Stanford.conllu" , "w") as f:
    for i in id_to_parse:
        conll_f = id_to_parse[i][1]
        for line in conll_f:
            f.write(str(line) + "\t")
            f.write("\t".join(conll_f[line]))
        f.write("\n")
converter.to_tokenized_output("test-results-inflected/", "tokenized_test_results/")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09550249861188229, 'fraction_numerical': 0.004441976679622432, 'mean_word_length': 5.158974358974359, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '286651', 'license': 'MIT', 'n_tokens_mistral': 1278, 'n_tokens_neox': 1231, 'n_words': 288, 'path': 'code/surface_realization.ipynb', 'repo_name': 'evelinacs/semantic_parsing_with_IRTGs', 'size': '3602'}