<commit_before>Embedding-Based Selection
=========================

One of the answer features currently used uses a classifier based
on word embeddings to count probability of property or passage sentence
containing the correct answer.

We term this either:

  * Property selection (for structured data sources, based on the property
    label embedding)

  * Sentence selection (for unstructured data sources, "answering sentence
    selection" problem)


Property Selection
------------------

Word embeding dictionary is downloaded from our maven repository, while the
weights used are located in:

	src/main/resources/cz/brmlab/yodaqa/analysis/rdf/Mbprop.txt

To re-train the property selection model, first run:

	mkdir data/ml/embsel/propdata-curated-train
	./gradlew tsvgs -PexecArgs="data/eval/curated-train.tsv curated-train.tsv" -Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug -Dcz.brmlab.yodaqa.dump_property_labels=data/ml/embsel/propdata-curated-train 2>&1 | tee train_embsel.log

Then, take the data from data/ml/embsel and to re-train the weights,
use the toolset in:

	https://github.com/brmson/Sentence-selection

For more information, check the README there.


Sentence Selection
------------------

We didn't get this to work in a beneficial way yet.
Work in progress.<commit_msg>data/ml/embsel/README Propsel: We use moviesD now, both on curated and d/movies<commit_after>Embedding-Based Selection
=========================

One of the answer features currently used uses a classifier based
on word embeddings to count probability of property or passage sentence
containing the correct answer.

We term this either:

  * Property selection (for structured data sources, based on the property
    label embedding)

  * Sentence selection (for unstructured data sources, "answering sentence
    selection" problem)


Property Selection
------------------

Word embeding dictionary is downloaded from our maven repository, while the
weights used are located in:

	src/main/resources/cz/brmlab/yodaqa/analysis/rdf/Mbprop.txt

To re-train the property selection model, first run (we use the ``d/movies``
branch for this):

	mkdir data/ml/embsel/propdata-moviesD-train
	./gradlew tsvgs -PexecArgs="data/eval/moviesD-train.tsv moviesD-train.tsv" -Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug -Dcz.brmlab.yodaqa.dump_property_labels=data/ml/embsel/propdata-moviesD-train 2>&1 | tee train_embsel.log

Then, take the data from data/ml/embsel and to re-train the weights,
use the toolset in:

	https://github.com/brmson/Sentence-selection

For more information, check the README there.


Sentence Selection
------------------

We didn't get this to work in a beneficial way yet.
Work in progress.
{'fraction_non_alphanumeric': 0.11377025036818851, 'fraction_numerical': 0.0022091310751104565, 'mean_word_length': 4.022181146025878, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7452667', 'n_tokens_mistral': 833, 'n_tokens_neox': 788, 'n_words': 278}