<commit_before>;
; link-pipeline.scm
;
; Link-grammar word and link-counting pipeline.  Currently counts
; words, several kinds of word-pairs (links, and order-relations),
; and also disjuncts, parses and sentences.
;
; Copyright (c) 2013, 2017 <NAME> <<EMAIL>>
;
; This code is part of the language-learning effort.  The project
; requires that a lot of text be observed, with the goal of deducing
; a grammar from it, using entropy and other basic probability methods.
;
; Main entry point: `(observe-text plain-text)`
;
; Call this entry point with exactly one sentence as a plain text
; string. It will be parsed, and the resulting link-grammar link usage
; counts will be updated in the atomspace. The counts are flushed to
; the SQL database so that they're not forgotten.
;
; This tracks multiple, independent counts:
; *) how many sentences have been observed.
; *) how many parses were observed.
; *) how many words have been observed (counting once-per-word-per-parse)
; *) how many word-order pairs have been observed.
; *) the distance between words in the above pairs.
; *) how many link-relationship triples have been observed.
; *) how many disjuncts have been observed.
;
; Sentences are counted by updating the count on `(SentenceNode "ANY")`.
; Parses are counted by updating the count on `(ParseNode "ANY")`.
; Words are counted by updating the count on the `WordNode` for that
; word. It is counted with multiplicity: once for each time it occurs
; in a parse.  That is, if a word appears twice in a parse, it is counted
; twice.
;
; Word-pairs show up, and are counted in four different ways. First,
; a count is made if two words appear, left-right ordered, in the same
; sentence. This count is stored in the CountTV for the EvaluationLink
; on (PredicateNode "*-Sentence Word Pair-*").  A second count is
; maintained for this same pair, but including the distance between the
; two words. This is on (SchemaNode "*-Pair Distance-*").  Since
; sentences always start with LEFT-WALL, this can be used to reconstruct
; the typical word-order in a sentence.
;
; Word-pairs are also designated by means of Link Grammar parses of a
; sentence. A Link Grammar parse creates a list of typed links between
; pairs of words in the sentence. Each such link is counted once, for
; each time that it occurs.  These counts are maintained in the CountTV
; on the EvaluationLink for the LinkGrammarRelationshipNode for that
; word-pair.  In addition, a count is maintained of the length of that
; link.
;
; For the initial stages of the language-learning project, the parses
; are produced by the "any" language parser, which produces random planar
; trees.  This creates a sampling of word-pairs that is different than
; merely having them show up in the same sentence.  That is, a covering
; of a sentence by random trees does not produce the same edge statistics
; as a clique of edges drawn between all words. This is explored further
; in the diary, in a section devoted to this topic.
;
; The Link Grammar parse also produces and reports the disjuncts that were
; used for each word. These are useful in and of themselves; they indicate
; the hubbiness (link-multiplicity) of each word. The disjunct counts are
; maintained on the LgWordCset for a given word.
;
(use-modules (opencog) (opencog nlp) (opencog persist))<commit_msg>updated file description<commit_after>;
; link-pipeline.scm
;
; Link-grammar word and link-counting pipeline.  Currently counts
; words, several kinds of word-pairs (links, and order-relations),
; and also disjuncts, parses and sentences.
;
; Copyright (c) 2013, 2017 <NAME> <<EMAIL>>
;
; This code is part of the language-learning effort.  The project
; requires that a lot of text be observed, with the goal of deducing
; a grammar from it, using entropy and other basic probability methods.
;
; Main entry point: `(observe-text plain-text)`
;
; Call this entry point with exactly one sentence as a plain text
; string. It will be parsed, and the resulting link-grammar link usage
; counts will be updated in the atomspace. The counts are flushed to
; the SQL database so that they're not forgotten.
;
; This tracks multiple, independent counts:
; *) how many sentences have been observed.
; *) how many parses were observed.
; *) how many words have been observed (counting once-per-word-per-parse)
; *) how many word-order pairs have been observed.
; *) the distance between words in the above pairs.
; *) how many link-relationship triples have been observed.
; *) how many disjuncts have been observed.
;
; Sentences are counted by updating the count on `(SentenceNode "ANY")`.
; Parses are counted by updating the count on `(ParseNode "ANY")`.
; Words are counted by updating the count on the `WordNode` for that
; word. It is counted with multiplicity: once for each time it occurs
; in a parse.  That is, if a word appears twice in a parse, it is counted
; twice.
;
; Word-pairs show up, and are counted in four different ways. First,
; a count is made if two words appear, left-right ordered, in the same
; sentence. This count is stored in the CountTV for the EvaluationLink
; on (PredicateNode "*-Sentence Word Pair-*").  A second count is
; maintained for this same pair, but including the distance between the
; two words. This is on (SchemaNode "*-Pair Distance-*").  Since
; sentences always start with LEFT-WALL, this can be used to reconstruct
; the typical word-order in a sentence.
;
; Word-pairs are also designated by means of Link Grammar parses of a
; sentence. A Link Grammar parse creates a list of typed links between
; pairs of words in the sentence. Each such link is counted once, for
; each time that it occurs.  These counts are maintained in the CountTV
; on the EvaluationLink for the LinkGrammarRelationshipNode for that
; word-pair.  In addition, a count is maintained of the length of that
; link.
;
; For the initial stages of the language-learning project, the parses
; are produced by the "any" language parser, which produces random planar
; trees.  This creates a sampling of word-pairs that is different than
; merely having them show up in the same sentence.  That is, a covering
; of a sentence by random trees does not produce the same edge statistics
; as a clique of edges drawn between all words. This is explored further
; in the diary, in a section devoted to this topic.
;
; Update: Additional counting modes can be requested by calling
;         observe-text-mode. See function's documentation for details.
;
; The Link Grammar parse also produces and reports the disjuncts that were
; used for each word. These are useful in and of themselves; they indicate
; the hubbiness (link-multiplicity) of each word. The disjunct counts are
; maintained on the LgWordCset for a given word.
;
(use-modules (opencog) (opencog nlp) (opencog persist))
{'fraction_non_alphanumeric': 0.06889994148624927, 'fraction_numerical': 0.0023405500292568754, 'mean_word_length': 4.121348314606742, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5202927', 'n_tokens_mistral': 1858, 'n_tokens_neox': 1805, 'n_words': 1026}