<commit_before>==================
 JMdict DB Schema
==================

Original JMdict XML format (summarized)
=======================================

::

  Entry
    - ent_seq
    * k_ele
      - keb
      * ke_inf
      * ke_pri
    + r_ele
      - reb
      ? re_nokanji
      * re_restr
      * re_inf
      * re_pri
    ? info
      * links
        - link_tag
        - link_desc
        - link_uri
      * bibl
        ? bib_tag
        ? bib_txt
      * etym (UNUSED)
      * audit
        - upd_date
        - upd_detl
    + sense
      * stagk
      * stagr
      * pos
      * xref*  # not sure why it's (#PCDATA)*...
      * ant*   # not sure why it's (#PCDATA)*...
      * field
      * misc
      * s_inf
      * lsource
      * dial
      * gloss
        * pri (UNUSED)
      * example

Description of JMdict SQLite database design
============================================

Basically, the table structure of the SQLite 3 database follows this
very closely.  Here's generally how it's designed:

1. All tables (except entry) have two integer keys, always named id
   and fk.  id is an auto-increment value, while fk is used for
   joining tables.  (fk is of course indexed.)

2. Any data with a one-to-one relationship with a parent XML node was
   moved into a column for the parent node's table.  (Example: <keb>
   is now k_ele.value.)

3. Attributes have similar but usually different names than in JMdict.
   Generally speaking, if an attribute had a prefix (like xml:lang or
   ls_wasei), it is stored without it (as lang or wasei).

4. Info nodes have a one-to-one relationship with entries, so there's
   no table for them.  XML children of the <info> element are linked
   directly to the entry table rather than to a meaningless
   intermediate table.

Examples
========

Example 1: 魚 (entry 1578010)

::

  SELECT * FROM entry
  LEFT JOIN k_ele ON entry.id = k_ele.fk
  LEFT JOIN ke_inf ON k_ele.id = ke_inf.fk
  LEFT JOIN ke_pri ON k_ele.id = ke_pri.fk
  WHERE entry.ent_seq = 1578010

Result:

========  =============  ========  ========  ===========  =========  =========  =============  =========  =========  ============
entry.id  entry.ent_seq  k_ele.id  k_ele.fk  k_ele.value  ke_inf.id  ke_inf.fk  ke_inf.entity  ke_pri.id  ke_pri.fk  ke_pri.value
========  =============  ========  ========  ===========  =========  =========  =============  =========  =========  ============
55777     1578010        48939     55777     u'\u9b5a'    None       None       None           42889      48939      u'ichi1'
55777     1578010        48939     55777     u'\u9b5a'    None       None       None           42890      48939      u'ichi2'
55777     1578010        48939     55777     u'\u9b5a'    None       None       None           42891      48939      u'news1'
55777     1578010        48939     55777     u'\u9b5a'    None       None       None           42892      48939      u'nf03'
========  =============  ========  ========  ===========  =========  =========  =============  =========  =========  ============<commit_msg>Put in proper unicode strings in rst table.<commit_after>==================
 JMdict DB Schema
==================

Original JMdict XML format (summarized)
=======================================

::

  Entry
    - ent_seq
    * k_ele
      - keb
      * ke_inf
      * ke_pri
    + r_ele
      - reb
      ? re_nokanji
      * re_restr
      * re_inf
      * re_pri
    ? info
      * links
        - link_tag
        - link_desc
        - link_uri
      * bibl
        ? bib_tag
        ? bib_txt
      * etym (UNUSED)
      * audit
        - upd_date
        - upd_detl
    + sense
      * stagk
      * stagr
      * pos
      * xref*  # not sure why it's (#PCDATA)*...
      * ant*   # not sure why it's (#PCDATA)*...
      * field
      * misc
      * s_inf
      * lsource
      * dial
      * gloss
        * pri (UNUSED)
      * example

Description of JMdict SQLite database design
============================================

Basically, the table structure of the SQLite 3 database follows this
very closely.  Here's generally how it's designed:

1. All tables (except entry) have two integer keys, always named id
   and fk.  id is an auto-increment value, while fk is used for
   joining tables.  (fk is of course indexed.)

2. Any data with a one-to-one relationship with a parent XML node was
   moved into a column for the parent node's table.  (Example: <keb>
   is now k_ele.value.)

3. Attributes have similar but usually different names than in JMdict.
   Generally speaking, if an attribute had a prefix (like xml:lang or
   ls_wasei), it is stored without it (as lang or wasei).

4. Info nodes have a one-to-one relationship with entries, so there's
   no table for them.  XML children of the <info> element are linked
   directly to the entry table rather than to a meaningless
   intermediate table.

Examples
========

Example 1: 魚 (entry 1578010)

::

  SELECT * FROM entry
  LEFT JOIN k_ele ON entry.id = k_ele.fk
  LEFT JOIN ke_inf ON k_ele.id = ke_inf.fk
  LEFT JOIN ke_pri ON k_ele.id = ke_pri.fk
  WHERE entry.ent_seq = 1578010

Result:

========  =============  ========  ========  ===========  =========  =========  =============  =========  =========  ============
entry.id  entry.ent_seq  k_ele.id  k_ele.fk  k_ele.value  ke_inf.id  ke_inf.fk  ke_inf.entity  ke_pri.id  ke_pri.fk  ke_pri.value
========  =============  ========  ========  ===========  =========  =========  =============  =========  =========  ============
55777     1578010        48939     55777     魚           None       None       None           42889      48939      ichi1
55777     1578010        48939     55777     魚           None       None       None           42890      48939      ichi2
55777     1578010        48939     55777     魚           None       None       None           42891      48939      news1
55777     1578010        48939     55777     魚           None       None       None           42892      48939      nf03
========  =============  ========  ========  ===========  =========  =========  =============  =========  =========  ============
{'fraction_non_alphanumeric': 0.1984023475709162, 'fraction_numerical': 0.051190088033909356, 'mean_word_length': 2.08136614766449, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1507844', 'n_tokens_mistral': 2165, 'n_tokens_neox': 1826, 'n_words': 595}