<commit_before>tenhou-log
==========

Scripts for downloading logs from tenhou.net

tenhou-download-game-xml.py
---------------------------

Scans flash local storage for tenhou log file names and then downloads the game xml from tenhou.net.

Log Format
==========

```
<SHUFFLE>
 - seed         Seed for RNG for generating walls and dice rolls.
 - ref          ?
<GO>            Start of game
 - type             Lobby type.
 - lobby            Lobby number.
<UN>            User list or user reconnect
 - n[0-3]           Names for each player as URLEncoded UTF-8.
 - dan              List of ranks for each player.
 - rate             List of rates for each player.
 - sx               List of sex ("M" or "F") for each player.
<BYE>           User disconnect
 - who              Player who disconnected.
<TAIKYOKU>      Start of round
 - oya              Dealer
<INIT>          Start of hand
 - seed             Six element list:
                        Round number,
                        Number of combo sticks,
                        Number of riichi sticks,
                        ?,
                        ?,
                        Dora indicator.
 - ten              List of scores for each player     
 - oya              Dealer
 - hai[0-3]         Starting hands as a list of tiles for each player.
<[T-W][0-9]*>   Player draws a tile.
<[D-G][0-9]*>   Player discards a tile.
<N>             Player calls a tile.
 - who              The player who called the tile.
 - m                The meld.
<REACH>         Player declares riichi.
 - who              The player who declared riichi
 - step             Where the player is in declaring riichi:
                        1 -> Called "riichi"
                        2 -> Placed point stick on table after discarding.<commit_msg>Add documentation for dice roll in INIT tag<commit_after>tenhou-log
==========

Scripts for downloading logs from tenhou.net

tenhou-download-game-xml.py
---------------------------

Scans flash local storage for tenhou log file names and then downloads the game xml from tenhou.net.

Log Format
==========

```
<SHUFFLE>
 - seed         Seed for RNG for generating walls and dice rolls.
 - ref          ?
<GO>            Start of game
 - type             Lobby type.
 - lobby            Lobby number.
<UN>            User list or user reconnect
 - n[0-3]           Names for each player as URLEncoded UTF-8.
 - dan              List of ranks for each player.
 - rate             List of rates for each player.
 - sx               List of sex ("M" or "F") for each player.
<BYE>           User disconnect
 - who              Player who disconnected.
<TAIKYOKU>      Start of round
 - oya              Dealer
<INIT>          Start of hand
 - seed             Six element list:
                        Round number,
                        Number of combo sticks,
                        Number of riichi sticks,
                        First dice minus one,
                        Second dice minus one,
                        Dora indicator.
 - ten              List of scores for each player     
 - oya              Dealer
 - hai[0-3]         Starting hands as a list of tiles for each player.
<[T-W][0-9]*>   Player draws a tile.
<[D-G][0-9]*>   Player discards a tile.
<N>             Player calls a tile.
 - who              The player who called the tile.
 - m                The meld.
<REACH>         Player declares riichi.
 - who              The player who declared riichi
 - step             Where the player is in declaring riichi:
                        1 -> Called "riichi"
                        2 -> Placed point stick on table after discarding.
{'fraction_non_alphanumeric': 0.086315214422289, 'fraction_numerical': 0.006009287080032778, 'mean_word_length': 1.2604938271604937, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4504355', 'n_tokens_mistral': 956, 'n_tokens_neox': 905, 'n_words': 402}