<jupyter_start><jupyter_text>Leela Chess Zero RescoringThis Colab runs the rescorer to improve the training data used for model training in leela chess zero. Rescoring makes uses of [syzygy tablebases](https://syzygy-tables.info/), which are lists of perfect endgames for all games with less than 5 (or 6 or 7) pieces remaining on the board. If any training games end up in a position that's known in the tablebase, the rescorer will rewrite the game using the known perfect play from the tablebase so that leela can lean from a perfect endgame rather than the potential mistakes in the traning game.Rescoring is used in the training of all the best nets for leela, and should always be used before running training for best results. Rescoring is CPU-only.<jupyter_code># Download and untar some training data to rescore

!mkdir -p /content/raw_training_data
%cd /content/raw_training_data

# change this to download whichever training runs you want, or load from google drive
! wget https://storage.lczero.org/files/training_data/test78/training-run3-test78-20220217-0717.tar
! wget https://storage.lczero.org/files/training_data/test78/training-run3-test78-20220216-1817.tar

# untar the files we just downloaded and delete the original tar files
! for f in *.tar; do tar -xf "$f"; rm "$f"; done<jupyter_output><empty_output><jupyter_text>Check out the rescorer from Github, which is in https://github.com/Tilps/lc0 in the `rescore_tb` branch, and build from source.<jupyter_code># install build deps
! apt install ninja-build
! pip3 install meson

# checkout the repo from Github
%cd /content
!rm -rf lc0
!git clone --recurse-submodules https://github.com/Tilps/lc0.git
%cd lc0
!git checkout rescore_tb
!git pull

# start building
! ./build.sh<jupyter_output><empty_output><jupyter_text>Download szyzygy 3-4-5 tablebases for use by the rescorer<jupyter_code>! mkdir -p /content/syzygy-3-4-5
%cd /content/syzygy-3-4-5

from bs4 import BeautifulSoup
import requests
import urllib.parse

# we'll scrape https://tablebase.lichess.ovh/tables/standard/3-4-5/ 
# and download each file listed there
TABLEBASE_HOME_URL = 'https://tablebase.lichess.ovh/tables/standard/3-4-5/'

soup = BeautifulSoup(requests.get(TABLEBASE_HOME_URL).content, "html.parser")
tablebase_links = soup.select('a[href*=.rt]')
for tablebase_link in tablebase_links:
  filename = tablebase_link.attrs['href']
  url = urllib.parse.urljoin(TABLEBASE_HOME_URL, filename)
  tablebase_file = requests.get(url)
  with open(filename, "wb") as file:
      file.write(tablebase_file.content)<jupyter_output><empty_output><jupyter_text>Next, run rescoring and save the output to `/content/rescored_training_data`.NOTE: The rescorer will delete the original data files as it runs.<jupyter_code>! mkdir -p /content/rescored_training_data/
%cd /content/raw_training_data/

# The rescorer script can't handle file globs like /content/raw_training_data/*
# So we need to do a loop in base and run it individually on each folder
! for data_dir in *; do \
  mkdir "/content/rescored_training_data/$data_dir"; \
  /content/lc0/build/release/rescorer rescore \
    --threads=2 \
    --syzygy-paths=/content/syzygy-3-4-5/ \
    --input="/content/raw_training_data/$data_dir/" \
    --output="/content/rescored_training_data/$data_dir"; \
done<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09668674698795181, 'fraction_numerical': 0.01897590361445783, 'mean_word_length': 4.775652173913043, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '384261', 'license': 'MIT', 'n_tokens_mistral': 1128, 'n_tokens_neox': 1081, 'n_words': 341, 'path': 'lc0_rescoring.ipynb', 'repo_name': 'chanind/lc0_colab_notebooks', 'size': '3320'}