task_categories:text-generation size_categories:100B<n<1T language:en license:cc0-1.0 region:us
Lichess.org is a free/libre, open-source chess server powered by volunteers and donations and provides all of its content
in CC0. This script download all the games from the database and provide them in LLM pretraining friendly format.
NOTTHING YET
---
license: cc0-1.0
task_categories:
- text-generation
language:
- en
pretty_name: Lichess Games
size_categories:
- 100B<n<1T
viewer: false
---
# Dataset Card for Lichess Games

## Dataset Description

- **Homepage:** https://database.lichess.org/
- **Point of Contact:** maxime.darrin@outlook.com

### Dataset Summary

This is an easy-to-use huggingface dataset to access the [lichess game database](https://database.lichess.org/). For now it supports only the standard games 
but other variant will be added shortly.

Requirements:

```
chess
zstandard
```


### Supported Tasks and Leaderboards

It is intended for pretraining text generation models for chess games (in a PGN format).

## Dataset Structure

### Data Instances

Available configs consist on the year and month of the file as described here: https://database.lichess.org/.
For example to get a small sample one can try to download the dataset for june 2013 (~40mo). 

```python
from datasets import load_dataset

dataset = load_dataset("Icannos/lichess_games", "2013-06", streaming=True)
```

Examples (3 rows from june 2013):

<details>
  
```
{'text': '[Event "Rated Bullet game"]\n'
         '[Site "https://lichess.org/in28emmw"]\n'
         '[Date "????.??.??"]\n'
         '[Round "?"]\n'
         '[White "Kazuma"]\n'
         '[Black "kikeillana"]\n'
         '[Result "1-0"]\n'
         '[BlackElo "1684"]\n'
         '[BlackRatingDiff "-9"]\n'
         '[ECO "A07"]\n'
         '[Opening "King\'s Indian Attack: Keres Variation #2"]\n'
         '[Termination "Normal"]\n'
         '[TimeControl "60+0"]\n'
         '[UTCDate "2013.05.31"]\n'
         '[UTCTime "22:00:22"]\n'
         '[WhiteElo "1756"]\n'
         '[WhiteRatingDiff "+11"]\n'
         '\n'
         '1. Nf3 d5 2. g3 Bg4 3. Bg2 Bxf3 4. Bxf3 e6 5. O-O Bb4 6. d4 Nd7 7. '
         'c3 Ba5 8. Bf4 Bb6 9. b4 a6 10. a4 c6 11. Nd2 Ngf6 12. e4 dxe4 13. '
         'Nxe4 Nxe4 14. Bxe4 f6 15. c4 h6 16. c5 Bc7 17. Qb3 Bxf4 18. Qxe6+ '
         'Qe7 19. Bg6+ Kd8 20. Qxe7+ Kxe7 21. gxf4 Rhe8 22. Bxe8 Rxe8 23. '
         'Rfe1+ Kf7 24. Rxe8 Kxe8 25. Re1+ Kf7 26. Re4 g6 27. Kg2 f5 28. Re3 '
         'h5 29. Kf3 Kg7 30. Re7+ Kf6 31. Rxd7 g5 32. Rxb7 1-0'}
{'text': '[Event "Rated Bullet game"]\n'
         '[Site "https://lichess.org/e174t8h7"]\n'
         '[Date "????.??.??"]\n'
         '[Round "?"]\n'
         '[White "Aceves"]\n'
         '[Black "calculus"]\n'
         '[Result "0-1"]\n'
         '[BlackElo "1568"]\n'
         '[BlackRatingDiff "+9"]\n'
         '[ECO "D00"]\n'
         '[Opening "Queen\'s Pawn Game #3"]\n'
         '[Termination "Time forfeit"]\n'
         '[TimeControl "60+1"]\n'
         '[UTCDate "2013.05.31"]\n'
         '[UTCTime "22:02:13"]\n'
         '[WhiteElo "1487"]\n'
         '[WhiteRatingDiff "-9"]\n'
         '\n'
         '1. d4 d5 2. e3 Nf6 3. c3 Bg4 4. Qc2 e6 5. Bd3 Bd6 6. Nd2 c6 7. e4 '
         'dxe4 8. Nxe4 Nxe4 9. Bxe4 Bc7 10. Bxh7 g6 11. h3 Bf5 12. Qe2 Rxh7 '
         '13. Be3 Qd6 14. Nf3 Nd7 15. Ng5 Rh8 16. g3 f6 17. Bf4 e5 18. dxe5 '
         'fxe5 19. Bxe5 Qxe5 20. Qe3 Qxe3+ 21. fxe3 Bxg3+ 22. Ke2 Bh4 23. Nf3 '
         'Be4 24. Rad1 O-O-O 25. Rhf1 Rhf8 26. Nd4 Rxf1 27. Rxf1 Ne5 28. Ne6 '
         'Re8 29. Ng7 Re7 30. Rf4 Bd3+ 31. Kd2 Rxg7 32. Rxh4 Nf3+ 33. Kd1 Nxh4 '
         '34. Kd2 Bf5 0-1'}
{'text': '[Event "Rated Blitz game"]\n'
         '[Site "https://lichess.org/d4ui60z6"]\n'
         '[Date "????.??.??"]\n'
         '[Round "?"]\n'
         '[White "melro"]\n'
         '[Black "patrimpas"]\n'
         '[Result "0-1"]\n'
         '[BlackElo "1912"]\n'
         '[BlackRatingDiff "+0"]\n'
         '[ECO "B20"]\n'
         '[Opening "Sicilian Defense: Staunton-Cochrane Variation"]\n'
         '[Termination "Normal"]\n'
         '[TimeControl "240+0"]\n'
         '[UTCDate "2013.05.31"]\n'
         '[UTCTime "22:02:15"]\n'
         '[WhiteElo "1144"]\n'
         '[WhiteRatingDiff "-1"]\n'
         '\n'
         '1. e4 c5 2. c4 Nc6 3. d3 g6 4. Bd2 Bg7 5. Bc3 Nf6 6. Nd2 d6 7. Rb1 '
         'O-O 8. Bxf6 Bxf6 9. b3 Qa5 10. a4 Bc3 11. f3 e6 12. Ne2 Bg7 13. g4 '
         'd5 14. h3 Nd4 15. Nxd4 cxd4 16. Be2 dxe4 17. fxe4 Bh6 18. Rb2 e5 19. '
         'O-O Be3+ 20. Kh1 Qd8 21. Nf3 Bf4 22. Rf2 h5 23. Rg2 hxg4 24. hxg4 '
         'Kg7 25. Kg1 Rh8 26. Kf2 Qf6 27. Qc2 Rh3 28. Qd1 Be3+ 29. Ke1 Rh1+ '
         '30. Rg1 0-1'}
```

</details>


### Data Fields

Only a single column "text". Each row contains a single game in PGN format.

### How to use with python-chess

```python
from datasets import load_dataset
import chess.pgn
import io


dataset = load_dataset("lichess_games", "2013-06", streaming=True)

for d in dataset['train']:
    pgn = io.StringIO(d['text'])
    game = chess.pgn.read_game(pgn)
    print(game.headers['White'], game.headers['Black'])
    print(game.headers['Result'])
    print(game.mainline_moves())
    break
```

### Data Splits

No splits only the file per dates.

### Source Data

The underlying data are provided and maintained by the Lichess team and provided under CC0 license (https://database.lichess.org/). I only provide the huggingface interface here.
The loading script download the zstd files and reads from them on the fly without decompressing the whole file, and parses the games using python-chess.

#### Initial Data Collection and Normalization

The data comes from all the standard rated games played on lichess.org. Every rated game played on lichess and its metadata are recorded and stored by lichess.
Lichess.org provides a forever free to use, libre and open-source plateform to play chess online.

### Annotations

Some of the games (~6% according to lichess: https://database.lichess.org/) comes annotated (directly in the PGN format) with computer analysis of the move:

```
About 6% of the games include Stockfish analysis evaluations: [%eval 2.35] (235 centipawn advantage), [%eval #-4] (getting mated in 4), always from White's point of view.

The WhiteElo and BlackElo tags contain Glicko2 ratings.

Games contain clock information as PGN %clk comments since April 2017.

Variant games have a Variant tag, e.g., [Variant "Antichess"].
```

### Personal and Sensitive Information

The metadata of the different PGN contains information of the players (their pseudo on lichess), the date and times when the game happened, the strength of the players
(in terms of ELO rating) and a link to the game on the platform.


An example of metadata from one the games.

```
[Event "Rated Bullet tournament https://lichess.org/tournament/yc1WW2Ox"]
[Site "https://lichess.org/PpwPOZMq"]
[Date "2017.04.01"]
[Round "-"]
[White "Abbot"]
[Black "Costello"]
[Result "0-1"]
[UTCDate "2017.04.01"]
[UTCTime "11:32:01"]
[WhiteElo "2100"]
[BlackElo "2000"]
[WhiteRatingDiff "-4"]
[BlackRatingDiff "+1"]
[WhiteTitle "FM"]
[ECO "B30"]
[Opening "Sicilian Defense: Old Sicilian"]
[TimeControl "300+0"]
[Termination "Time forfeit"]
```


## Additional Information

### Licensing Information

Lichess provides all the data under CC0.

### Citation Information

TO COME.
