<commit_before># Erben

A web-based tool for creating full e-books from single pages digitized by Czech national Library.

## How it should work

First, this system will import the entire catalog of books from digitized archives of Czech National Library. The CzNL metadata archive is available at http://kramerius.nkp.cz/kramerius/oai (see http://www.openarchives.org/OAI/openarchivesprotocol.html for protocol specification).

Next, an admin will select a batch of books to work on, the system will download their contents and convert DjVu images of digitized pages to PNG for viewing through web browser without any additional plugins. Storing the contents of all books in PNG format would require nearly 20 TB of storage space so only a handful of books will be open for work at any time. When a book is finished, PNG images will be deleted to free space for other books but the full history of text data will be kept.

Users will then edit the OCR text wiki-style to clean any mistakes in it, add basic formatting markup and vote for pages that they believe are clean and finished.

As the last step, the cleaned text of all pages in each book will be combined together and exported for advanced formatting and conversion into an actual e-book format (PDF, epub etc.).

## System requirements

- Linux/BSD
- PostgreSQL database
- PHP 5.3 or newer with the following extensions:
  - curl
  - date
  - dom
  - libxml
  - pcntl (command line only, not listed as module in phpinfo() page)
  - pcre
  - PDO
  - pdo_pgsql
  - posix
  - xml

## Installation instructions

1. Write database connection details into config/config.php.example and rename it to config/config.php
2. Run install.php from command line

## TODO<commit_msg>Add forgotten required dependency<commit_after># Erben

A web-based tool for creating full e-books from single pages digitized by Czech national Library.

## How it should work

First, this system will import the entire catalog of books from digitized archives of Czech National Library. The CzNL metadata archive is available at http://kramerius.nkp.cz/kramerius/oai (see http://www.openarchives.org/OAI/openarchivesprotocol.html for protocol specification).

Next, an admin will select a batch of books to work on, the system will download their contents and convert DjVu images of digitized pages to PNG for viewing through web browser without any additional plugins. Storing the contents of all books in PNG format would require nearly 20 TB of storage space so only a handful of books will be open for work at any time. When a book is finished, PNG images will be deleted to free space for other books but the full history of text data will be kept.

Users will then edit the OCR text wiki-style to clean any mistakes in it, add basic formatting markup and vote for pages that they believe are clean and finished.

As the last step, the cleaned text of all pages in each book will be combined together and exported for advanced formatting and conversion into an actual e-book format (PDF, epub etc.).

## System requirements

- Linux/BSD
- PostgreSQL database
- ImageMagick
- PHP 5.3 or newer with the following extensions:
  - curl
  - date
  - dom
  - libxml
  - pcntl (command line only, not listed as module in phpinfo() page)
  - pcre
  - PDO
  - pdo_pgsql
  - posix
  - xml

## Installation instructions

1. Write database connection details into config/config.php.example and rename it to config/config.php
2. Run install.php from command line

## TODO
{'fraction_non_alphanumeric': 0.04607899255867201, 'fraction_numerical': 0.0034344590726960505, 'mean_word_length': 3.8744769874476988, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7331375', 'n_tokens_mistral': 923, 'n_tokens_neox': 882, 'n_words': 521}