<commit_before>BatchUploadTools [![Build Status](https://travis-ci.org/lokal-profil/BatchUploadTools.svg?branch=master)](https://travis-ci.org/lokal-profil/BatchUploadTools)
=======

An attempt of unifying the various tools for batch uploads under one repo.

Heavily based on the [LSH](https://github.com/lokal-profil/LSH) repo


## Running as a different user:

To run as a different user to your standard pywikibot simply place a
modified `user-config.py`-file in the top directory.

To use a different user for a particular batchupload place the `user-config.py`
in the subdirectory and run the script with `-dir:<sub-driectory>`.
Remember to set `password_file = "<sub-driectory>/secretPasswords"` in the
`user-config`.

## Protocol for a batch upload
1. Load indata to a dictionary
2. Process the indata to generate mapping lists
3. Load the indata and the mappings to produce a list of original filenames
   (of media files) and their final filenames as well as json holding the
   following for each file:
    - Maintanance categories
    - Content categories
    - File description
    - Output filename
    - Input filename (as key)
4. Run the prep-uploader to rename the media files and create the text file
   for the associated file description page.
5. Run the upoader to upload it all

## When creating a new batch upload<commit_msg>Patch to pywikibot made that unnecessary<commit_after>BatchUploadTools [![Build Status](https://travis-ci.org/lokal-profil/BatchUploadTools.svg?branch=master)](https://travis-ci.org/lokal-profil/BatchUploadTools)
=======

An attempt of unifying the various tools for batch uploads under one repo.

Heavily based on the [LSH](https://github.com/lokal-profil/LSH) repo


## Running as a different user:

To run as a different user to your standard pywikibot simply place a
modified `user-config.py`-file in the top directory.

To use a different user for a particular batchupload place the `user-config.py`
in the subdirectory and run the script with `-dir:<sub-directory>`.

## Protocol for a batch upload
1. Load indata to a dictionary
2. Process the indata to generate mapping lists
3. Load the indata and the mappings to produce a list of original filenames
   (of media files) and their final filenames as well as json holding the
   following for each file:
    - Maintenance categories
    - Content categories
    - File description
    - Output filename
    - Input filename (as key)
4. Run the prep-uploader to rename the media files and create the text file
   for the associated file description page.
5. Run the uploader to upload it all

## When creating a new batch upload
{'fraction_non_alphanumeric': 0.07484802431610943, 'fraction_numerical': 0.003799392097264438, 'mean_word_length': 4.053742802303263, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '845610', 'n_tokens_mistral': 741, 'n_tokens_neox': 738, 'n_words': 338}