<commit_before># Phase 1 to Phase 2 Migration Scripts

This directory contains the scripts that need to be run in order to migrate from Phase 1 of DOAJ to Phase 2 of DOAJ

They should be run individually in the following order:


1. userroles.py - adds the "publisher" role to user accounts where appropriate.  Should be run a couple of times, since for some reason it doesn't always get everyone.  Weird.
2. journalowners.py - adds the "owner" field to the journals for the relevant user accounts.  As with (1) needs to be run a couple of times.  Debug if time permits.
3. /scripts/journalinfo.py - this script is in the standard doaj scripts folder.  Running it will correct any issues with the in_doaj flag on articles.  Check the size of the article index before and after to see the effect it has on the amount of content.
4. flushuploads.py - remove all the old uploaded file records in the index, and set it up for its new usage
5. uploadcorrections.py - builds the corrections table for the next script (uploadedxml.py).  You will need to modify the file to point to the relevant file-paths
6. uploadedxml.py - migrates all the new articles from uploaded files into the database.  You will need to modify the file to point to the directory of the XML files, and the script itself could take upwards of 10 minutes to run. 
7. Do this yourself: move the upload directory out of the way and make a new empty one (we should keep the old upload directory in case we need the content again)
8. Set up the cron jobs for ingesting articles

The following files are scripts generated in the development of the migrate scripts and should NOT be run as part of the migration:

1. uploadedfilenames.py - investigates uploaded files for the filenames and publishers to see if there are any obvious duplicates
<commit_msg>add extra step to migrate<commit_after># Phase 1 to Phase 2 Migration Scripts

This directory contains the scripts that need to be run in order to migrate from Phase 1 of DOAJ to Phase 2 of DOAJ

They should be run individually in the following order:


1. userroles.py - adds the "publisher" role to user accounts where appropriate.  Should be run a couple of times, since for some reason it doesn't always get everyone.  Weird.
2. journalowners.py - adds the "owner" field to the journals for the relevant user accounts.  As with (1) needs to be run a couple of times.  Debug if time permits.
3. /scripts/journalinfo.py - this script is in the standard doaj scripts folder.  Running it will correct any issues with the in_doaj flag on articles.  Check the size of the article index before and after to see the effect it has on the amount of content.
4. flushuploads.py - remove all the old uploaded file records in the index, and set it up for its new usage
5. uploadcorrections.py - builds the corrections table for the next script (uploadedxml.py).  You will need to modify the file to point to the relevant file-paths
6. uploadedxml.py - migrates all the new articles from uploaded files into the database.  You will need to modify the file to point to the directory of the XML files, and the script itself could take upwards of 10 minutes to run. 
7. Do this yourself: move the upload directory out of the way and make a new empty one (we should keep the old upload directory in case we need the content again)
8. /scripts/journalcsv.py - generate the initial csv file
9. Set up the cron jobs for ingesting articles

The following files are scripts generated in the development of the migrate scripts and should NOT be run as part of the migration:

1. uploadedfilenames.py - investigates uploaded files for the filenames and publishers to see if there are any obvious duplicates

{'fraction_non_alphanumeric': 0.0337655321447866, 'fraction_numerical': 0.008914100486223663, 'mean_word_length': 3.8852242744063323, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7036764', 'n_tokens_mistral': 951, 'n_tokens_neox': 892, 'n_words': 629}