<commit_before>[Back to README](../README.md)
# Script Setup
We have a script to automate the process of setting up your developmental environment. Right now, it supports Debian-based systems(Debian, Ubuntu etc.), Fedora and MacOS.

**Warning**: If you run this python script on Windows system, it will run a still in Development script, which is **not** tested fully and might **not** work properly.

**Note**: We have a batch file for Windows which is still a **Work-in-progress**. If you are windows user, We would love your contribution in testing and developing the script for Windows platform. You can find the batch file under setup directory.

## Prerequisite
There are some basic requirements for the script to work:
- git(to clone the repository)
- python 3
- ruby-2.5.0
- apt(debian)/homebrew(MacOS)

## Instructions
- Clone the repository
- From the repository directory, run `python3 setup.py`
  - The python file checks for your operating system and runs the corresponding system dependent script
- The script will ask for your root password and MySQL passwords.
  - While installing MySQL, the MySQL installer might ask you to setup the root password
  - In case of MacOS systems, the MySQL root password will be blank if the installer doesn't ask you to setup one.
- Wait for the installation to complete
- If you face any errors, you can find the log for the script in setup directory by the name of log.txt

In case of any errors please post your error logs on: https://github.com/WikiEducationFoundation/WikiEduDashboard/issues/1709.
You can also contact us on slack for any further queries.

## Troubleshooting
- If you want to setup your own manual Database config(Advanced users)
  - First, create your manual config file, `config/database.yml` from the sample file provided, `config/database.example.yml`
  - Run the script
  - Run Migrations if needed.
- If you face issues related to MySQL default password on your system
  - Please confirm your Password for MySQL
  - Delete `config/Database.yml`
  - Run the script again.
- If you face the error that `Sorry! Your operating is not supported by this script`
  - You can try running the system dependent scripts from setup directory, according to your system
  - You can try manual installation

# Manual Setup
## TL;DR bare minimum version
If you know your way around Rails, here's the very short version. Some additional requirements are necessary to make all the tests pass and all the features work, but this should be enough to stand up the app quickly.

* copy `config/application.example.yml` to `config/application.yml`
* copy `config/database.example.yml` to `config/database.yml`
* create a MySQL database, `dashboard`
* install ruby 2.5.0 and nodejs
* `bundle install`
* `rake db:migrate`
* install yarn
* `yarn` for more javascript requirements
* `gulp` to build assets
* `guard` or `rails s` to start a server
* localhost:3000 should load the home page

## Project Setup

- Pre-requisites for setup on OSX (Mac)
    - You will need to have xcode installed in order to have `git` on your machine.  If you run `git` and it is not there, you will be prompted to install xcode.
    - To install rvm, you'll first need a gpg utility. You can install the GPG Suite from gpgtools.org
    - Homebrew will install itself when you run the rvm install command, if you don't have it already.

- Pre-requisites for setup on Windows:
    - Install Git from [the official Windows package](https://git-scm.com/download/win)

- Fork this repo, so that you can make changes and push them freely to GitHub.
- Clone the new WikiEduDashboard repo and enter that directory.
- On OSX/Debian, make sure you are in the "sudo" group.
- Install Ruby 2.5.0 (RVM is documented here; rbenv also works fine.)
    - OSX/Debian:
       - From the WikiEduDashboard directory, run the curl script from [rvm.io](https://rvm.io/)
       - `rvm install ruby-2.5.0`
    - Windows:
       - Use [RailsInstaller](http://railsinstaller.org/en)
       - Install [Ruby DevKit](https://github.com/oneclick/rubyinstaller/wiki/Development-Kit)
- Install Node:
  - Debian: `sudo apt install nodejs`
  - OSX: `brew install node` (this assumes you are using [homebrew](brew.sh))
  - Windows: [Download the installer](https://nodejs.org/)

- Install Gems:
    - $ `gem install bundler`
    - $ `bundle install`
    - If some gems fail to install, you may need to install some dependencies, such as: `libmysqlclient-dev libpq-dev libqtwebkit-dev`

- [Install Yarn](https://yarnpkg.com/lang/en/docs/install/)
- Install node modules via Yarn:
    - $ `yarn`

- Install PhantomJS:
    - $ `sudo yarn global add phantomjs-prebuilt`

- Install Pandoc
    - See the Pandoc [installation guide](http://pandoc.org/installing.html) for your environment's specifics.
    - Only Pandoc itself is needed; no additional related components (eg, LaTeX) are required.

- Add config files:
    - Save `application.example.yml` and `database.example.yml` as `application.yml` and `database.yml`, respectively, in the `config` directory. The default settings in `database.yml` will suffice for a development environment.

- Create mysql development and test database:
    - Install mariadb-server (or mysql-server)
        - Debian: `sudo apt-get install -y mariadb-server`
        - OSX: `brew install mariadb`
        - Windows: Install [XAMPP](https://www.apachefriends.org/index.html)
    - Start a mysql command line:
        - Debian: `sudo mysql`
        - OSX: `brew services start mariadb` then `sudo mysql`
        - Windows: `C:\xampp\mysql\bin\mysql -u root`
    - `CREATE DATABASE dashboard DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;`
    - `CREATE DATABASE dashboard_testing DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;`
    - `exit`

- Install Redis:
  - Debian: `sudo apt install redis-server`
  - OSX: `brew install redis`
  - Windows: Download [the Windows port](https://github.com/MSOpenTech/redis/releases) by the Microsoft Open Tech Group

- Install Gulp (if not already installed)
  - `sudo yarn global add gulp`

## Initialize
1. **Migrate the development and test databases**
  - $ `rake db:migrate`
  - $ `rake db:migrate RAILS_ENV=test`

## [Set up OAuth integration](oauth.md) (optional — skip unless you are working on WikiEdits features)

## Populate example data

Running these tasks will take several minutes, and should populate your database
with a few example events with editing activity.

1. **Create courses with users**
  - $ `rake dev:populate`

2. **Import revision and upload data**
  - $ `rake batch:update_constantly`
  - $ `rake batch:update_daily`
<commit_msg>Add a link on how to populate course with Data<commit_after>[Back to README](../README.md)
# Script Setup
We have a script to automate the process of setting up your developmental environment. Right now, it supports Debian-based systems(Debian, Ubuntu etc.), Fedora and MacOS.

**Warning**: If you run this python script on Windows system, it will run a still in Development script, which is **not** tested fully and might **not** work properly.

**Note**: We have a batch file for Windows which is still a **Work-in-progress**. If you are windows user, We would love your contribution in testing and developing the script for Windows platform. You can find the batch file under setup directory.

## Prerequisite
There are some basic requirements for the script to work:
- git(to clone the repository)
- python 3
- ruby-2.5.0
- apt(debian)/homebrew(MacOS)

## Instructions
- Clone the repository
- From the repository directory, run `python3 setup.py`
  - The python file checks for your operating system and runs the corresponding system dependent script
- The script will ask for your root password and MySQL passwords.
  - While installing MySQL, the MySQL installer might ask you to setup the root password
  - In case of MacOS systems, the MySQL root password will be blank if the installer doesn't ask you to setup one.
- Wait for the installation to complete
- If you face any errors, you can find the log for the script in setup directory by the name of log.txt

In case of any errors please post your error logs on: https://github.com/WikiEducationFoundation/WikiEduDashboard/issues/1709.
You can also contact us on slack for any further queries.

## Troubleshooting
- If you want to setup your own manual Database config(Advanced users)
  - First, create your manual config file, `config/database.yml` from the sample file provided, `config/database.example.yml`
  - Run the script
  - Run Migrations if needed.
- If you face issues related to MySQL default password on your system
  - Please confirm your Password for MySQL
  - Delete `config/Database.yml`
  - Run the script again.
- If you face the error that `Sorry! Your operating is not supported by this script`
  - You can try running the system dependent scripts from setup directory, according to your system
  - You can try manual installation

# Manual Setup
## TL;DR bare minimum version
If you know your way around Rails, here's the very short version. Some additional requirements are necessary to make all the tests pass and all the features work, but this should be enough to stand up the app quickly.

* copy `config/application.example.yml` to `config/application.yml`
* copy `config/database.example.yml` to `config/database.yml`
* create a MySQL database, `dashboard`
* install ruby 2.5.0 and nodejs
* `bundle install`
* `rake db:migrate`
* install yarn
* `yarn` for more javascript requirements
* `gulp` to build assets
* `guard` or `rails s` to start a server
* localhost:3000 should load the home page

## Project Setup

- Pre-requisites for setup on OSX (Mac)
    - You will need to have xcode installed in order to have `git` on your machine.  If you run `git` and it is not there, you will be prompted to install xcode.
    - To install rvm, you'll first need a gpg utility. You can install the GPG Suite from gpgtools.org
    - Homebrew will install itself when you run the rvm install command, if you don't have it already.

- Pre-requisites for setup on Windows:
    - Install Git from [the official Windows package](https://git-scm.com/download/win)

- Fork this repo, so that you can make changes and push them freely to GitHub.
- Clone the new WikiEduDashboard repo and enter that directory.
- On OSX/Debian, make sure you are in the "sudo" group.
- Install Ruby 2.5.0 (RVM is documented here; rbenv also works fine.)
    - OSX/Debian:
       - From the WikiEduDashboard directory, run the curl script from [rvm.io](https://rvm.io/)
       - `rvm install ruby-2.5.0`
    - Windows:
       - Use [RailsInstaller](http://railsinstaller.org/en)
       - Install [Ruby DevKit](https://github.com/oneclick/rubyinstaller/wiki/Development-Kit)
- Install Node:
  - Debian: `sudo apt install nodejs`
  - OSX: `brew install node` (this assumes you are using [homebrew](brew.sh))
  - Windows: [Download the installer](https://nodejs.org/)

- Install Gems:
    - $ `gem install bundler`
    - $ `bundle install`
    - If some gems fail to install, you may need to install some dependencies, such as: `libmysqlclient-dev libpq-dev libqtwebkit-dev`

- [Install Yarn](https://yarnpkg.com/lang/en/docs/install/)
- Install node modules via Yarn:
    - $ `yarn`

- Install PhantomJS:
    - $ `sudo yarn global add phantomjs-prebuilt`

- Install Pandoc
    - See the Pandoc [installation guide](http://pandoc.org/installing.html) for your environment's specifics.
    - Only Pandoc itself is needed; no additional related components (eg, LaTeX) are required.

- Add config files:
    - Save `application.example.yml` and `database.example.yml` as `application.yml` and `database.yml`, respectively, in the `config` directory. The default settings in `database.yml` will suffice for a development environment.

- Create mysql development and test database:
    - Install mariadb-server (or mysql-server)
        - Debian: `sudo apt-get install -y mariadb-server`
        - OSX: `brew install mariadb`
        - Windows: Install [XAMPP](https://www.apachefriends.org/index.html)
    - Start a mysql command line:
        - Debian: `sudo mysql`
        - OSX: `brew services start mariadb` then `sudo mysql`
        - Windows: `C:\xampp\mysql\bin\mysql -u root`
    - `CREATE DATABASE dashboard DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;`
    - `CREATE DATABASE dashboard_testing DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;`
    - `exit`

- Install Redis:
  - Debian: `sudo apt install redis-server`
  - OSX: `brew install redis`
  - Windows: Download [the Windows port](https://github.com/MSOpenTech/redis/releases) by the Microsoft Open Tech Group

- Install Gulp (if not already installed)
  - `sudo yarn global add gulp`

## Initialize
1. **Migrate the development and test databases**
  - $ `rake db:migrate`
  - $ `rake db:migrate RAILS_ENV=test`

## [Set up OAuth integration](oauth.md) (optional — skip unless you are working on WikiEdits features)

## Populate example data

Running these tasks will take several minutes, and should populate your database
with a few example events with editing activity.

1. **Create courses with users**
  - $ `rake dev:populate`
    - To populate your course with data, check [Populating a course with data.](https://github.com/WikiEducationFoundation/WikiEduDashboard/blob/master/docs/user_roles.md#populating-a-course-with-data)
  

2. **Import revision and upload data**
  - $ `rake batch:update_constantly`
  - $ `rake batch:update_daily`

{'fraction_non_alphanumeric': 0.0857079580688026, 'fraction_numerical': 0.0042817067769083126, 'mean_word_length': 3.694040194040194, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 17, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7505496', 'n_tokens_mistral': 4002, 'n_tokens_neox': 3791, 'n_words': 1719}