<commit_before># WishBuilder

## Prepping Data for Geney

Fortunately, many valuable biology-related datasets are publicly available on the Web. Commonly, a researcher wants to filter a dataset to include biological samples that meet specific criteria and perhaps to examine data for a few specific entities (genes, proteins, etc.). Unfortunately, researchers often have a difficult time working with these datasets due to their large size or because they are stored in a wide variety of formats. Accordingly, computational skills are required to _extract_ such data and _transform_ them into a format that can be _loaded_ easily into analytical tools. However, many biologists lack the expertise to _extract_, _transform_, and _load_ (ETL) such data.

To address this problem, the Piccolo lab is developing _Geney_, a Web-based tool that will enable researchers to query such datasets in a consistent and easy manner. In addition to Geney, we are developing _WishBuilder_ (name subject to change), a system that enables datasets to be imported into Geney, irrespective of the format in which the data were originally stored. This system downloads data from public Web servers, reformats the data, and stores it in a consistent, queryable format. To facilitate this process, we are asking for help from students to write scripts to prepare such datasets. The more datasets we prepare, the more useful Geney will be!

## Getting Started

Please complete the following steps to get started.

1. You will need an active account on the BYU Supercomputer. If you don't currently have an active account, please go to https://marylou.byu.edu and click on "Request an Account." Read the information on that page. Then request an account, listing me as a mentor. When it asks what type of work you plan to perform, explain this research project briefly. Indicate that you will execute custom Python and/or R scripts, that you expect to execute only single-core jobs, and that these jobs will typically require 1-4 GB of memory per job.
2. If you haven't already done so, create a [GitHub](https://github.com/) account.
3. Send an email to me with your GitHub user ID and request to be added as a contributor to [https://github.com/srp33/WishBuilder](https://github.com/srp33/WishBuilder).

## Preparing a Dataset

Please complete the following steps for each dataset that you prepare. Let me know if you have any questions or run into any problems.

1. Examine the list of [open issues](https://github.com/srp33/WishBuilder/issues). Each "issue" represents a dataset that needs to be prepared. Identify one issue that you would like to work on (and that nobody else is currently working on).
2. Edit the issue you selected. Put your name where it says "Status." Put **IN PROGRESS** next to your name.
3. Log in to the Supercomputer. Then clone the WishBuilder git repository:

  ```
  git clone https://github.com/srp33/WishBuilder.git
  cd WishBuilder
  ```

4. Or if you previously cloned the WishBuilder git repository, make sure it is up to date:

  ```
  git pull origin master
  ```

5. Create a new branch on your copy of the git repository (see below). Replace `<new-branch-name>` with the ID of the dataset you are working with.

  ```
  git checkout -b <new-branch-name>
  ```

6. Create a new directory within your branched repository; the name of this directory should also be the ID of the dataset you are working with.
7. Now `cd` into the new directory.
8. Write a bash script called `download.sh` that downloads the data file(s) from the source location to the current directory. You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
9. Create a file called `.gitignore`. Within that file, list the name(s) of the file(s) that were downloaded. You can learn more about `.gitignore` files [here](https://help.github.com/articles/ignoring-files/).
10. Open the data file(s) in a text editor and examine them to understand how they are structured. (If the data file is too large for a text editor, use commands such as head, tail, and less to examine the file.)
11. Using a text editor, create a file called `testdata.tsv`. [Below](#test-files) you can learn about the purpose of this file and how it should be structured. You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
12. Write a bash script called `parse.sh`. This script should parse the data file(s) and reformat the data (as needed) into the output format described [below](#output-file-format). In most cases, `parse.sh` will invoke other script(s) written in Python or R. The name of the output file *must* be `data.tsv.gz`. _Recommendation: work with a smaller version of the data file(s) initially so it is easier to test._ You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
13. Write a bash script called `install.sh` that installs any software that are necessary to execute `parse.sh`. If no extra software must be installed, it can be blank. You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
14. Compare `data.tsv.gz` and `testdata.tsv`. Make sure the data values were parsed correctly.
15. Add a line to `.gitignore` for `data.tsv.gz`.
16. Add, commit, and push your changes to the GitHub branch that you created earlier. Replace `<message>` with a brief messages that describes the work you have done. Replace `<new-branch-name>` with the name of the branch you created previously.

  ```
  git add --all
  git commit -m "<message>"
  git push origin <new-branch-name>
  ```

17. Go [here](https://github.com/srp33/WishBuilder/compare?expand=1) to create a GitHub pull request. Put "master" as the base branch and your new branch as the compare branch. Click on "Create pull request".
18. Modify the [issue](https://github.com/srp33/WishBuilder/issues) for this dataset. Where it says, "Status," put **READY FOR TESTING** next to your name.
19. Send me an email indicating that your dataset is ready for testing.

## Notes

- Python 3.5 is installed on the Supercomputer; use `module load python/3/5`.
- R is also installed on the Supercomputer; use `module load r/3/3`.<commit_msg>Added steps to upload SSH public key<commit_after># WishBuilder

## Prepping Data for Geney

Fortunately, thousands of biology-related datasets are publicly available on the Web. Commonly, a researcher wants to filter a dataset to include biological samples that meet specific criteria and perhaps to examine data for a few specific variables (genes, proteins, etc.). Unfortunately, researchers often have a difficult time working with these datasets due to their large size or because they are stored in a wide variety of formats. Accordingly, computational skills are required to _extract_ such data and _transform_ them into a format that can be _loaded_ easily into analytical tools. However, many biologists lack the expertise to _extract_, _transform_, and _load_ (ETL) such data.

To address this problem, the Piccolo lab is developing _Geney_, a Web-based tool that will enable researchers to query such datasets in a consistent and easy manner. In addition to Geney, we are developing _WishBuilder_, a system that enables datasets to be imported into Geney, irrespective of the format in which the data were originally stored. This system downloads data from public Web servers, reformats the data, and stores it in a consistent, queryable format. To facilitate this process, we are asking for help from students to write scripts to prepare such datasets. The more datasets we prepare, the more useful Geney will be!

## Getting Started

Please complete the following steps to get started.

1. You will need an active account on the BYU Supercomputer. If you don't currently have an active account, please go to https://marylou.byu.edu and click on "Request an Account." Read the information on that page. Then request an account, listing me as a mentor. When it asks what type of work you plan to perform, explain this research project briefly. Indicate that you will execute custom Python and/or R scripts, that you expect to execute only single-core jobs, and that these jobs will typically require 1-4 GB of memory per job.
2. If you haven't already done so, create a [GitHub](https://github.com) account.
3. Send an email to me with your GitHub user ID and request to be added as a contributor to this repository.

## Preparing a Dataset

Please complete the following steps for each dataset that you prepare. Let me know if you have any questions or run into any problems.

1. Examine the list of [open issues](https://github.com/srp33/WishBuilder/issues). Each "issue" represents a dataset that needs to be prepared. Identify one issue that you would like to work on (and that nobody else is currently working on).
2. Send an email to me indicating which issue you would like to work on.
3. Log in to the Supercomputer. At the command line, enter the following commands (but substitute your actual email address where it says `<EMAIL>`):

  ```
  ssh-keygen -t rsa -b 4096 -C "<EMAIL>"
  ```

4. When it asks you to "Enter a file in which to save the key," press Enter. This uses the default file location.

5. When it asks you to enter a passphrase, press Enter (twice).

6. Now there should be a file at ~/.ssh/id_rsa.pub. Enter the following command to display the contents of this file. Then copy this output to your clipboard. This is your public key and enables you to connect from Linux to GitHub without a password.

  ```
  cat ~/.ssh/id_rsa.pub
  ```
  
7. Go to https://github.com/settings/keys. This should display the SSH keys that are currently set for your GitHub account. Click on "New SSH key", enter a Title (maybe "FSL"), paste the public key from your clipboard, and click on "Add SSH key."

8. Clone the WishBuilder git repository:

  ```
  git clone https://github.com/srp33/WishBuilder.git
  cd WishBuilder
  ```

9. Or if you previously cloned the WishBuilder git repository, make sure it is up to date:

  ```
  git pull origin master
  ```

10. Create a new branch on your copy of the git repository (see below). Replace `<new-branch-name>` with the ID of the dataset you are working with.

  ```
  git checkout -b <new-branch-name>
  ```

11. Create a new directory within your branched repository; the name of this directory should also be the ID of the dataset you are working with.
12. Now `cd` into the new directory.
13. Write a bash script called `download.sh` that downloads the data file(s) from the source location to the current directory. You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
14. Create a file called `.gitignore`. Within that file, list the name(s) of the file(s) that were downloaded. You can learn more about `.gitignore` files [here](https://help.github.com/articles/ignoring-files/).
15. Open the data file(s) in a text editor and examine them to understand how they are structured. (If the data file is too large for a text editor, use commands such as head, tail, and less to examine the file.)
16. Using a text editor, create a file called `testdata.tsv`. [Below](#test-files) you can learn about the purpose of this file and how it should be structured. You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
17. Write a bash script called `parse.sh`. This script should parse the data file(s) and reformat the data (as needed) into the output format described [below](#output-file-format). In most cases, `parse.sh` will invoke other script(s) written in Python or R. The name of the output file *must* be `data.tsv.gz`. _Recommendation: work with a smaller version of the data file(s) initially so it is easier to test._ You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
18. Write a bash script called `install.sh` that installs any software that are necessary to execute `parse.sh`. If no extra software must be installed, it can be blank. You can see an example [here](https://github.com/srp33/WishBuilder/tree/master/ICGC_Donor_Clinical).
19. Compare `data.tsv.gz` and `testdata.tsv`. Make sure the data values were parsed correctly.
20. Add a line to `.gitignore` for `data.tsv.gz`.
21. Add, commit, and push your changes to the GitHub branch that you created earlier. Replace `<message>` with a brief messages that describes the work you have done. Replace `<new-branch-name>` with the name of the branch you created previously.

  ```
  git add --all
  git commit -m "<message>"
  git push origin <new-branch-name>
  ```

22. Go [here](https://github.com/srp33/WishBuilder/compare?expand=1) to create a GitHub pull request. Put "master" as the base branch and your new branch as the compare branch. Click on "Create pull request".
23. Modify the [issue](https://github.com/srp33/WishBuilder/issues) for this dataset. Where it says, "Status," put **READY FOR TESTING** next to your name.
24. Send me an email indicating that your dataset is ready for testing.

## Notes

- Python 3.5 is installed on the Supercomputer; use `module load python/3/5`.
- R is also installed on the Supercomputer; use `module load r/3/3`.
{'fraction_non_alphanumeric': 0.07300202992256222, 'fraction_numerical': 0.009924065859709796, 'mean_word_length': 4.1678321678321675, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 25, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5226829', 'n_tokens_mistral': 3629, 'n_tokens_neox': 3444, 'n_words': 1980}