<commit_before>edX Course Usage Clustering Pipeline
==============
Authors:      <NAME>, <NAME>, <NAME>, and <NAME>  
Affiliation:  Purdue University

_&lt;<https://tzwilliams.github.io/edX-clustering/>&gt;_  
_&lt;<https://github.com/tzwilliams/edX-clustering>&gt;_

Please note that this project is still under active development.  As is, the code is highly experimental and will need to be cleaned before production use.


## Required data files
This pipieline requires three edX-provided data files in the input stack.  These files will have the following generic filename structures:  

* `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql` (Courseware; contatins records of which course modeules each student has interacted)
* `{org}-{course}-{date}-auth_userprofile-prod-analytics.sql` (User profile; contains user provided demographic information)
* `{org}-{course}-{date}-course_structure-prod-analytics.json` (Course structure; contains which modules belong together and in what sequence)
    
The following table shows where in the pipeline these files are used.  (Note that each of the pipeline steps also generates data files which are used in subserquent steps of the pipeline.  These internally generated files are not included in the table.)

Pipeline step | edX file(s) required 
-------------- | ------------------ 
0) Pipeline Initiator | none | 
1) Extract Modules | `{org}-{course}-{date}-course_structure-prod-analytics.json` <br> `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql`
2) Preprocessing | `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql` 
2b) Gendered Subsets | `{org}-{course}-{date}-auth_userprofile-prod-analytics.sql`  
3) Clustering | none | 



## Environment setup
At present this pipeline is designed to work within the RStudio IDE (https://www.rstudio.com/).  You will need to either install R and RStudio or use an online version of RStudio.  Alternatively, an online version is available for free on nanoHUB.org (https://nanohub.org/resources/rstudio); however, the following instructions may need to be slightly modified if using the online version.

## Instructions for using the package
1)	Create a working folder.  

1)  Download the source code from GitHub as a zip file.  Extract the zip file into the working directory created above. 

1)	The required three edX datafiles are discussed in a previous section above.  You will need to be able to point the pipeline scripts to their location.  You may move these into your working directory if you wish (although this is not necessary).  The generic filename structures of the files you need are:  
    * `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql`
    * `{org}-{course}-{date}-auth_userprofile-prod-analytics.sql` 
    * `{org}-{course}-{date}-course_structure-prod-analytics.json`
      

1)  Open the `edX-clustering.Rproj` file found within the root directory of the gitHUB download.  This should launch RStudio (if working in the desktop).  You will see RStudio informing you of packrat packages being installed--this install may take some time the first time you open the project.

1)  Execute the `PipelineInitator.R` file either by using the RStudio Console command `source("PipelineInitiator.R")` or by opening the file in RStudio and clicking on ‘Source file...’ in the ‘Code’ menu.  This R script will sequentially call the individual R scripts which constitute the pipeline.  Follow the directions on the console (you will be asked for the location of the data files--more than once for some files).  Be aware some processing steps may require multiple minutes to complete.
<commit_msg>update to input stack table<commit_after>edX Course Usage Clustering Pipeline
==============
Authors:      <NAME>, <NAME>, <NAME>, and <NAME>  
Affiliation:  Purdue University

_&lt;<https://tzwilliams.github.io/edX-clustering/>&gt;_  
_&lt;<https://github.com/tzwilliams/edX-clustering>&gt;_

Please note that this project is still under active development.  As is, the code is highly experimental and will need to be cleaned before production use.


## Required data files
This pipieline requires three edX-provided data files in the input stack.  These files will have the following generic filename structures:  

* `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql` (Courseware; contatins records of which course modeules each student has interacted)
* `{org}-{course}-{date}-auth_userprofile-prod-analytics.sql` (User profile; contains user provided demographic information)
* `{org}-{course}-{date}-course_structure-prod-analytics.json` (Course structure; contains which modules belong together and in what sequence)
    
The following table shows where in the pipeline these files are used.  (Note that each of the pipeline steps also generates data files which are used in subserquent steps of the pipeline.  These internally generated files are not included in the table.)

| Pipeline step | edX file(s) required | Data used from file |
| -------------- | ------------------ | ------------- |
| 0) Pipeline Initiator | none |  |
| 1) Extract Modules | `{org}-{course}-{date}-course_structure-prod-analytics.json` | Course module order |
| . | `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql`  | Removing modules with no accesses events  |
| 2) Preprocessing | `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql` | Did student X accessed module Y? |
| 2b) Gendered Subsets | `{org}-{course}-{date}-auth_userprofile-prod-analytics.sql` | User reported gender |
| 3) Clustering | none |  |



## Environment setup
At present this pipeline is designed to work within the RStudio IDE (https://www.rstudio.com/).  You will need to either install R and RStudio or use an online version of RStudio.  Alternatively, an online version is available for free on nanoHUB.org (https://nanohub.org/resources/rstudio); however, the following instructions may need to be slightly modified if using the online version.

## Instructions for using the package
1)	Create a working folder.  

1)  Download the source code from GitHub as a zip file.  Extract the zip file into the working directory created above. 

1)	The required three edX datafiles are discussed in a previous section above.  You will need to be able to point the pipeline scripts to their location.  You may move these into your working directory if you wish (although this is not necessary).  The generic filename structures of the files you need are:  
    * `{org}-{course}-{date}-courseware_studentmodule-prod-analytics.sql`
    * `{org}-{course}-{date}-auth_userprofile-prod-analytics.sql` 
    * `{org}-{course}-{date}-course_structure-prod-analytics.json`
      

1)  Open the `edX-clustering.Rproj` file found within the root directory of the gitHUB download.  This should launch RStudio (if working in the desktop).  You will see RStudio informing you of packrat packages being installed--this install may take some time the first time you open the project.

1)  Execute the `PipelineInitator.R` file either by using the RStudio Console command `source("PipelineInitiator.R")` or by opening the file in RStudio and clicking on ‘Source file...’ in the ‘Code’ menu.  This R script will sequentially call the individual R scripts which constitute the pipeline.  Follow the directions on the console (you will be asked for the location of the data files--more than once for some files).  Be aware some processing steps may require multiple minutes to complete.

{'fraction_non_alphanumeric': 0.0980262286395549, 'fraction_numerical': 0.0026493575307987814, 'mean_word_length': 4.2945301542777, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2904421', 'n_tokens_mistral': 2075, 'n_tokens_neox': 1981, 'n_words': 885}