<commit_before># Things to do for SparkR

## Functions to support in RDD

1. `reduceByKey` and `groupByKey` -- Depends on implementing partitioner and PairRDD
2. Similar to `stats.py` in Python, add support for mean, median, stdev etc.

## Other features to support

1. Broadcast variables.
2. Allow R packages to be loaded into the run time. Also consider if we need to extend
this for any given R file to be sourced in the worker.
3. Use long-running R worker daemons to avoid forking a process each time.
4. Memoizations of frequently queried vals in RDD, such as numPartitions.

## Longer term wishlist

1. RRDDs are distributed lists. Extend them to create a distributed data frame.
2. Integration with ML Lib to run ML algorithms from R.
3. Profile serialization overhead and see if there is anything better we can do.
4. Reduce code duplication between SparkR and PySpark3. 4. .

## Testing plan

### sparkR, context

1. Fix JAR path in sparkR.R
2. textFile + collect -- use README.md, or some test file
   1. Check if minSplits works correctly
3. Check if parallelize has enough tests for tuples (+ collect)

### RRDD

4. Add a test for count, length
5. Add one test for lapply, lapplyPartition
6. Add a test for reduce
7. Add tests for groupByKey, reduceByKey

### Utils
8. utils.R - Check if dependencies are serialized correctly
9. convertJListToRList

## Documentation<commit_msg>Add perf todo<commit_after># Things to do for SparkR

## Functions to support in RDD

1. `reduceByKey` and `groupByKey` -- Depends on implementing partitioner and PairRDD
2. Similar to `stats.py` in Python, add support for mean, median, stdev etc.

## Other features to support

1. Broadcast variables.
2. Allow R packages to be loaded into the run time. Also consider if we need to extend
this for any given R file to be sourced in the worker.
3. Use long-running R worker daemons to avoid forking a process each time.
4. Memoizations of frequently queried vals in RDD, such as numPartitions.

## Longer term wishlist

1. RRDDs are distributed lists. Extend them to create a distributed data frame.
2. Integration with ML Lib to run ML algorithms from R.
3. Profile serialization overhead and see if there is anything better we can do.
4. Reduce code duplication between SparkR and PySpark
. Add more examples (machine learning ?) and some performance benchmarks.


## Testing plan

### sparkR, context

1. Fix JAR path in sparkR.R
2. textFile + collect -- use README.md, or some test file
   1. Check if minSplits works correctly
3. Check if parallelize has enough tests for tuples (+ collect)

### RRDD

4. Add a test for count, length
5. Add one test for lapply, lapplyPartition
6. Add a test for reduce
7. Add tests for groupByKey, reduceByKey

### Utils
8. utils.R - Check if dependencies are serialized correctly
9. convertJListToRList

## Documentation
{'fraction_non_alphanumeric': 0.06030855539971949, 'fraction_numerical': 0.014726507713884993, 'mean_word_length': 3.803030303030303, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5569842', 'n_tokens_mistral': 836, 'n_tokens_neox': 816, 'n_words': 447}