<commit_before># Sparklyr 0.7.1 (UNRELEASED)

- `ml_transform()` now also takes a list of transformers, e.g. the result of `ml_stages()` on a `PipelineModel` (#1444).

- Default to Spark 2.3.0 for installation and local connections (#1449).

- Added `collect_sub_models` parameter to `ml_cross_validator()` and `ml_train_validation_split()` and helper function `ml_sub_models()` to allow inspecting models trained for each fold/parameter set (#1362).

- Added `parallelism` parameter to `ml_cross_validator()` and `ml_train_validation_split()` to allow tuning in parallel (#1446).

- Added support for `feature_subset_strategy` parameter in GBT algorithms (#1445).

- Added `string_order_type` to `ft_string_indexer()` to allow control over how strings are indexed (#1443).

- Added `ft_string_indexer_model()` constructor for the string indexer transformer (#1442).

- Added `ml_feature_importances()` for extracing feature importances from tree-based models (#1436). `ml_tree_feature_importance()` is maintained as an alias.

- `invoke*()` method dispatch now supports `Char` and `Short` parameters. Also, `Long` parameters now allow numeric arguments, but integers are supported for backwards compatibility (#1395).

- `sdf_copy_to()` no longer gives a spurious warning when user enters a multiline expression for `x` (#1386).

- Added `ml_vocabulary()` to extract vocabulary from count vectorizer model and `ml_topics_matrix()` to extract matrix from LDA model.

- `ml_tree_feature_importance()` now works properly with decision tree classification models (#1401).

- Added `ml_corr()` for calculating correlation matrices and `ml_chisquare_test()` for performing chi-square hypothesis testing (#1247).

- `yarn-cluster` now supported by connecting with `master="yarn"` and
  `config` entry `sparklyr.shell.deploy-mode` set to `cluster` (#1404).
  
- Added support for `offset` argument in `ml_generalized_linear_regression()` (#1396).

- `ml_save()` outputs message when model is successfully saved (#1348).

- `ml_` routines no longer capture the calling expression (#1393).

- Added support for `profile` parameter in `spark_apply()` that collects a
  profile to measure perpformance that can be rendered using the `profvis`
  package.

- Added support for Livy 0.4 and Livy 0.5.

- Added support for Huber loss for linear regression (#1335).

- `ft_bucketizer()` and `ft_quantile_discretizer()` now support
  multiple input columns (#1338, #1339).
  
- Added `ft_feature_hasher()` (#1336).

- Added `ml_clustering_evaluator()` (#1333).

- Livy now supports Kerberos authentication.

- `invoke_static()` now supports calling Scala's package objects (#1384).

- `spark_connection` and `spark_jobj` classes are now exported (#1374).

- Fixed issue with dates or date-times not roundtripping with `spark_apply() (#1376).

- `spark_available_versions()` was changed to only return available Spark versions, Hadoop versions
  can be still retrieved using `hadoop = TRUE`.

- `spark_installed_versions()` was changed to retrieve the full path to the installation folder.

- Fixed serialization issues most commonly hit while using `spark_apply()` with NAs (#1365, #1366).

- `cbind()` and `sdf_bind_cols()` don't use NSE internally anymore and no longer output names of mismatched data frames on error (#1363).

- Fixed data frame provided by `spark_apply()` to not provide characters not factors (#1313).

- `ml_default_stop_words()` now returns English stop words by default (#1280).

- Support the `sdf_predict(ml_transformer, dataset)` signature with a deprecation warning. Also added a deprecation warning to the usage of `sdf_predict(ml_model, dataset)`. (#1287)

- `sample_frac()` and `sample_n()` now work properly in nontrivial queries (#1299)

- Added support for `spark_apply()` under Livy connections.

- Fixed regression blocking use of response-features syntax in some `ml_`functions (#1302).

- Fixed file not found error in `spark_apply()` while working under low
  disk space.

- Fixed regression blocking use of `ml_kmeans()` in Spark 1.6.x.

- Fixed regression blocking use of `livy.session.start.timeout` parameter
  in Livy connections.

- Added support for `sparklyr.apply.options.rscript.before` to run a custom
  command before launching the R worker role.

- Added support for `sparklyr.apply.options.vanilla` to be set to `FALSE`
  to avoid using `--vanilla` while launching R worker role.

# Sparklyr 0.7

- Added support for Spark 2.2.1.

- Switched `copy_to` serializer to use Scala implementation, this change can be
  reverted by setting the `sparklyr.copy.serializer` option to `csv_file`.

- Added support for `spark_web()` for Livy and Databricks connections when
  using Spark 2.X.

- Fixed `SIGPIPE` error under `spark_connect()` immediately after
  a `spark_disconnect()` operation.

- `spark_web()` is is more reliable under Spark 2.X by making use of a new API
  to programmatically find the right address.

- Added support in `dbWriteTable()` for `temporary = FALSE` to allow persisting
  table across connections. Changed default value for `temporary` to `TRUE` to match
  `DBI` specification, for compatibility, default value can be reverted back to 
  `FALSE` using the `sparklyr.dbwritetable.temp` option.

- `ncol()` now returns the number of columns instead of `NA`, and `nrow()` now
  returns `NA_real_`.

- Added support to collect `VectorUDT` column types with nested arrays.

- Fixed issue in which connecting to Livy would fail due to long user names
  or long passwords.
<commit_msg>copy-edit NEWS.md<commit_after># Sparklyr 0.8.0

### Spark ML

- `ml_transform()` now also takes a list of transformers, e.g. the result of `ml_stages()` on a `PipelineModel` (#1444).

- Added `collect_sub_models` parameter to `ml_cross_validator()` and `ml_train_validation_split()` and helper function `ml_sub_models()` to allow inspecting models trained for each fold/parameter set (#1362).

- Added `parallelism` parameter to `ml_cross_validator()` and `ml_train_validation_split()` to allow tuning in parallel (#1446).

- Added support for `feature_subset_strategy` parameter in GBT algorithms (#1445).

- Added `string_order_type` to `ft_string_indexer()` to allow control over how strings are indexed (#1443).

- Added `ft_string_indexer_model()` constructor for the string indexer transformer (#1442).

- Added `ml_feature_importances()` for extracing feature importances from tree-based models (#1436). `ml_tree_feature_importance()` is maintained as an alias.

- Added `ml_vocabulary()` to extract vocabulary from count vectorizer model and `ml_topics_matrix()` to extract matrix from LDA model.

- `ml_tree_feature_importance()` now works properly with decision tree classification models (#1401).

- Added `ml_corr()` for calculating correlation matrices and `ml_chisquare_test()` for performing chi-square hypothesis testing (#1247).

- `ml_save()` outputs message when model is successfully saved (#1348).

- `ml_` routines no longer capture the calling expression (#1393).

- Added support for `offset` argument in `ml_generalized_linear_regression()` (#1396).

- Fixed regression blocking use of response-features syntax in some `ml_`functions (#1302).

- Added support for Huber loss for linear regression (#1335).

- `ft_bucketizer()` and `ft_quantile_discretizer()` now support
  multiple input columns (#1338, #1339).
  
- Added `ft_feature_hasher()` (#1336).

- Added `ml_clustering_evaluator()` (#1333).

- `ml_default_stop_words()` now returns English stop words by default (#1280).

- Support the `sdf_predict(ml_transformer, dataset)` signature with a deprecation warning. Also added a deprecation warning to the usage of `sdf_predict(ml_model, dataset)`. (#1287)

- Fixed regression blocking use of `ml_kmeans()` in Spark 1.6.x.

### Extensions

- `invoke*()` method dispatch now supports `Char` and `Short` parameters. Also, `Long` parameters now allow numeric arguments, but integers are supported for backwards compatibility (#1395).

- `invoke_static()` now supports calling Scala's package objects (#1384).

- `spark_connection` and `spark_jobj` classes are now exported (#1374).

### Distributed R

- Added support for `profile` parameter in `spark_apply()` that collects a
  profile to measure perpformance that can be rendered using the `profvis`
  package.

- Added support for `spark_apply()` under Livy connections.

- Fixed file not found error in `spark_apply()` while working under low
  disk space.

- Added support for `sparklyr.apply.options.rscript.before` to run a custom
  command before launching the R worker role.

- Added support for `sparklyr.apply.options.vanilla` to be set to `FALSE`
  to avoid using `--vanilla` while launching R worker role.
  
- Fixed serialization issues most commonly hit while using `spark_apply()` with NAs (#1365, #1366).

- Fixed issue with dates or date-times not roundtripping with `spark_apply() (#1376).

- Fixed data frame provided by `spark_apply()` to not provide characters not factors (#1313).

### Miscellaneous

- Fixed regression blocking use of `livy.session.start.timeout` parameter
  in Livy connections.

- Added support for Livy 0.4 and Livy 0.5.

- Livy now supports Kerberos authentication.

- Default to Spark 2.3.0 for installation and local connections (#1449).

- `yarn-cluster` now supported by connecting with `master="yarn"` and
  `config` entry `sparklyr.shell.deploy-mode` set to `cluster` (#1404).
  
- `sample_frac()` and `sample_n()` now work properly in nontrivial queries (#1299)

- `sdf_copy_to()` no longer gives a spurious warning when user enters a multiline expression for `x` (#1386).

- `spark_available_versions()` was changed to only return available Spark versions, Hadoop versions
  can be still retrieved using `hadoop = TRUE`.

- `spark_installed_versions()` was changed to retrieve the full path to the installation folder.

- `cbind()` and `sdf_bind_cols()` don't use NSE internally anymore and no longer output names of mismatched data frames on error (#1363).

# Sparklyr 0.7.0

- Added support for Spark 2.2.1.

- Switched `copy_to` serializer to use Scala implementation, this change can be
  reverted by setting the `sparklyr.copy.serializer` option to `csv_file`.

- Added support for `spark_web()` for Livy and Databricks connections when
  using Spark 2.X.

- Fixed `SIGPIPE` error under `spark_connect()` immediately after
  a `spark_disconnect()` operation.

- `spark_web()` is is more reliable under Spark 2.X by making use of a new API
  to programmatically find the right address.

- Added support in `dbWriteTable()` for `temporary = FALSE` to allow persisting
  table across connections. Changed default value for `temporary` to `TRUE` to match
  `DBI` specification, for compatibility, default value can be reverted back to 
  `FALSE` using the `sparklyr.dbwritetable.temp` option.

- `ncol()` now returns the number of columns instead of `NA`, and `nrow()` now
  returns `NA_real_`.

- Added support to collect `VectorUDT` column types with nested arrays.

- Fixed issue in which connecting to Livy would fail due to long user names
  or long passwords.

{'fraction_non_alphanumeric': 0.09726088908845981, 'fraction_numerical': 0.026493039964077234, 'mean_word_length': 4.728395061728395, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1602623', 'n_tokens_mistral': 3613, 'n_tokens_neox': 3130, 'n_words': 1346}