<commit_before># PySpark Stubs

A collection of the Apache Spark [stub files](https://www.python.org/dev/peps/pep-0484/#stub-files). These files were generated by [stubgen](https://github.com/python/mypy/blob/master/mypy/stubgen.py) and manually edited to include accurate type hints.

Tests and configuration files have been originally contributed to the [Typeshed project](https://github.com/python/typeshed/). Please refer to its [contributors list](https://github.com/python/typeshed/graphs/contributors) and [license](https://github.com/python/typeshed/blob/master/LICENSE) for details.

## Usage

According to [PEP 484](https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files): 

> Third-party stub packages can use any location for stub storage. Type checkers should search for them using PYTHONPATH. 

Adding `third_party/3` to the `PYTHONPATH` seems to the trick.

## Disclaimer

Apache Spark, Spark, Apache, and the Spark logo are <a href="https://www.apache.org/foundation/marks/">trademarks</a> of
  <a href="http://www.apache.org">The Apache Software Foundation</a>. This project is not owned, endorsed, or sponsored by The Apache Software Foundation.<commit_msg>Add API Coverage section to README<commit_after># PySpark Stubs

A collection of the Apache Spark [stub files](https://www.python.org/dev/peps/pep-0484/#stub-files). These files were generated by [stubgen](https://github.com/python/mypy/blob/master/mypy/stubgen.py) and manually edited to include accurate type hints.

Tests and configuration files have been originally contributed to the [Typeshed project](https://github.com/python/typeshed/). Please refer to its [contributors list](https://github.com/python/typeshed/graphs/contributors) and [license](https://github.com/python/typeshed/blob/master/LICENSE) for details.


## Usage

According to [PEP 484](https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files): 

> Third-party stub packages can use any location for stub storage. Type checkers should search for them using PYTHONPATH. 

Adding `third_party/3` to the `PYTHONPATH` seems to the trick.

## API Coverage

| Module                                      | Dynamically typed | Statically typed | Notes            |
|---------------------------------------------|-------------------|------------------|------------------|
| pyspark.shell                               | ✘                 | ✘                |                  |
| pyspark.rddsampler                          | ✘                 | ✘                |                  |
| pyspark.java\_gateway                       | ✘                 | ✘                |                  |
| pyspark.tests                               | ✘                 | ✘                |                  |
| pyspark.join                                | ✘                 | ✘                |                  |
| pyspark.statcounter                         | ✔                 | ✘                |                  |
| pyspark.serializers                         | ✔                 | ✘                |                  |
| pyspark.heapq3                              | ✘                 | ✘                |                  |
| pyspark                                     | ✔                 | ✘                |                  |
| pyspark.traceback\_utils                    | ✘                 | ✘                |                  |
| pyspark.profiler                            | ✔                 | ✘                |                  |
| pyspark.storagelevel                        | ✔                 | ✘                |                  |
| pyspark.broadcast                           | ✔                 | ✔                | Mixed            |
| pyspark.worker                              | ✘                 | ✘                |                  |
| pyspark.shuffle                             | ✘                 | ✘                |                  |
| pyspark.files                               | ✘                 | ✘                |                  |
| pyspark.find\_spark\_home                   | ✘                 | ✘                |                  |
| pyspark.status                              | ✔                 | ✘                |                  |
| pyspark.daemon                              | ✘                 | ✘                |                  |
| pyspark.context                             | ✘                 | ✔                |                  |
| pyspark.cloudpickle                         | ✘                 | ✘                |                  |
| pyspark.accumulators                        | ✔                 | ✘                |                  |
| pyspark.version                             | ✘                 | ✘                |                  |
| pyspark.resultiterable                      | ✘                 | ✘                |                  |
| pyspark.conf                                | ✔                 | ✘                |                  |
| pyspark.rdd                                 | ✘                 | ✔                |                  |
| pyspark.ml.tuning                           | ✘                 | ✘                |                  |
| pyspark.ml.clustering                       | ✘                 | ✘                |                  |
| pyspark.ml.feature                          | ✘                 | ✘                |                  |
| pyspark.ml.tests                            | ✘                 | ✘                |                  |
| pyspark.ml.common                           | ✘                 | ✘                |                  |
| pyspark.ml                                  | ✘                 | ✘                |                  |
| pyspark.ml.wrapper                          | ✘                 | ✘                |                  |
| pyspark.ml.classification                   | ✘                 | ✘                |                  |
| pyspark.ml.recommendation                   | ✘                 | ✘                |                  |
| pyspark.ml.base                             | ✘                 | ✘                |                  |
| pyspark.ml.regression                       | ✘                 | ✘                |                  |
| pyspark.ml.pipeline                         | ✘                 | ✘                |                  |
| pyspark.ml.util                             | ✘                 | ✘                |                  |
| pyspark.ml.evaluation                       | ✘                 | ✘                |                  |
| pyspark.ml.param._shared_params\_code\_gen  | ✘                 | ✘                |                  |
| pyspark.ml.param                            | ✘                 | ✘                |                  |
| pyspark.ml.param.shared                     | ✘                 | ✘                |                  |
| pyspark.ml.linalg                           | ✘                 | ✘                |                  |
| pyspark.mllib.clustering                    | ✘                 | ✘                |                  |
| pyspark.mllib.feature                       | ✘                 | ✘                |                  |
| pyspark.mllib.tests                         | ✘                 | ✘                |                  |
| pyspark.mllib.common                        | ✘                 | ✘                |                  |
| pyspark.mllib.tree                          | ✘                 | ✘                |                  |
| pyspark.mllib                               | ✘                 | ✘                |                  |
| pyspark.mllib.classification                | ✘                 | ✘                |                  |
| pyspark.mllib.recommendation                | ✘                 | ✘                |                  |
| pyspark.mllib.regression                    | ✘                 | ✘                |                  |
| pyspark.mllib.util                          | ✘                 | ✘                |                  |
| pyspark.mllib.evaluation                    | ✘                 | ✘                |                  |
| pyspark.mllib.random                        | ✘                 | ✘                |                  |
| pyspark.mllib.fpm                           | ✘                 | ✘                |                  |
| pyspark.mllib.linalg.distributed            | ✘                 | ✘                |                  |
| pyspark.mllib.linalg                        | ✘                 | ✘                |                  |
| pyspark.mllib.stat._statistics              | ✘                 | ✘                |                  |
| pyspark.mllib.stat.test                     | ✘                 | ✘                |                  |
| pyspark.mllib.stat.distribution             | ✘                 | ✘                |                  |
| pyspark.mllib.stat.KernelDensity            | ✘                 | ✘                |                  |
| pyspark.mllib.stat                          | ✘                 | ✘                |                  |
| pyspark.streaming.kafka                     | ✘                 | ✘                |                  |
| pyspark.streaming.flume                     | ✘                 | ✘                |                  |
| pyspark.streaming.tests                     | ✘                 | ✘                |                  |
| pyspark.streaming                           | ✘                 | ✘                |                  |
| pyspark.streaming.listener                  | ✘                 | ✘                |                  |
| pyspark.streaming.util                      | ✘                 | ✘                |                  |
| pyspark.streaming.dstream                   | ✘                 | ✘                |                  |
| pyspark.streaming.context                   | ✘                 | ✘                |                  |
| pyspark.streaming.kinesis                   | ✘                 | ✘                |                  |
| pyspark.sql.types                           | ✔                 | ✘                |                  |
| pyspark.sql.functions                       | ✘                 | ✔                |                  |
| pyspark.sql.catalog                         | ✘                 | ✘                |                  |
| pyspark.sql.tests                           | ✘                 | ✘                |                  |
| pyspark.sql.streaming                       | ✔                 | ✘                |                  |
| pyspark.sql.column                          | ✘                 | ✔                |                  |
| pyspark.sql                                 | ✔                 | ✘                |                  |
| pyspark.sql.window                          | ✔                 | ✘                |                  |
| pyspark.sql.session                         | ✔                 | ✘                |                  |
| pyspark.sql.utils                           | ✘                 | ✘                |                  |
| pyspark.sql.group                           | ✘                 | ✔                |                  |
| pyspark.sql.readwriter                      | ✔                 | ✘                |                  |
| pyspark.sql.dataframe                       | ✘                 | ✔                |                  |
| pyspark.sql.context                         | ✘                 | ✔                |                  |
| pyspark.sql.conf                            | ✘                 | ✘                |                  |




## Disclaimer

Apache Spark, Spark, Apache, and the Spark logo are <a href="https://www.apache.org/foundation/marks/">trademarks</a> of
  <a href="http://www.apache.org">The Apache Software Foundation</a>. This project is not owned, endorsed, or sponsored by The Apache Software Foundation.
{'fraction_non_alphanumeric': 0.09817180065113949, 'fraction_numerical': 0.002086985558059938, 'mean_word_length': 0.5510098394614189, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 14, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 265, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3123542', 'n_tokens_mistral': 3286, 'n_tokens_neox': 2872, 'n_words': 497}