<commit_before># Skale Reference

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->


- [Overview](#overview)
- [Core concepts](#core-concepts)
- [Working with datasets](#working-with-datasets)
  - [Sources](#sources)
  - [Transformations](#transformations)
  - [Actions](#actions)
- [Skale module](#skale-module)
  - [skale.context([config])](#skalecontextconfig)
      - [sc.env](#scenv)
      - [sc.end()](#scend)
      - [sc.lineStream(input_stream)](#sclinestreaminput_stream)
      - [sc.objectStream(input_stream)](#scobjectstreaminput_stream)
      - [sc.parallelize(array)](#scparallelizearray)
      - [sc.range(start[, end[, step]])](#scrangestart-end-step)
      - [sc.source(size, callback[, args])](#scsourcesize-callback-args)
      - [sc.textFile(path[, options])](#sctextfilepath-options)
  - [Dataset methods](#dataset-methods)
      - [ds.aggregate(reducer, combiner, init[, obj][, done])](#dsaggregatereducer-combiner-init-obj-done)
      - [ds.aggregateByKey(reducer, combiner, init,[ obj])](#dsaggregatebykeyreducer-combiner-init-obj)
      - [ds.cartesian(other)](#dscartesianother)
      - [ds.coGroup(other)](#dscogroupother)
      - [ds.collect([done])](#dscollectdone)
      - [ds.count([done])](#dscountdone)
      - [ds.countByKey([done])](#dscountbykeydone)
      - [ds.countByValue([done])](#dscountbyvaluedone)
      - [ds.distinct()](#dsdistinct)
      - [ds.filter(filter[, obj])](#dsfilterfilter-obj)
      - [ds.first([done])](#dsfirstdone)
      - [ds.flatMap(flatMapper[, obj])](#dsflatmapflatmapper-obj)
      - [ds.flatMapValues(flatMapper[, obj])](#dsflatmapvaluesflatmapper-obj)
      - [ds.forEach(callback[, obj][, done])](#dsforeachcallback-obj-done)
      - [ds.groupByKey()](#dsgroupbykey)
      - [ds.intersection(other)](#dsintersectionother)
      - [ds.join(other)](#dsjoinother)
      - [ds.keys()](#dskeys)
      - [ds.leftOuterJoin(other)](#dsleftouterjoinother)
      - [ds.lookup(k[, done])](#dslookupk-done)
      - [ds.map(mapper[, obj])](#dsmapmapper-obj)
      - [ds.mapValues(mapper[, obj])](#dsmapvaluesmapper-obj)
      - [ds.partitionBy(partitioner)](#dspartitionbypartitioner)
      - [ds.persist()](#dspersist)
      - [ds.reduce(reducer, init[, obj][, done])](#dsreducereducer-init-obj-done)
      - [ds.reduceByKey(reducer, init[, obj])](#dsreducebykeyreducer-init-obj)
      - [ds.rightOuterJoin(other)](#dsrightouterjoinother)
      - [ds.sample(withReplacement, frac, seed)](#dssamplewithreplacement-frac-seed)
      - [ds.save(url[, options][, done])](#dssaveurl-options-done)
      - [ds.sortBy(keyfunc[, ascending])](#dssortbykeyfunc-ascending)
      - [ds.sortByKey(ascending)](#dssortbykeyascending)
      - [ds.stream([opt])](#dsstreamopt)
      - [ds.subtract(other)](#dssubtractother)
      - [ds.take(num[, done])](#dstakenum-done)
      - [ds.takeSample(withReplacement, num[, done])](#dstakesamplewithreplacement-num-done)
      - [ds.top(num[, done])](#dstopnum-done)
      - [ds.union(other)](#dsunionother)
      - [ds.values()](#dsvalues)
  - [Partitioners](#partitioners)
      - [HashPartitioner(numPartitions)](#hashpartitionernumpartitions)
      - [RangePartitioner(numPartitions, keyfunc, dataset)](#rangepartitionernumpartitions-keyfunc-dataset)
  - [Environment variables](#environment-variables)
- [Machine Learning Module](#machine-learning-module)
  - [ml.binaryClassifiationMetrics(measures, options[, done])](#mlbinaryclassificationmetricsmeasures-options-done)
  - [ml.SGDLinearModel([options])](#mlsgdlinearmodeloptions)
      - [sgdClassifier.fit(trainingSet, iterations[, done])](#sgdclassifierfittrainingset-iterations-done)
      - [sgdClassifier.predict(sample)](#sgdclassifierpredictsample)
  - [ml.StandardScaler()](#mlstandardscaler)
      - [standardScaler.fit(dataset[, done])](#standardscalerfitdataset-done)
      - [standardScaler.transform(features)](#standardscalertransformfeatures)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Overview

Skale is a fast and general purpose distributed data processing
system. It provides a high-level API in Javascript and an optimized
parallel execution engine.

A Skale application consists of a *master* program that runs the
user code and executes various *parallel operations* on a cluster
of *workers*.

The main abstraction Skale provides is a *dataset* which is similar
to a Javascript *array*, but partitioned accross the workers that
can be operated in parallel.

There are several ways to create a dataset: *parallelizing* an existing
array in the master program, or referencing a dataset in a distributed
storage system (such as HDFS), or *streaming* the content of any
source that can be processed through Node.js *Streams*. We call
*source* a function which initializes a dataset.

Datasets support two kinds of operations: *transformations*, which create
a new dataset from an existing one, and *actions*, which<commit_msg>doc: rebuild toc<commit_after># Skale Reference

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->


- [Overview](#overview)
- [Core concepts](#core-concepts)
  - [Partitioning](#partitioning)
  - [Pipeline stages and shuffles](#pipeline-stages-and-shuffles)
- [Working with datasets](#working-with-datasets)
  - [Sources](#sources)
  - [Transformations](#transformations)
  - [Actions](#actions)
- [Skale module](#skale-module)
  - [skale.context([config])](#skalecontextconfig)
    - [sc.env](#scenv)
    - [sc.end()](#scend)
    - [sc.lineStream(input_stream)](#sclinestreaminput_stream)
    - [sc.objectStream(input_stream)](#scobjectstreaminput_stream)
    - [sc.parallelize(array)](#scparallelizearray)
    - [sc.range(start[, end[, step]])](#scrangestart-end-step)
    - [sc.source(size, callback[, args])](#scsourcesize-callback-args)
    - [sc.textFile(path[, options])](#sctextfilepath-options)
  - [Dataset methods](#dataset-methods)
    - [ds.aggregate(reducer, combiner, init[, obj][, done])](#dsaggregatereducer-combiner-init-obj-done)
    - [ds.aggregateByKey(reducer, combiner, init,[ obj])](#dsaggregatebykeyreducer-combiner-init-obj)
    - [ds.cartesian(other)](#dscartesianother)
    - [ds.coGroup(other)](#dscogroupother)
    - [ds.collect([done])](#dscollectdone)
    - [ds.count([done])](#dscountdone)
    - [ds.countByKey([done])](#dscountbykeydone)
    - [ds.countByValue([done])](#dscountbyvaluedone)
    - [ds.distinct()](#dsdistinct)
    - [ds.filter(filter[, obj])](#dsfilterfilter-obj)
    - [ds.first([done])](#dsfirstdone)
    - [ds.flatMap(flatMapper[, obj])](#dsflatmapflatmapper-obj)
    - [ds.flatMapValues(flatMapper[, obj])](#dsflatmapvaluesflatmapper-obj)
    - [ds.forEach(callback[, obj][, done])](#dsforeachcallback-obj-done)
    - [ds.groupByKey()](#dsgroupbykey)
    - [ds.intersection(other)](#dsintersectionother)
    - [ds.join(other)](#dsjoinother)
    - [ds.keys()](#dskeys)
    - [ds.leftOuterJoin(other)](#dsleftouterjoinother)
    - [ds.lookup(k[, done])](#dslookupk-done)
    - [ds.map(mapper[, obj])](#dsmapmapper-obj)
    - [ds.mapValues(mapper[, obj])](#dsmapvaluesmapper-obj)
    - [ds.partitionBy(partitioner)](#dspartitionbypartitioner)
    - [ds.persist()](#dspersist)
    - [ds.reduce(reducer, init[, obj][, done])](#dsreducereducer-init-obj-done)
    - [ds.reduceByKey(reducer, init[, obj])](#dsreducebykeyreducer-init-obj)
    - [ds.rightOuterJoin(other)](#dsrightouterjoinother)
    - [ds.sample(withReplacement, frac, seed)](#dssamplewithreplacement-frac-seed)
    - [ds.save(url[, options][, done])](#dssaveurl-options-done)
      - [File protocol](#file-protocol)
      - [AWS S3 protocol](#aws-s3-protocol)
      - [Azure blob storage protocol](#azure-blob-storage-protocol)
    - [ds.sortBy(keyfunc[, ascending])](#dssortbykeyfunc-ascending)
    - [ds.sortByKey(ascending)](#dssortbykeyascending)
    - [ds.stream([opt])](#dsstreamopt)
    - [ds.subtract(other)](#dssubtractother)
    - [ds.take(num[, done])](#dstakenum-done)
    - [ds.takeSample(withReplacement, num[, done])](#dstakesamplewithreplacement-num-done)
    - [ds.top(num[, done])](#dstopnum-done)
    - [ds.union(other)](#dsunionother)
    - [ds.values()](#dsvalues)
  - [Partitioners](#partitioners)
    - [HashPartitioner(numPartitions)](#hashpartitionernumpartitions)
    - [RangePartitioner(numPartitions, keyfunc, dataset)](#rangepartitionernumpartitions-keyfunc-dataset)
  - [Environment variables](#environment-variables)
- [Machine Learning module](#machine-learning-module)
  - [ml.binaryClassificationMetrics(measures, options[, done])](#mlbinaryclassificationmetricsmeasures-options-done)
  - [ml.KMeans(nbClusters[, options])](#mlkmeansnbclusters-options)
    - [kmeans.fit(trainingSet[, done])](#kmeansfittrainingset-done)
    - [kmeans.predict(sample)](#kmeanspredictsample)
  - [ml.SGDLinearModel([options])](#mlsgdlinearmodeloptions)
    - [sgdClassifier.fit(trainingSet, iterations[, done])](#sgdclassifierfittrainingset-iterations-done)
    - [sgdClassifier.predict(sample)](#sgdclassifierpredictsample)
  - [ml.StandardScaler()](#mlstandardscaler)
    - [standardScaler.fit(dataset, [done])](#standardscalerfitdataset-done)
    - [standardScaler.transform(features)](#standardscalertransformfeatures)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Overview

Skale is a fast and general purpose distributed data processing
system. It provides a high-level API in Javascript and an optimized
parallel execution engine.

A Skale application consists of a *master* program that runs the
user code and executes various *parallel operations* on a cluster
of *workers*.

The main abstraction Skale provides is a *dataset* which is similar
to a Javascript *array*, but partitioned accross the workers that
can be operated in parallel.

There are several ways to create a dataset: *parallelizing* an existing
array in the master program, or referencing a dataset in a distributed
storage system (such as HDFS), or *streaming* the content of any
source that can be processed through Node.js *Streams*. We call
*source* a function which initializes a dataset.

Datasets support two kinds of operations: *transformations*, which create
a new dataset from an existing one, and *actions*, which
{'fraction_non_alphanumeric': 0.16036556036556038, 'fraction_numerical': 0.0001924001924001924, 'mean_word_length': 4.827354260089686, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7272267', 'n_tokens_mistral': 3569, 'n_tokens_neox': 3275, 'n_words': 612}