<commit_before># TypedDataset: Feature Overview

This tutorial introduces `TypedDataset` using a simple example.
The following imports are needed to make all code examples compile.

```tut:silent
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SparkSession
import frameless.functions.aggregate._
import frameless.TypedDataset

val conf = new SparkConf().setMaster("local[*]").setAppName("frameless repl").set("spark.ui.enabled", "false")
val spark = SparkSession.builder().config(conf).appName("REPL").getOrCreate()
implicit val sqlContext = spark.sqlContext
spark.sparkContext.setLogLevel("WARN")

import spark.implicits._
```

## Creating TypedDataset instances

We start by defining a case class:

```tut:silent
case class Apartment(city: String, surface: Int, price: Double)
```

Let's define few `Apartment` instances:

```tut:silent
val apartments = Seq(
  Apartment("Paris", 50, 300000.0),
  Apartment("Paris", 100, 450000.0),
  Apartment("Paris", 25, 250000.0),
  Apartment("Lyon", 83, 200000.0),
  Apartment("Lyon", 45, 133000.0),
  Apartment("Nice", 74, 325000.0)
)
```

We are now ready to instantiate a `TypedDataset[Apartment]:`

```tut:book
val aptTypedDs = TypedDataset.create(apartments)
```

We can also create it from an existing `Dataset:`

```tut:book
val aptDs = spark.createDataset(apartments)
val aptTypedDs = TypedDataset.create(aptDs)
```

Or use the Frameless syntax:

```tut:book
import frameless.syntax._

val aptTypedDs2 = spark.createDataset(apartments).typed
```

## Typesafe column referencing
This is how we select a particular column from a `TypedDataset`:

```tut:book
val cities: TypedDataset[String] = aptTypedDs.select(aptTypedDs('city))
```

This is completely type-safe, for instance suppose we misspell `city` as `citi:`

```tut:book:fail
aptTypedDs.select(aptTypedDs('citi))
```

This gets raised at compile-time, whereas with the standard `Dataset` API the error appears at run-time (enjoy the stack trace):

```tut:book:fail
aptDs.select('citi)
```

`select()` supports arbitrary column operations:

```tut:book
aptTypedDs.select(aptTypedDs('surface) * 10, aptTypedDs('surface) + 2).show().run()
```

Note that unlike the standard Spark API where some operations are lazy and some are not, **TypedDatasets have all operations to be lazy.** 
In the above example, `show()` is lazy. It requires to apply `run()` for the `show` job to materialize.


Let us now try to compute the price by surface unit:

```tut:book:fail
val priceBySurfaceUnit = aptTypedDs.select(aptTypedDs('price) / aptTypedDs('surface))
```

As the error suggests, we can't divide a `TypedColumn` of `Double` by `Int.` 
For safety, in Frameless only math operations between same types is allowed. 
There are two ways to proceed here: 

(a) Explicitly cast `Int` to `Double` (manual)

```tut:book
val priceBySurfaceUnit = aptTypedDs.select(aptTypedDs('price) / aptTypedDs('surface).cast[Double])
priceBySurfaceUnit.collect().run()
```

(b) Perform the cast implicitly (automated)

```tut:book
import frameless.implicits.widen._

val priceBySurfaceUnit = aptTypedDs.select(aptTypedDs('price) / aptTypedDs('surface))
priceBySurfaceUnit.collect.run()
```

Looks like it worked, but that `cast` seems unsafe right? Actually it is safe.
Let's try to cast a `TypedColumn` of `String` to `Double:`

```tut:book:fail
aptTypedDs('city).cast[Double]
```

The compile-time error tells us that to perform the cast, an evidence 
(in the form of `CatalystCast[String, Double]`) must be available. 
Since casting from `String` to `Double` is not allowed, this results 
in a compilation error. 

Check [here](https://github.com/typelevel/frameless/blob/master/core/src/main/scala/frameless/CatalystCast.scala) 
for the set of available `CatalystCast.`

## TypeSafe TypedDataset casting and projections

With `select()` the resulting TypedDataset is of type `TypedDataset[TupleN[...]]` (with N in `[1...10]`).
For example, if we select three columns with types `String`, `Int`, and `Boolean` the result will have type
`TypedDataset[(String, Int, Boolean)]`. To select more than ten columns use the `selectMany()` method. 
Select has better IDE support that the macro based selectMany, so prefer `select()` for the general case. 

We often want to give more expressive types to the result of our computations.
`as[T]` allows us to safely cast a `TypedDataset[U]` to another of type `TypedDataset[T]` as long
as the types in `U` and `T` align.

When the cast is valid the expression compiles:

```tut:book
case class UpdatedSurface(city: String, surface: Int)
val updated = aptTypedDs.select(aptTypedDs('city), aptTypedDs('surface) + 2).as[UpdatedSurface]
updated.show(2).run()
```

Next we try to cast a `(String, String)` to an `UpdatedSurface` (which has types `String`, `Int`).
The cast is not valid and the expression does not compile:

```tut:book:fail
aptTypedDs.select(aptTypedDs('city), aptTypedDs('city)).as[UpdatedSurface]
```

### Projections<commit_msg>Cleaning up the getting started guide.<commit_after># TypedDataset: Feature Overview

This tutorial introduces `TypedDataset` using a simple example.
The following imports are needed to make all code examples compile.

```tut:silent
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SparkSession
import frameless.functions.aggregate._
import frameless.TypedDataset

val conf = new SparkConf().setMaster("local[*]").setAppName("frameless repl").set("spark.ui.enabled", "false")
val spark = SparkSession.builder().config(conf).appName("REPL").getOrCreate()
implicit val sqlContext = spark.sqlContext
spark.sparkContext.setLogLevel("WARN")

import spark.implicits._
```

## Creating TypedDataset instances

We start by defining a case class:

```tut:silent
case class Apartment(city: String, surface: Int, price: Double)
```

And few `Apartment` instances:

```tut:silent
val apartments = Seq(
  Apartment("Paris", 50, 300000.0),
  Apartment("Paris", 100, 450000.0),
  Apartment("Paris", 25, 250000.0),
  Apartment("Lyon", 83, 200000.0),
  Apartment("Lyon", 45, 133000.0),
  Apartment("Nice", 74, 325000.0)
)
```

We are now ready to instantiate a `TypedDataset[Apartment]:`

```tut:book
val aptTypedDs = TypedDataset.create(apartments)
```

We can also create one from an existing Spark `Dataset:`

```tut:book
val aptDs = spark.createDataset(apartments)
val aptTypedDs = TypedDataset.create(aptDs)
```

Or use the Frameless syntax:

```tut:book
import frameless.syntax._

val aptTypedDs2 = aptDs.typed
```

## Typesafe column referencing
This is how we select a particular column from a `TypedDataset`:

```tut:book
val cities: TypedDataset[String] = aptTypedDs.select(aptTypedDs('city))
```

This is completely type-safe, for instance suppose we misspell `city` as `citi:`

```tut:book:fail
aptTypedDs.select(aptTypedDs('citi))
```

This gets raised at compile-time, whereas with the standard `Dataset` API the error appears at run-time (enjoy the stack trace):

```tut:book:fail
aptDs.select('citi)
```

`select()` supports arbitrary column operations:

```tut:book
aptTypedDs.select(aptTypedDs('surface) * 10, aptTypedDs('surface) + 2).show().run()
```

Note that unlike the standard Spark API where some operations are lazy and some are not, **TypedDatasets have all operations to be lazy.** 
In the above example, `show()` is lazy. It requires to apply `run()` for the `show` job to materialize.


Next we compute the price by surface unit:

```tut:book:fail
val priceBySurfaceUnit = aptTypedDs.select(aptTypedDs('price) / aptTypedDs('surface))
```

As the error suggests, we can't divide a `TypedColumn` of `Double` by `Int.` 
For safety, in Frameless only math operations between same types is allowed. 
There are two ways to proceed here: 

(a) Explicitly cast `Int` to `Double` (manual)

```tut:book
val priceBySurfaceUnit = aptTypedDs.select(aptTypedDs('price) / aptTypedDs('surface).cast[Double])
priceBySurfaceUnit.collect().run()
```

(b) Perform the cast implicitly (automated)

```tut:book
import frameless.implicits.widen._

val priceBySurfaceUnit = aptTypedDs.select(aptTypedDs('price) / aptTypedDs('surface))
priceBySurfaceUnit.collect.run()
```

Looks like it worked, but that `cast` seems unsafe right? Actually it is safe.
Let's try to cast a `TypedColumn` of `String` to `Double:`

```tut:book:fail
aptTypedDs('city).cast[Double]
```

The compile-time error tells us that to perform the cast, an evidence 
(in the form of `CatalystCast[String, Double]`) must be available. 
Since casting from `String` to `Double` is not allowed, this results 
in a compilation error. 

Check [here](https://github.com/typelevel/frameless/blob/master/core/src/main/scala/frameless/CatalystCast.scala) 
for the set of available `CatalystCast.`

## TypeSafe TypedDataset casting and projections

With `select()` the resulting TypedDataset is of type `TypedDataset[TupleN[...]]` (with N in `[1...10]`).
For example, if we select three columns with types `String`, `Int`, and `Boolean` the result will have type
`TypedDataset[(String, Int, Boolean)]`. To select more than ten columns use the `selectMany()` method. 
Select has better IDE support than the macro based selectMany, so prefer `select()` for the general case. 

We often want to give more expressive types to the result of our computations.
`as[T]` allows us to safely cast a `TypedDataset[U]` to another of type `TypedDataset[T]` as long
as the types in `U` and `T` align.

When the cast is valid the expression compiles:

```tut:book
case class UpdatedSurface(city: String, surface: Int)
val updated = aptTypedDs.select(aptTypedDs('city), aptTypedDs('surface) + 2).as[UpdatedSurface]
updated.show(2).run()
```

Next we try to cast a `(String, String)` to an `UpdatedSurface` (which has types `String`, `Int`).
The cast is not valid and the expression does not compile:

```tut:book:fail
aptTypedDs.select(aptTypedDs('city), aptTypedDs('city)).as[UpdatedSurface]
```

### Projections
{'fraction_non_alphanumeric': 0.12558980022086136, 'fraction_numerical': 0.012850115450255998, 'mean_word_length': 5.584269662921348, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4920178', 'n_tokens_mistral': 3366, 'n_tokens_neox': 3150, 'n_words': 1081}