<jupyter_start><jupyter_text>1. Variables, frames and data manipulationThere are two main data structures used all over the place: variables and frames. A variable is a list of values of the same type. You can think of a variable as a column in a tabular data format. A set of variables is a data frame. You can think about a frame as a table, with rows for observations and columns for variables.Let's take a simple example. We will load the iris data set, which is already contained in the library.<jupyter_code>import rapaio.data.*;
import rapaio.datasets.*;
import rapaio.core.distributions.*;
import rapaio.sys.*;

import java.util.stream.*
WS.getPrinter().withTextWidth(110);
Frame df = Datasets.loadIrisDataset();
df.printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 150
* complete: 150/150
* varCount: 5
* varNames: 

0. sepal-length : double  | 3.  petal-width : double  | 
1.  sepal-width : double  | 4.        class : nominal | 
2. petal-length : double  | 

   sepal-length      sepal-width     petal-length      petal-width            class 
   Min. : 4.300     Min. : 2.000     Min. : 1.000     Min. : 0.100      setosa : 50 
1st Qu. : 5.100  1st Qu. : 2.800  1st Qu. : 1.600  1st Qu. : 0.300  versicolor : 50 
 Median : 5.800   Median : 3.000   Median : 4.350   Median : 1.300   virginica : 50 
   Mean : 5.843     Mean : 3.057     Mean : 3.758     Mean : 1.199                  
2nd Qu. : 6.400  2nd Qu. : 3.300  2nd Qu. : 5.100  2nd Qu. : 1.800                  
   Max. : 7.900     Max. : 4.400     Max. : 6.900     Max. : 2.500<jupyter_text>Frame summary is a simple way to see some general information about a data frame. We see the data frame contains $150$ observations. The data set contains $5$ variables.The listing continues with enumerating the name and type of the variables contained in a data frame. Notice that there are four double variables and one nominal variable, named `class`.The summary listing ends with a section which describes each variable. For double variables the summary contains $6$ number summary. These are some sample order statistics and sample mean. We have there the minimum and maximum, median, first and third quartile and the mean. For nominal values we have an enumeration of the first most frequent levels and the associated counts. For our `class` variable we see that there are three levels, each with $50$ instances.  1.1 VariablesIn statistics a variable has multiple meanings. A random variable is a process which produces values according to a distribution. Random variables can have multiple dimensions, so they can be grouped in vectors. The rapaio library does not model the concept of random variable. Instead the `Var` objects models the concept of values drawn from a unidimensional random variable, in other words a sample of values. As a consequence a `Var` object has a size and uses indexes to access the values from the sample. A rapaio variable is a vector of values which have the same type and shares the same meaning. Each variable implements an interface called `Var`. Interface `Var` implements various useful methods for various kinds of tasks:* _manipulate values_ from the variable by adding, removing, inserting and updating with different representations* _naming_ a variable offers an alternate way to identify a variable into a frame and it is also useful for nice output information* _manipulate sets of values_ by allowing variables concatenation by binding rows and filter out values by mapping* _streaming_ allows traversal of variables by java 8 streams* other tools like deep copy, deep compare, summary, etc 1.1.1 VarType: storage and representation of a variableThere are two main concepts which have to be understood when working with variables: **storage** and **representation**. All the variables are able to store data inside using a certain Java data type, for example `double`, `int`, `String`, etc. One variable can use for storage and internal manipulations a single Java data type. In the same time, the data from variables can be represented in different ways, all of them being available through the `Var` interface for all types of variables.However not all the representations are possible for all types of variables, because some of them does not make sense. For example double floating values can be represented as strings, which is fine, however strings in general cannot be represented as double values.These are the following data representations all the `Var`-iables can implement:* **double** - double* **label** - String* **int** - int* **long** - long / InstantThe `Var` interface offers methods to get/update/insert values for all those data representations. Again, notice that not all data representations are available for all variables. For example the label representation is available for all sort of variables. This is acceptable, since when storing information into a text-like data format, any data type should be transformed into a string and should be read from a string representation.To accomodate all those legal possibilities, the rapaio library has a set of predefined variable types, which can be found in the enum `VarType`.The defined variable types:* **BINARY** - binary values* **INT** - integer values* **NOMINAL** - string values from a predefined set of values, with no ordering \(for example: _male_, _female_\)* **DOUBLE** - double precision floating point values* **LONG** - time related variables* **TEXT** - strings with free formA data type is important for the following reasons:* gives a certain useful meaning for variables in such a way that machine learning or statistical algorithms can leverage to maximum potential the meta information about variables* encapsulates the stored data type artifacts and hide those details from the user, while allowing the usage of a single unitar interface for all variables  1.1.2 Numeric double variablesNumeric double variables are defined by`VarDouble`and contains discrete or continuous numerical values stored as double precision float values. Double variables offers value and label representations. All other representations can be used, but with caution since it can alter the content. For example int representation truncates floating point values to the biggest integer values, however the int setter sets a correct value since an integer can be converted to a double. 1.1.2.1 Various buildersDouble variables can be built in various was and are handy shortcuts for various scenarios.**Empty variables**<jupyter_code>// builds a variable with no data
Var empty1 = VarDouble.empty();

// builds a variable of a given size which contains only missing data
Var empty2 = VarDouble.empty(100);<jupyter_output><empty_output><jupyter_text>**Scalar variables**<jupyter_code>Var scalar = VarDouble.scalar(Math.PI);<jupyter_output><empty_output><jupyter_text>**Sequence of values**<jupyter_code>// a sequence of numbers, starting from 0, ending with 5 with step 1
Var seq1 = VarDouble.seq(5);

// a sequence of numbers, starting from 1, ending with 5 with step 1
Var seq2 = VarDouble.seq(1, 5);

// a sequence of numbers starting at 0, ending at 1 with step 0.1
Var seq3 = VarDouble.seq(0, 1, 0.1);<jupyter_output><empty_output><jupyter_text>**Variables filled with the same number**<jupyter_code>// build a variable of a given size which contains only zeros
Var fill1 = VarDouble.fill(5);

// builds a variable of a given size which contains only ones
Var fill2 = VarDouble.fill(5, 1);<jupyter_output><empty_output><jupyter_text>**Copy from another source**<jupyter_code>// numeric variable which contains the values copied from another variable
Var copy1 = VarDouble.copy(seq1);

// numeric variable with values copied from a collection
Normal normal = Normal.std();
List<Double> list1 = DoubleStream.generate(normal::sampleNext)
.limit(10)
.boxed()
.collect(Collectors.toList());
Var copy2 = VarDouble.copy(list1);

// numeric variable with values copied from a double array
Var copy3 = VarDouble.copy(1, 3, 4.0, 7);

// numeric variable with values copied from an int array
Var copy4 = VarDouble.copy(1, 3, 4, 7);<jupyter_output><empty_output><jupyter_text>**Generated using a lambda function**<jupyter_code>// numeric variables with values generated as the sqrt of the row number
Var from1 = VarDouble.from(10, Math::sqrt);

// numeric variable with values generated using a function which receives a row value
// as parameter and outputs a double value; in this case we generate values as
// a sum of the values of other two variables
Var from2 = VarDouble.from(4, row -> copy3.getDouble(row) + copy4.getDouble(row));

// numeric variable with values generated from values of another variable using
// a transformation provided via a lambda function
Var from3 = VarDouble.from(from1, x -> x + 1);<jupyter_output><empty_output><jupyter_text>**Wrapper around a double array**This builder creates a new numeric variable instance as a wrapper around a double array of values. Notice that it is not the same as the copy builder, since in the wrapper case any change in the new numerical variable is reflected also in the original array of numbers. In the case of the copy builder this is not true, since the copy builder \(as its name implies\) creates an internal copy of the array.<jupyter_code>double[] src = new double[] {1, 4, 19, 23, 5};
Var wrap1 = VarDouble.wrap(src);<jupyter_output><empty_output><jupyter_text>1.1.2.2 Inspect a numerical variableMost of the objects which contains information implements the `Printable` interface. This interface allows one to display a summary of the content of the given object. This is the case also with the numerical variables. Additionally, the numerical variables implements also two other methods, one which displays all the values and another one which displays only the first values.<jupyter_code>// build a numerical variable with values as the sqrt
// of the first 30 integer values
Var x = VarDouble.from(30, Math::sqrt).withName("x");

// print the first 5 values
x.printLines(5);

// print all values of the variable
x.printLines();

// print a summary of the content of the variable
x.printSummary();<jupyter_output>[0]                0.0 [3] 1.7320508075688772 
[1]                1.0 [4]                2.0 
[2] 1.4142135623730951 

                                                                        
 [0]                0.0  [5]   2.23606797749979 [10] 3.1622776601683795 
 [1]                1.0  [6]  2.449489742783178 [11]    3.3166247903554 
 [2] 1.4142135623730951  [7] 2.6457513110645907 [12] 3.4641016151377544 
 [3] 1.7320508075688772  [8] 2.8284271247461903 [13]  3.605551275463989 
 [4]                2.0  [9]                3.0 [14] 3.7416573867739413 

> printSummary(var: x)
name: x
type: DOUBLE
rows: 30, complete: 30, missing: 0
   Min. : 0.000
1st Qu. : 2.691
 Median : 3.807
   Mean : 3.554
2nd Qu. : 4.663
   Max. : 5.385
{'fraction_non_alphanumeric': 0.07243589743589744, 'fraction_numerical': 0.04661172161172161, 'mean_word_length': 3.9393939393939394, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '570584', 'license': 'Apache-2.0', 'n_tokens_mistral': 3183, 'n_tokens_neox': 2844, 'n_words': 1469, 'path': 'notebooks/manual/1. Variables, frames and data manipulation.ipynb', 'repo_name': 'iromeo/rapaio', 'size': '10920'}