<commit_before># Hail's expression language

Several Hail commands provide the ability to perform a broad array of computations based on data structures exposed to the user.  

**Supported comparisons and transformations:**
 - Global comparisons: `a == b`, `a != b`
 - Boolean comparisons: `a || b`, `a && b`  *NB: Boolean comparisons short circuit.  If `a` is true, `a || b` is `true` without evaluating `b`.  If `a` is missing, `b` is evaluated and the comparison returns `true` if `b` is true, otherwise missing.*
 - Missingness:
   - isMissing: `a.isMissing` -- returns true if `a` is missing
   - isNotMissing: `a.isNotMissing` -- returns true if `a` is defined
   - orElse: `a.OrElse(x)` -- return `a` if `a` is defined, otherwise `x`.  `x` is only evaluated if `a` is NA.
 - Numerical comparisons: `<`, `<=`, `>`, `>=`
 - Numerical conversions: 
   - toDouble: `i.toDouble`
   - toInt: `i.toInt`
   - toFloat: `i.toFloat`
   - toLong: `i.toLong`
 - Numerical transformations:
   - +, -, /, *, %: `a + b - c / d * e % f`
   - abs: `i.abs` -- returns the absolute value of `i`
   - signum: `i.signum` -- returns the sign of `i` (1, 0, or -1)
   - min: `i.min(j)` -- returns the minimum of `i` and `j`
   - max: `i.max(j)` -- returns the maximum of `i` and `j`
 - String operations: 
   - apply: `str(index)` -- returns the character at `index`
   - length: `str.length` -- returns the length of the string
   - concatenate: `str1 + str2` -- returns the two strings joined start-to-end
   - split: `str.split(delimiter)` -- returns an array of strings, split on the given `delimiter` 
   - ~: `regex ~ str` -- returns true if the given regex was found in `str`.  For example, `' "HI" ~ "THIS" '` returns true, and `' "HH" ~ "THIS" '` returns false.
 - String conversions:
   - toInt: `str.toInt`
   - toDouble: `str.toDouble`
   - toLong: `str.toLong`
   - toFloat: `str.toFloat`
 - Array Operations:
   - apply: `arr[index]` -- get a value from the array, or NA if array or index is missing
   - contains: `arr.contains(elem)` -- returns true if the element is contained in the array, otherwise false
   - length: `arr.length` -- returns the length of the array as an integer
   - isEmpty: `arr.isEmpty` -- returns true if the array has length 0
   - mkString: `arr.mkString(sep)` -- returns a string generated by joining elements sequentially, delimited by `sep`
   - find: `arr.find(v => expr)` -- Returns the first non-missing element of `arr` for which `expr` is true.  If no element satisfies the predicate, `find` returns NA.
 - Set Operations: 
   - contains: `set.contains(elem)` -- returns true if the element is contained in the array, otherwise false
   - size: `set.size` -- returns the number of elements in the set as an integer
   - isEmpty: `set.isEmpty` -- returns true if the set contains 0 elements
   - equals: `set1 == set2` -- returns true if both sets contain the same elements
   - plus: `set1 + elem` -- adds `elem` to `set1` and returns the new set
   - minus: `set1 - elem` -- removes `elem` from `set1` (if it is present) and returns the new set
   - union: `set1.union(set2)` -- returns a new set with all elements in `set1` or `set2`
   - intersect: `set1.intersect(set2)` -- returns a new set with all elements in both `set1` and `set2`
   - diff: `set1.diff(set2)` -- returns a new set with all elements not shared between `set1` and `set2`

**Note:**
 - All variables and values are case sensitive
 - Missingness propagates up.  If any element in an expression is missing, the expression will evaluate to missing.  

____

### Examples

#### Filtering

Filtering requires an expression that evaluates to a boolean.

```
filtersamples --keep -c '"PT-1234" ~ s.id'
```


```
filtersamples --keep -c 'sa.qc.callRate > 0.99'
```

In the below expression, we will use a different cutoff for samples with European and non-European ancestry.  This can be done with an if/else statement.

```
filtersamples --keep -c 'if (sa.ancestry == "EUR") sa.qc.nSingleton < 100 else sa.qc.nSingleton < 200'
```

<commit_msg>First round of review complete<commit_after># Hail's expression language

Several Hail commands provide the ability to perform a broad array of computations based on data structures exposed to the user.  

**Supported comparisons and transformations:**
 - Global comparisons: `a == b`, `a != b`
 - Boolean comparisons: `a || b`, `a && b`  *NB: Boolean comparisons short circuit.  If `a` is true, `a || b` is `true` without evaluating `b`.  If `a` is missing, `b` is evaluated and the comparison returns `true` if `b` is true, otherwise missing.*
 - Missingness:
   - isMissing: `a.isMissing` -- returns true if `a` is missing
   - isNotMissing: `a.isNotMissing` -- returns true if `a` is defined
   - orElse: `a.OrElse(x)` -- return `a` if `a` is defined, otherwise `x`.  `x` is only evaluated if `a` is NA.
 - Numerical comparisons: `<`, `<=`, `>`, `>=`
 - Numerical conversions: 
   - toDouble: `i.toDouble`
   - toInt: `i.toInt`
   - toFloat: `i.toFloat`
   - toLong: `i.toLong`
 - Numerical transformations:
   - +, -, /, *, %: `a + b - c / d * e % f`
   - abs: `i.abs` -- returns the absolute value of `i`
   - signum: `i.signum` -- returns the sign of `i` (1, 0, or -1)
   - min: `i.min(j)` -- returns the minimum of `i` and `j`
   - max: `i.max(j)` -- returns the maximum of `i` and `j`
 - String operations: 
   - apply: `str(index)` -- returns the character at `index`
   - length: `str.length` -- returns the length of the string
   - concatenate: `str1 + str2` -- returns the two strings joined start-to-end
   - split: `str.split(delimiter)` -- returns an array of strings, split on the given `delimiter` 
   - ~: `regex ~ str` -- returns true if the given regex was found in `str`.  For example, `' "HI" ~ "THIS" '` returns true, and `' "HH" ~ "THIS" '` returns false.
 - String conversions:
   - toInt: `str.toInt`
   - toDouble: `str.toDouble`
   - toLong: `str.toLong`
   - toFloat: `str.toFloat`
 - Array Operations:
   - apply: `arr[index]` -- get a value from the array, or NA if array or index is missing
   - contains: `arr.contains(elem)` -- returns true if the element is contained in the array, otherwise false
   - length: `arr.length` -- returns the length of the array as an integer
   - isEmpty: `arr.isEmpty` -- returns true if the array has length 0
   - mkString: `arr.mkString(sep)` -- returns a string generated by joining elements sequentially, delimited by `sep`
   - find: `arr.find(v => expr)` -- Returns the first non-missing element of `arr` for which `expr` is true.  If no element satisfies the predicate, `find` returns NA.
 - Set Operations: 
   - contains: `set.contains(elem)` -- returns true if the element is contained in the array, otherwise false
   - size: `set.size` -- returns the number of elements in the set as an integer
   - isEmpty: `set.isEmpty` -- returns true if the set contains 0 elements
   - equals: `set1 == set2` -- returns true if both sets contain the same elements
   - plus: `set1 + elem` -- adds `elem` to `set1` and returns the new set
   - minus: `set1 - elem` -- removes `elem` from `set1` (if it is present) and returns the new set
   - union: `set1.union(set2)` -- returns a new set with all elements in `set1` or `set2`
   - intersect: `set1.intersect(set2)` -- returns a new set with all elements in both `set1` and `set2`
   - diff: `set1.diff(set2)` -- returns a new set with all elements not shared between `set1` and `set2`

**Note:**
 - All variables and values are case sensitive
 - Missingness propagates up.  If any element in an expression is missing, the expression will evaluate to missing.  

____

### Examples

#### Filtering

Filtering requires an expression that evaluates to a boolean.

```
filtersamples --keep -c '"PT-1234" ~ s.id'
```


```
filtersamples --keep -c 'sa.qc.callRate > 0.99'
```

In the below expression, we will use a different cutoff for samples with European and non-European ancestry.  This can be done with an if/else statement.

```
filtersamples --keep -c 'if (sa.ancestry == "EUR") sa.qc.nSingleton < 100 else sa.qc.nSingleton < 200'
```

See documentatation on [exporting to TSV](ExportTSV.md) and [programmatic annotation](ProgrammaticAnnotation.md) for more examples of what Hail's language can do.
{'fraction_non_alphanumeric': 0.13430127041742287, 'fraction_numerical': 0.009195402298850575, 'mean_word_length': 3.1810824481537683, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5424683', 'n_tokens_mistral': 2707, 'n_tokens_neox': 2573, 'n_words': 1084}