<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>

<head>
<title>Neko Language Specification</title>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-15"/>
<link rel="StyleSheet" href="style.css" type="text/css"/>
</head>

<body>

<center>
<table class="main">
<tr class="main">
<td class="main">

<h1>Neko Language Specification</h1>

<!-- ****************************************************************************** -->

<a name="summary"></a>
<h2>Summary</h2>

<ul>
	<li><a href="#syntax">Syntax</a></li>
	<li><a href="#values">Values</a></li>
	<li><a href="#execution">Execution Flow</a></li>
	<li><a href="#variables">Variables</a></li>
	<li><a href="#operations">Basic Type Operations</a>
		<ul>
			<li><a href="#arithm">Arithmetic Operations</a></li>
			<li><a href="#bitwise">Bitwise Operations</a></li>
			<li><a href="#boolops">Boolean Operations</a></li>
			<li><a href="#compares">Equality and Comparisons</a></li>
			<li><a href="#converts">Conversions</a></li>
			<li><a href="#optim">Optimized Operations</a></li>
		</ul>
	</li>
	<li><a href="#rtti">Runtime Type Information</a></li>
	<li><a href="#functions">Functions and Function Calls</a></li>
	<li><a href="#objects">Objects</a></li>
	<li><a href="#arrays">Arrays</a></li>
	<li><a href="#strings">Strings</a></li>
	<li><a href="#exceptions">Exceptions</a></li>
	<li><a href="#credits">Credits</a></li>
</ul>

<!-- ****************************************************************************** -->

<a name="syntax"></a>
<h2>Syntax</h2>

<p>
Neko language syntax have been created with "easy to parse" and "easy to generate" in mind. It is not especially designed to be written by a programmer but more to be generated from a higher level language. For example one could write easily a PHP-to-Neko or a Java-to-Neko that would generate equivalent code but using Neko syntax and semantics instead of PHP or Java ones.
</p>

<p>
In particular, there are not multiple levels of expression, as in C. Every statement is also an expression, and thus enable some constructs that are not possible in other languages (for example : <code>return if(x) { ... } else { ... }</code>). This make the generation to Neko from functional languages more easy.
</p>

<p>
The syntax is parsed using a left-to-right LL(1) parser. That means that after reading a token we have enough information to know which expresion it will produce. This enable very lightweight parser which is easy to improve without creating ambiguities. Here's an Abstract Syntax Tree description of the language syntax, with the additionnal constraint that a program must be terminated by EOF :
</p>

<pre>
	    <i>program</i> :=
		| <i>expr</i> <i>program</i>
		| <b>SEMICOLON</b> <i>program</i>
		| &epsilon;

	    <i>ident</i> :=
		| [a-zA-Z_@] [a-zA-Z0-9_@:]*

	    <i>binop</i> :=
		| [!=*/&lt;&gt;&amp;|^%+:-]+

	    <i>value</i> :=
		| [0-9]+
		| 0x[0-9A-Fa-f]+
		| [0-9]+ <b>DOT</b> [0-9]*
		| <b>DOT</b> [0-9]+
		| <b>DOUBLEQUOTE</b> <i>characters</i> <b>DOUBLEQUOTE</b>
		| <b>DOLLAR</b> <i>ident</i>
		| <b>QUOTE</b> <i>ident</i>
		| <b>true</b>
		| <b>false</b>
		| <b>null</b>
		| <b>this</b>
		| <i>ident</i>

	    <i>expr</i> :=
		| <i>value</i>
		| <b>{</b> <i>program</i> <b>}</b>
		| <i>expr</i> <b>DOT</b> <i>ident</i>
		| <i>expr</i> <b>(</b> <i>parameters</i> <b>)</b>
		| <i>expr</i> <b>[</b> <i>expr</i> <b>]</b>
		| <i>expr</i> <i>binop</i> <i>expr</i>
		| <b>(</b> <i>expr</i> <b>)</b>
		| <b>var</b> <i>variables</i>
		| <b>while</b> <i>expr</i> <i>expr</i>
		| <b>do</b> <i>expr</i> <b>while</b> <i>expr</i>
		| <b>if</b> <i>expr</i> <i>expr</i> [<b>else</b> <i>expr</i>]
		| <b>try</b> <i>expr</i> <b>catch</b> <i>ident</i> <i>expr</i>
		| <b>function (</b> <i>parameters-names</i> <b>)</b> <i>expr</i>
		| <b>return</b> [<i>expr</i> | <b>SEMICOLON</b>]
		| <b>break</b> [<i>expr</i> | <b>SEMICOLON</b>]
		| <b>continue</b>

	    <i>variables</i> :=
		| <i>ident</i> [<b>=</b> <i>expr</i>] <i>variables</i>
		| <b>COMMA</b> <i>variables</i>
		| &epsilon;

	    <i>parameters</i> :=
		| <i>expr</i> <i>parameters</i>
		| <b>COMMA</b> <i>parameters</i>
		| &epsilon;

	    <i>parameters-names</i> :=
		| <i>ident</i> <i>parameters-names</i>
		| <b>COMMA</b> <i>parameters-names</i>
		| &epsilon;

	    <i>cases</i> :=
		| <i>expr</i> <b>=></b> <i>expr</i> <i>cases</i>
		| <b>default</b> <b>=></b> <i>expr</i> <i>cases</i>
		| &epsilon;

</pre>

<p>
	<i>Random notes : </i>
</p>

<ul>
<li><code>continue</code> and <code>break</code> outside a while are not specified.</li>
<li><code>return</code> in the toplevel (outside a function) is an accepted way of interupting module execution without error.</li>
<li>There are some few ambiguous cases when two expressions follows each others (in while and if). If the second expression is inside parenthesis, it will be parsed as a call of first expression, while such a representation e1 (e2) exists in the AST (the semicolons are optional).</li>
</ul>

<!-- ****************************************************************************** -->

<a name="values"></a>
<h2>Values</h2>

<p>
A value in Neko can be one of the following :
</p>

<ul>
<li><b>Null :</b> the special value <code>null</code> is used for uninitialized variables as well as programmer/langage specific coding techniques.</li>
<li><b>Integer :</b> integers can be represented in either decimal form (such as <code>12345</code> or <code>-12</code>) or hexadecimal (<code>0x1A2B3C4D</code>).</li>
<li><b>Floating point :</b> floating point values are represented using a dot (such as <code>12.345</code> or <code>-0.123</code>)</li>
<li><b>Boolean :</b> the two booleans are available as specific lowercased identifiers : <code>true</code> and <code>false</code>.</li>
<li><a href="#strings"><b>String</b></a> : strings are double quoted (such as <code>"foo"</code>, or <code>"hello,\nworld !"</code>, or <code>"My name is \"Bond\\<NAME>\"."</code>). Neko strings are mutable, that means that you can modify them.</li>
<li><a href="#arrays"><b>Array</b></a> : arrays are indexed table of values, starting at index 0, they provide fast random access.</li>
<li><a href="#objects"><b>Object</b></a> : an object is an association table, which bind an ident or a string to a value, see how you can create and manage objects later.</li>
<li><a href="#functions"><b>Function</b></a> : a function is also a value in Neko, and can then be stored into any variable.</li>
<li><b>Abstract</b> : an abstract value is some C data that cannot be accessed from a Neko program.</li>
</ul>

<p>
Some precisions :
</p>

<ul>
<li>integers are 31 bits for virtual machine performances reasons, an access to full 32 bits integers will be made possible soon.</li>
<li>floating points are 64 bits double precision floating points values.</li>
<li>strings are 8-bit binary values. A string can contain \0 characters, the string length is determined by the amount of data available and not by the number of characters before the first \0. Maximum string size might depend of virtual machine specification, but should be at least 65535 bytes.</li>
</ul>

<!-- ****************************************************************************** -->

<a name="execution"></a>
<h2>Execution Flow</h2>

<p>
Here's some explanation on how each expression is evaluated :
</p>

<h3>
Values :
</h3>

<ul>
<li><code>[0-9]+ | 0x[0-9A-Fa-f]+</code> : evaluate to the corresponding integer value</li>
<li><code>[0-9]+ <b>DOT</b> [0-9]* | <b>DOT</b> [0-9]+</code> : evaluate to the corresponding floating point value.</li>
<li><code><b>DOUBLEQUOTE</b> <i>characters</i> <b>DOUBLEQUOTE</b></code> : evaluate to the corresponding string. Escaped characters are simular to the C language.</li>
<li><code><b>DOLLAR</b> <i>ident</i></code> : identifiers prefixed with a dollar are primitives. They enable you to call some compiler constructors or optimized calls (see Primitives).</li>
<li><code><b>true</b> | <b>false</b></code> : evaluate to the corresponding boolean.</li>
<li><code><b>null</b></code> : evaluate to the null value.</li>
<li><code><b>this</b></code> : evaluate to the local object value (see later for objects).</li>
<li><code><i>ident</i></code> : evaluate to the value currently bound to this variable name.</li>
</ul>

<h3>Expressions</h3>
<p>
Before evaluating any expression, all sub-expressions are evaluated in an unspecified order. the "v" here are showing the values returned by evaluation of sub-expressions.
</p>

<ul>
<li><code><b>{</b> <i>v1, v2, .... vk</i> <b>}</b></code> : the evaluation order is specified here, it follows the expressions declarations. The last value <code>vk</code> is returned, or <code>null</code> if <code>program</code> does not contain any expression.</li>
<li><code><i>v</i> <b>DOT</b> <i>ident</i></code> : <code>v</code> is accessed as an object using <code>ident</code> as key (see Objects).</li>
<li><code><i>v</i> <b>(</b> <i>v1, v2, ... vk</i> <b>)</b></code> : the function <code>v</code> is called with the parameters <code>v1, v2... vk</code> (see Function Calls)</li>
<li><code><i>v1</i> <b>[</b> <i>v2</i> <b>]</b></code> : <code>v1</code> is accessed as an array using <code>v2</code> as key (see Arrays)</li>
<li><code><i>v1</i> <i>binop</i> <i>v2</i></code> : calculate v1 op v2 (see Operations).</li>
<li><code><i>expr</i> <i>assign-binop</i> <i>v</i></code> : this is a special case, when operation is an assignation (see Operations).</li>
<li><code><b>(</b> <i>v</i> <b>)</b></code> : evaluate to <code>v</code>.</li>
<li><code><b>var</b> <i>i1 = v1, i2 = v2, .... ik = vk</i></code> : each variable <code>i</code> is set to the corresponding value <code>v</code> or to <code>null</code> if no initialization expression is provided.</li>
<li><code><b>while</b> ....  | <b>do</b> ... <b>while</b> ...</code> : classic while-loop, the value returned is either the value returned by a <code>break</code> inside the while or unspecified if the loop ends without break.</li>
<li><code><b>if</b> <i>v1</i> <i>e1</i></code> : if v1 is the boolean <code>true</code>, then <code>e1</code> is evaluated and its value returned, else the evaluated value is unspecified.</li>
<li><code><b>if</b> <i>v1</i> <i>e1</i> <b>else</b> <i>e2</i></code> : if v1 is the boolean <code>true</code>, then <code>e1</code> is evaluated and its value returned else <code>e2</code> is evaluated and its value returned.</li>
<li><code><b>try</b> <i>e1</i> <b>catch</b> <i>i</i> <i>e2</i></code> : evaluate <code>e1</code> and returns the evaluated value. If an exception is raised, then evaluate <code>e2</code> and returns evaluated value with local <code>i</code> variable being set to the raised exception value (see Exceptions).</li>
<li><code><b>function (</b> <i>parameters-names</i> <b>)</b> <i>expr</i></code> : evaluate to the corresponding function.</li>
<li><code><b>return</b>;</code> : exit current function with unspecified return value.</li>
<li><code><b>return</b> v</code> : exit current function and return value <code>v</code>.</li>
<li><code><b>break</b>;</code> : exit current while loop with unspecified return value.</li>
<li><code><b>break</b> v</code> : exit current while loop and return value <code>v</code>.</li>
<li><code><b>continue</b></code> : continue execution to next while loop step.</li>
</ul>

<p>
	Please note that <code>if</code> and <code>while</code> conditions only sucess for the boolean <code>true</code> and for no other value. You might need to add a <code>$istrue</code> code before each expression in order to convert the expression result into a boolean.
</p>

<!-- ****************************************************************************** -->

<a name="variables"></a>
<h2>Variables</h2>

<p>
When an identifier is found, it must be resolved to a given context holding the variable value. The first context searched for is the local context.
</p>

<h3>Local Context</h3>

<p>
The local context contains all variable defined with the "var" keyword in previous syntaxical blocks. In the following sample, <code>x</code> resolve to the value 3 :
</p>

<pre>
	var x = 3;
	$print(x);
</pre>

<p>
A "var" declaration is only local to the curly-braced block it has been declared in. For example :
</p>

<pre>
	var x = 3;
	// x defined
	if( ... ) {
		// x defined
		var y;
		// x and y defined;
	}
	// x defined
</pre>

<p>
Same variable name can be reused in same block or in another block. It will hide or erase previous value :
</p>

<pre>
	var x = 1;
	$print(x); // print 1
	var x = 3;
	$print(x); // print 3
	if( ... ) {
		var x = "neko";
		$print(x); // print "neko"
	}
	$print(x); // print 3
</pre>

<p>
Function parameters are also local variables, they are defined into the whole function :
</p>

<pre>
	var x = 3;
	f = function(x) {
		$print(x);
	}
	f("neko"); // print "neko"
</pre>

<p>
Since contexts are resolved at purely syntactic level, local variables does not depends on current call-stack, and you cannot access variables defined outside current local context.
</p>

<pre>
	f = function() {
		$print(x);
	}
	...
	var x = 3;
	f(); // null or error !
</pre>

<h3>Function Environment</h3>

<p>
Local variables can be used inside functions if they're accessible at the time the function is declared. In this case, the value of the variable is a copy of its value at the time the function was defined :
</p>

<pre>
	var x = 3;
	f = function() {
		$print(x);
	}
	x = 4;
	f(); // print 3
</pre>

<p>
Such variables are called environment variables because they're no longer part of the local context but of the function "environment" context. A function can still modify an environment variable but this will not modify the original variable reference :
</p>

<pre>
	var x = 3;
	f = function() {
		$print(x);
		x = x + 1;
	}
	x = 50;
	f(); // print 3
	f(); // print 4
	$print(x); // print 50
</pre>

<p>
Please note also that each function instance have its own environment :
</p>

<pre>
	gen = function() {
		var i = 0;
		return function() { $print(i); i = i + 1; };
	}
	f1 = gen();
	f2 = gen();
	f1(); // print 0
	f1(); // print 1
	f2(); // print 0
	f1(); // print 2
	...
</pre>

<h3>Global Context</h3>

<p>
When a variable is not found in the local context or in the local function environment, it is a global. A global can be accessed through the whole file, it is shared among all code.
</p>

<pre>
	f = function() {
		$print(x);
		x = x + 1;
	}
	x = 0;
	f(); // print 0
	f(); // print 1
	$print(x); // print 2
</pre>

<p>
It might be useful to add primitives that can directly access the global context.
</p>

<!-- ****************************************************************************** -->

<a name="operations"></a>
<h2>Basic Types Operations</h2>

<p>
Basic types are numbers (int and float), booleans (bool), the null value, strings, objects, arrays and functions. There are several operations available to use them, see the following tables. On the row is first operand type and on the columns is second operand type. The result is either the type of the returned value, or "concat" if we use string concatenation (in that case the two values are converted to strings and then concatened together). The "-" means that the operation is invalid and will return <code>null</code>.
</p>

<a name="arithm"></a>
<h3>Arithmetic operations</h3>

<p>
Operation add ( + ) :
</p>

<table>
<tr>
	<th> + </th>
	<th>null</th>
	<th>int</th>
	<th>float</th>
	<th>string</th>
	<th>bool</th>
	<th>object</th>
	<th>array</th>
	<th>function</th>
</tr>
<tr>
	<th>null</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__preadd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>int</th>
	<td>-</td>
	<td>int</td>
	<td>float</td>
	<td>concat</td>
	<td>-</td>
	<td>__preadd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>float</th>
	<td>-</td>
	<td>float</td>
	<td>float</td>
	<td>concat</td>
	<td>-</td>
	<td>__preadd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>string</th>
	<td>concat</td>
	<td>concat</td>
	<td>concat</td>
	<td>concat</td>
	<td>concat</td>
	<td>__preadd</td>
	<td>concat</td>
	<td>concat</td>
</tr>
<tr>
	<th>bool</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__preadd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>object</th>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
</tr>
<tr>
	<th>array</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__preadd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>function</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__preadd</td>
	<td>-</td>
	<td>-</td>
</tr>
</table>

<p>
	Addition can be overridden for objects : <code>a + b</code> will call <code>a.__add(b)</code> if <code>a</code> is an object, or <code>b.__preadd(a)</code> if <code>b</code> is an object.
</p>

<p>
Operations substract ( - ) divide ( / ) multiply ( * ) and modulo ( % ) :
</p>


<table>
<tr>
	<th> - / * % </th>
	<th>int</th>
	<th>float</th>
</tr>
<tr>
	<th>int</th>
	<td>int (float for /)</td>
	<td>float</td>
</tr>
<tr>
	<th>float</th>
	<td>float</td>
	<td>float</td>
</tr>
</table>

<p>
	Please note that unlike some languages, the divide operation between two integers does returns a float. You can use the <code>$idiv</code> primitive that performs integer division.
</p>

<p>
	Dividing or doing modulo of one integer by the integer zero returns <code>null</code>. For floats, it's hardware dependant and usually return the float +infinity. You can test it using the primitive <code>$isinfinite</code>, and there is also <code>$isnan</code> for testing NaN :
</p>

<pre>
	$print($isinfinite(1/0)); // prints true
	$print($isnan(0/0)); // prints true
</pre>

<p>
	Please note also that overflow on integer operations does not convert them to floats and are not catchable errors. If you want to control overflow you can define your own functions for operations or use floats everywhere.
</p>

<a name="bitwise"></a>
<h3>Bitwise operations</h3>

<p>
	The following operations are available for integers only, the perform on a bit basis. Please remember that for performances reasons, Neko integers are signed and 31 bits only, so there is only 30 bits of "unsigned" part :
</p>

<ul>
	<li><code>&lt;&lt;</code> left bit shift</li>
	<li><code>&gt;&gt;</code> right bit shift</li><commit_msg>fixes about overriding.<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>

<head>
<title>Neko Language Specification</title>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-15"/>
<link rel="StyleSheet" href="style.css" type="text/css"/>
</head>

<body>

<center>
<table class="main">
<tr class="main">
<td class="main">

<h1>Neko Language Specification</h1>

<!-- ****************************************************************************** -->

<a name="summary"></a>
<h2>Summary</h2>

<ul>
	<li><a href="#syntax">Syntax</a></li>
	<li><a href="#values">Values</a></li>
	<li><a href="#execution">Execution Flow</a></li>
	<li><a href="#variables">Variables</a></li>
	<li><a href="#operations">Basic Type Operations</a>
		<ul>
			<li><a href="#arithm">Arithmetic Operations</a></li>
			<li><a href="#bitwise">Bitwise Operations</a></li>
			<li><a href="#boolops">Boolean Operations</a></li>
			<li><a href="#compares">Equality and Comparisons</a></li>
			<li><a href="#converts">Conversions</a></li>
			<li><a href="#optim">Optimized Operations</a></li>
		</ul>
	</li>
	<li><a href="#rtti">Runtime Type Information</a></li>
	<li><a href="#functions">Functions and Function Calls</a></li>
	<li><a href="#objects">Objects</a></li>
	<li><a href="#arrays">Arrays</a></li>
	<li><a href="#strings">Strings</a></li>
	<li><a href="#exceptions">Exceptions</a></li>
	<li><a href="#credits">Credits</a></li>
</ul>

<!-- ****************************************************************************** -->

<a name="syntax"></a>
<h2>Syntax</h2>

<p>
Neko language syntax have been created with "easy to parse" and "easy to generate" in mind. It is not especially designed to be written by a programmer but more to be generated from a higher level language. For example one could write easily a PHP-to-Neko or a Java-to-Neko that would generate equivalent code but using Neko syntax and semantics instead of PHP or Java ones.
</p>

<p>
In particular, there are not multiple levels of expression, as in C. Every statement is also an expression, and thus enable some constructs that are not possible in other languages (for example : <code>return if(x) { ... } else { ... }</code>). This make the generation to Neko from functional languages more easy.
</p>

<p>
The syntax is parsed using a left-to-right LL(1) parser. That means that after reading a token we have enough information to know which expresion it will produce. This enable very lightweight parser which is easy to improve without creating ambiguities. Here's an Abstract Syntax Tree description of the language syntax, with the additionnal constraint that a program must be terminated by EOF :
</p>

<pre>
	    <i>program</i> :=
		| <i>expr</i> <i>program</i>
		| <b>SEMICOLON</b> <i>program</i>
		| &epsilon;

	    <i>ident</i> :=
		| [a-zA-Z_@] [a-zA-Z0-9_@:]*

	    <i>binop</i> :=
		| [!=*/&lt;&gt;&amp;|^%+:-]+

	    <i>value</i> :=
		| [0-9]+
		| 0x[0-9A-Fa-f]+
		| [0-9]+ <b>DOT</b> [0-9]*
		| <b>DOT</b> [0-9]+
		| <b>DOUBLEQUOTE</b> <i>characters</i> <b>DOUBLEQUOTE</b>
		| <b>DOLLAR</b> <i>ident</i>
		| <b>QUOTE</b> <i>ident</i>
		| <b>true</b>
		| <b>false</b>
		| <b>null</b>
		| <b>this</b>
		| <i>ident</i>

	    <i>expr</i> :=
		| <i>value</i>
		| <b>{</b> <i>program</i> <b>}</b>
		| <i>expr</i> <b>DOT</b> <i>ident</i>
		| <i>expr</i> <b>(</b> <i>parameters</i> <b>)</b>
		| <i>expr</i> <b>[</b> <i>expr</i> <b>]</b>
		| <i>expr</i> <i>binop</i> <i>expr</i>
		| <b>(</b> <i>expr</i> <b>)</b>
		| <b>var</b> <i>variables</i>
		| <b>while</b> <i>expr</i> <i>expr</i>
		| <b>do</b> <i>expr</i> <b>while</b> <i>expr</i>
		| <b>if</b> <i>expr</i> <i>expr</i> [<b>else</b> <i>expr</i>]
		| <b>try</b> <i>expr</i> <b>catch</b> <i>ident</i> <i>expr</i>
		| <b>function (</b> <i>parameters-names</i> <b>)</b> <i>expr</i>
		| <b>return</b> [<i>expr</i> | <b>SEMICOLON</b>]
		| <b>break</b> [<i>expr</i> | <b>SEMICOLON</b>]
		| <b>continue</b>

	    <i>variables</i> :=
		| <i>ident</i> [<b>=</b> <i>expr</i>] <i>variables</i>
		| <b>COMMA</b> <i>variables</i>
		| &epsilon;

	    <i>parameters</i> :=
		| <i>expr</i> <i>parameters</i>
		| <b>COMMA</b> <i>parameters</i>
		| &epsilon;

	    <i>parameters-names</i> :=
		| <i>ident</i> <i>parameters-names</i>
		| <b>COMMA</b> <i>parameters-names</i>
		| &epsilon;

	    <i>cases</i> :=
		| <i>expr</i> <b>=></b> <i>expr</i> <i>cases</i>
		| <b>default</b> <b>=></b> <i>expr</i> <i>cases</i>
		| &epsilon;

</pre>

<p>
	<i>Random notes : </i>
</p>

<ul>
<li><code>continue</code> and <code>break</code> outside a while are not specified.</li>
<li><code>return</code> in the toplevel (outside a function) is an accepted way of interupting module execution without error.</li>
<li>There are some few ambiguous cases when two expressions follows each others (in while and if). If the second expression is inside parenthesis, it will be parsed as a call of first expression, while such a representation e1 (e2) exists in the AST (the semicolons are optional).</li>
</ul>

<!-- ****************************************************************************** -->

<a name="values"></a>
<h2>Values</h2>

<p>
A value in Neko can be one of the following :
</p>

<ul>
<li><b>Null :</b> the special value <code>null</code> is used for uninitialized variables as well as programmer/langage specific coding techniques.</li>
<li><b>Integer :</b> integers can be represented in either decimal form (such as <code>12345</code> or <code>-12</code>) or hexadecimal (<code>0x1A2B3C4D</code>).</li>
<li><b>Floating point :</b> floating point values are represented using a dot (such as <code>12.345</code> or <code>-0.123</code>)</li>
<li><b>Boolean :</b> the two booleans are available as specific lowercased identifiers : <code>true</code> and <code>false</code>.</li>
<li><a href="#strings"><b>String</b></a> : strings are double quoted (such as <code>"foo"</code>, or <code>"hello,\nworld !"</code>, or <code>"My name is \"Bond\\<NAME>\"."</code>). Neko strings are mutable, that means that you can modify them.</li>
<li><a href="#arrays"><b>Array</b></a> : arrays are indexed table of values, starting at index 0, they provide fast random access.</li>
<li><a href="#objects"><b>Object</b></a> : an object is an association table, which bind an ident or a string to a value, see how you can create and manage objects later.</li>
<li><a href="#functions"><b>Function</b></a> : a function is also a value in Neko, and can then be stored into any variable.</li>
<li><b>Abstract</b> : an abstract value is some C data that cannot be accessed from a Neko program.</li>
</ul>

<p>
Some precisions :
</p>

<ul>
<li>integers are 31 bits for virtual machine performances reasons, an access to full 32 bits integers will be made possible soon.</li>
<li>floating points are 64 bits double precision floating points values.</li>
<li>strings are 8-bit binary values. A string can contain \0 characters, the string length is determined by the amount of data available and not by the number of characters before the first \0. Maximum string size might depend of virtual machine specification, but should be at least 65535 bytes.</li>
</ul>

<!-- ****************************************************************************** -->

<a name="execution"></a>
<h2>Execution Flow</h2>

<p>
Here's some explanation on how each expression is evaluated :
</p>

<h3>
Values :
</h3>

<ul>
<li><code>[0-9]+ | 0x[0-9A-Fa-f]+</code> : evaluate to the corresponding integer value</li>
<li><code>[0-9]+ <b>DOT</b> [0-9]* | <b>DOT</b> [0-9]+</code> : evaluate to the corresponding floating point value.</li>
<li><code><b>DOUBLEQUOTE</b> <i>characters</i> <b>DOUBLEQUOTE</b></code> : evaluate to the corresponding string. Escaped characters are simular to the C language.</li>
<li><code><b>DOLLAR</b> <i>ident</i></code> : identifiers prefixed with a dollar are primitives. They enable you to call some compiler constructors or optimized calls (see Primitives).</li>
<li><code><b>true</b> | <b>false</b></code> : evaluate to the corresponding boolean.</li>
<li><code><b>null</b></code> : evaluate to the null value.</li>
<li><code><b>this</b></code> : evaluate to the local object value (see later for objects).</li>
<li><code><i>ident</i></code> : evaluate to the value currently bound to this variable name.</li>
</ul>

<h3>Expressions</h3>
<p>
Before evaluating any expression, all sub-expressions are evaluated in an unspecified order. the "v" here are showing the values returned by evaluation of sub-expressions.
</p>

<ul>
<li><code><b>{</b> <i>v1, v2, .... vk</i> <b>}</b></code> : the evaluation order is specified here, it follows the expressions declarations. The last value <code>vk</code> is returned, or <code>null</code> if <code>program</code> does not contain any expression.</li>
<li><code><i>v</i> <b>DOT</b> <i>ident</i></code> : <code>v</code> is accessed as an object using <code>ident</code> as key (see Objects).</li>
<li><code><i>v</i> <b>(</b> <i>v1, v2, ... vk</i> <b>)</b></code> : the function <code>v</code> is called with the parameters <code>v1, v2... vk</code> (see Function Calls)</li>
<li><code><i>v1</i> <b>[</b> <i>v2</i> <b>]</b></code> : <code>v1</code> is accessed as an array using <code>v2</code> as key (see Arrays)</li>
<li><code><i>v1</i> <i>binop</i> <i>v2</i></code> : calculate v1 op v2 (see Operations).</li>
<li><code><i>expr</i> <i>assign-binop</i> <i>v</i></code> : this is a special case, when operation is an assignation (see Operations).</li>
<li><code><b>(</b> <i>v</i> <b>)</b></code> : evaluate to <code>v</code>.</li>
<li><code><b>var</b> <i>i1 = v1, i2 = v2, .... ik = vk</i></code> : each variable <code>i</code> is set to the corresponding value <code>v</code> or to <code>null</code> if no initialization expression is provided.</li>
<li><code><b>while</b> ....  | <b>do</b> ... <b>while</b> ...</code> : classic while-loop, the value returned is either the value returned by a <code>break</code> inside the while or unspecified if the loop ends without break.</li>
<li><code><b>if</b> <i>v1</i> <i>e1</i></code> : if v1 is the boolean <code>true</code>, then <code>e1</code> is evaluated and its value returned, else the evaluated value is unspecified.</li>
<li><code><b>if</b> <i>v1</i> <i>e1</i> <b>else</b> <i>e2</i></code> : if v1 is the boolean <code>true</code>, then <code>e1</code> is evaluated and its value returned else <code>e2</code> is evaluated and its value returned.</li>
<li><code><b>try</b> <i>e1</i> <b>catch</b> <i>i</i> <i>e2</i></code> : evaluate <code>e1</code> and returns the evaluated value. If an exception is raised, then evaluate <code>e2</code> and returns evaluated value with local <code>i</code> variable being set to the raised exception value (see Exceptions).</li>
<li><code><b>function (</b> <i>parameters-names</i> <b>)</b> <i>expr</i></code> : evaluate to the corresponding function.</li>
<li><code><b>return</b>;</code> : exit current function with unspecified return value.</li>
<li><code><b>return</b> v</code> : exit current function and return value <code>v</code>.</li>
<li><code><b>break</b>;</code> : exit current while loop with unspecified return value.</li>
<li><code><b>break</b> v</code> : exit current while loop and return value <code>v</code>.</li>
<li><code><b>continue</b></code> : continue execution to next while loop step.</li>
</ul>

<p>
	Please note that <code>if</code> and <code>while</code> conditions only sucess for the boolean <code>true</code> and for no other value. You might need to add a <code>$istrue</code> code before each expression in order to convert the expression result into a boolean.
</p>

<!-- ****************************************************************************** -->

<a name="variables"></a>
<h2>Variables</h2>

<p>
When an identifier is found, it must be resolved to a given context holding the variable value. The first context searched for is the local context.
</p>

<h3>Local Context</h3>

<p>
The local context contains all variable defined with the "var" keyword in previous syntaxical blocks. In the following sample, <code>x</code> resolve to the value 3 :
</p>

<pre>
	var x = 3;
	$print(x);
</pre>

<p>
A "var" declaration is only local to the curly-braced block it has been declared in. For example :
</p>

<pre>
	var x = 3;
	// x defined
	if( ... ) {
		// x defined
		var y;
		// x and y defined;
	}
	// x defined
</pre>

<p>
Same variable name can be reused in same block or in another block. It will hide or erase previous value :
</p>

<pre>
	var x = 1;
	$print(x); // print 1
	var x = 3;
	$print(x); // print 3
	if( ... ) {
		var x = "neko";
		$print(x); // print "neko"
	}
	$print(x); // print 3
</pre>

<p>
Function parameters are also local variables, they are defined into the whole function :
</p>

<pre>
	var x = 3;
	f = function(x) {
		$print(x);
	}
	f("neko"); // print "neko"
</pre>

<p>
Since contexts are resolved at purely syntactic level, local variables does not depends on current call-stack, and you cannot access variables defined outside current local context.
</p>

<pre>
	f = function() {
		$print(x);
	}
	...
	var x = 3;
	f(); // null or error !
</pre>

<h3>Function Environment</h3>

<p>
Local variables can be used inside functions if they're accessible at the time the function is declared. In this case, the value of the variable is a copy of its value at the time the function was defined :
</p>

<pre>
	var x = 3;
	f = function() {
		$print(x);
	}
	x = 4;
	f(); // print 3
</pre>

<p>
Such variables are called environment variables because they're no longer part of the local context but of the function "environment" context. A function can still modify an environment variable but this will not modify the original variable reference :
</p>

<pre>
	var x = 3;
	f = function() {
		$print(x);
		x = x + 1;
	}
	x = 50;
	f(); // print 3
	f(); // print 4
	$print(x); // print 50
</pre>

<p>
Please note also that each function instance have its own environment :
</p>

<pre>
	gen = function() {
		var i = 0;
		return function() { $print(i); i = i + 1; };
	}
	f1 = gen();
	f2 = gen();
	f1(); // print 0
	f1(); // print 1
	f2(); // print 0
	f1(); // print 2
	...
</pre>

<h3>Global Context</h3>

<p>
When a variable is not found in the local context or in the local function environment, it is a global. A global can be accessed through the whole file, it is shared among all code.
</p>

<pre>
	f = function() {
		$print(x);
		x = x + 1;
	}
	x = 0;
	f(); // print 0
	f(); // print 1
	$print(x); // print 2
</pre>

<p>
It might be useful to add primitives that can directly access the global context.
</p>

<!-- ****************************************************************************** -->

<a name="operations"></a>
<h2>Basic Types Operations</h2>

<p>
Basic types are numbers (int and float), booleans (bool), the null value, strings, objects, arrays and functions. There are several operations available to use them, see the following tables. On the row is first operand type and on the columns is second operand type. The result is either the type of the returned value, or "concat" if we use string concatenation (in that case the two values are converted to strings and then concatened together). The "-" means that the operation is invalid and will return <code>null</code>.
</p>

<a name="arithm"></a>
<h3>Arithmetic operations</h3>

<p>
Operation add ( + ) :
</p>

<table>
<tr>
	<th> + </th>
	<th>null</th>
	<th>int</th>
	<th>float</th>
	<th>string</th>
	<th>bool</th>
	<th>object</th>
	<th>array</th>
	<th>function</th>
</tr>
<tr>
	<th>null</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__radd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>int</th>
	<td>-</td>
	<td>int</td>
	<td>float</td>
	<td>concat</td>
	<td>-</td>
	<td>__radd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>float</th>
	<td>-</td>
	<td>float</td>
	<td>float</td>
	<td>concat</td>
	<td>-</td>
	<td>__radd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>string</th>
	<td>concat</td>
	<td>concat</td>
	<td>concat</td>
	<td>concat</td>
	<td>concat</td>
	<td>__radd</td>
	<td>concat</td>
	<td>concat</td>
</tr>
<tr>
	<th>bool</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__radd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>object</th>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
	<td>__add</td>
</tr>
<tr>
	<th>array</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__radd</td>
	<td>-</td>
	<td>-</td>
</tr>
<tr>
	<th>function</th>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>concat</td>
	<td>-</td>
	<td>__radd</td>
	<td>-</td>
	<td>-</td>
</tr>
</table>

<p>
	Addition can be overridden for objects : <code>a + b</code> will call <code>a.__add(b)</code> if <code>a</code> is an object, or <code>b.__radd(a)</code> if <code>b</code> is an object.
</p>

<p>
Operations substract ( - ) divide ( / ) multiply ( * ) and modulo ( % ) :
</p>


<table>
<tr>
	<th> - / * % </th>
	<th>int</th>
	<th>float</th>
</tr>
<tr>
	<th>int</th>
	<td>int (float for /)</td>
	<td>float</td>
</tr>
<tr>
	<th>float</th>
	<td>float</td>
	<td>float</td>
</tr>
</table>

<p>
	Please note that unlike some languages, the divide operation between two integers does returns a float. You can use the <code>$idiv</code> primitive that performs integer division.
</p>

<p>
	Dividing or doing modulo of one integer by the integer zero returns <code>null</code>. For floats, it's hardware dependant and usually return the float +infinity. You can test it using the primitive <code>$isinfinite</code>, and there is also <code>$isnan</code> for testing NaN :
</p>

<p>
	Objects does also have theses operations overridable, see the Objects section.
</p>

<pre>
	$print($isinfinite(1/0)); // prints true
	$print($isnan(0/0)); // prints true
</pre>

<p>
	Please note also that overflow on integer operations does not convert them to floats and are not catchable errors. If you want to control overflow you can define your own functions for operations or use floats everywhere.
</p>

<a name="bitwise"></a>
<h3>Bitwise operations</h3>

<p>
	The following operations are available for integers only, the perform on a bit basis. Please remember that for performances reasons, Neko integers are signed and 31 bits only, so there is only 30 bits of "unsigned" part :
</p>

<ul>
	<li><code>&lt;&lt;</code> left bit shift</li>
	<li><code>&gt;&gt;</code> right bit shift</li>
{'fraction_non_alphanumeric': 0.223172249580605, 'fraction_numerical': 0.009794902321554196, 'mean_word_length': 4.115432525951557, 'pattern_counts': {'":': 0, '<': 2083, '<?xml version=': 0, '>': 2087, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5355749', 'n_tokens_mistral': 14131, 'n_tokens_neox': 13501, 'n_words': 4234}