<commit_before># Runtime

This document describes the runtime API to manipulate scriban text templating.

Scriban provides a **safe runtime**, meaning it doesn't expose any .NET objects that haven't been made explicitly available to a Template. 

The runtime is composed of two main parts:

- The **parsing/compiler** infrastructure that is responsible for parsing a text template and build a runtime representation of it (we will call this an `Abstract Syntax Tree`)
- The **rendering/evaluation** infrastructure that is responsible to render a compiled template to a string. We will see also that we can evaluate expressions without rendering.

The scriban runtime was designed to provide an easy, powerful and extensible infrastructure. For example, we are making sure that nothing in the runtime is using a static, so that you can correctly overrides all the behaviors of the runtime.

- [Parsing a template](#parsing-a-template)
  - [Parsing modes](#parsing-modes)
- [Rendering a template](#rendering-a-template)
  - [Overview](#overview)
  - [The <code>TemplateContext</code> execution model](#the-templatecontext-execution-model)
  - [The <code>ScriptObject</code>](#the-scriptobject)
  - [The stack of <code>ScriptObject</code>](#the-stack-of-scriptobject)
- [Advanced usages](#advanced-usages)
  - [Member renamer](#member-renamer)
  - [Include and Template Loader](#include-and-itemplateloader)
  - [The Lexer and Parser](#the-lexer-and-parser)
  - [Abstract Syntax Tree](#abstract-syntax-tree)
  - [Extending <code>TemplateContext</code>](#extending-templatecontext)
  - [<code>ScriptObject</code> advanced usages](#scriptobject-advanced-usages)
    - [Advanced custom functions](#advanced-custom-functions)
    - [Hyper custom functions<code>IScriptCustomFunction</code>](#hyper-custom-functionsiscriptcustomfunction)

[:top:](#runtime)
## Parsing a template

The `Scriban.Template` class is a main entry point to easily parse a template and renders it. The action of parsing consist of compiling the template to a faster runtime representation, suitable later for rendering the template.

This class is mostly a user friendly frontend to the underlying classes used to parse a template. See [The Lexer and Parser](#TODO) section for advanded usages.

The `Template.Parse ` method is a convenient method to parse a template from a string and returns the compiled Template:

```c#
var inputTemplateAsText = "This is a {{ name }} template";

// Parse the template
var template = Template.Parse(inputTemplateAsText);

// Check for any errors
if (template.HasErrors)
{
    foreach(var error in template.Messages)
    {
        Console.WriteLine(error);
    }
    return; // or throw...etc.
}
```

The returned `Template` object has the following relevant properties:

- `ScriptPage Page {get;}` that contains the compiled template to a root Abstract Syntax Tree (AST). From this object you can navigate through all the statements parsed from the template if necessary. See the section about the [Abstract Syntax Tree](#abstract-syntax-tree)
- `bool HasErrors {get;}` to check if the parsed template has any errors. In that case, the `ScriptPage Page` property is null.
- `List<LogMessage> Messages {get;}` contains the list of warning and error messages while parsing the template.

If you are using the `Template.Parse` method, it is important to verify `HasErrors` is `false`, otherwise you will get a null `ScriptPage` object from the `Template.Page` property.

The parse method can take an additional argument `sourceFilePath` used when reporting syntax errors, typically used to associate a template file read from the disk or an editor and you want to report the exact error to the user.

```c#
// Parse the template
var template = Template.Parse(File.ReadAllText(filePath), filePath);
```

> Note that the `sourceFilePath` is not used for accessing the disk (it could be a logical path to a zip file, or the name of tab opened in an editor...etc.). It is only a logical name that is used when reporting errors, but also you will see with the include directive and the setup of the [Template Loader](#InsertLinkHere) that this value can be used to perform an include operation in the relative context to the template path being processed.

[:top:](#runtime)
### Parsing modes

By default, when parsing a template, the template is expected to have mixed content of text and scriban code blocks enclosed by `{{` and `}}`. But you can modify the way a template is parsed by passing a `LexerOptions` to the `Template.Parse` method.

The parsing mode is defined by the `LexerOptions.Mode` property which is `ScriptMode.Default` by default (i.e. mixed text and code).

But you can also parse a template that contains directly scripting code (without enclosing `{{` `}}`), in that case, you can use the `ScriptMode.ScriptOnly` mode.

For example illustrate how to use the `ScriptOnly` mode:

```c#
// Create a template in ScriptOnly mode
var lexerOptions = new LexerOptions() { Mode = ScriptMode.ScriptOnly };
// Notice that code is not enclosed by `{{` and `}}`
var template = Template.Parse("y = x + 1; y;", lexerOptions: lexerOptions);
// Renders it with the specified parameter
var result = template.Evaluate(new {x = 10});
// Prints 11
Console.WriteLine(result);
```

> Note: As we will see in the following section about rendering, you can also avoid rendering a script only mode by evaluating the template instead of rendering. 

[:top:](#runtime)
## Rendering a template

### Overview

In order to render a template, you need pass a context for the variables, objects, functions that will be accessed by the template.

In the following examples, we have a variable `name` that is used by the template:

```c#
var inputTemplateAsText = "This is a {{ name }} template";

// Parse the template
var template = Template.Parse(inputTemplateAsText);

// Renders the template with the variable `name` exposed to the template
var result = template.Render(new { name = "Hello World"});

// Prints the result: "This is a Hello World template"
Console.WriteLine(result);
```

As we can see, we are passing an anonymous objects that has this field/property `name` and by calling the Render method, the template is executed with this data model context.

While passing an anonymous object is nice for a hello world example, it is not always enough for more advanced data model scenarios. 

In this case, you want to use more directly the `TemplateContext` (used by the method `Template.Render(object)`) and a `ScriptObject` which are both at the core of scriban rendering architecture to provide more powerful constructions & hooks of the data model exposed (variables but also functions...etc.).

[:top:](#runtime)
### The `TemplateContext` execution model

The `TemplateContext` provides:

- **an execution context** when evaluating a template. The same instance can be used with many different templates, depending on your requirements.
- A **stack of `ScriptObject`** that provides the actual variables/functions accessible to the template, accessible through `Template.PushGlobal(scriptObj)` and `Template.PopGlobal()`. Why a stack and how to use this stack is described below.
- The **text output** when evaluating a template, which is accessible through the `Template.Output` property as a `StringBuilder` but because you can have nested rendering happening, it is possible to use `Template.PushOutput()` and `Template.PopOutput()` to redirect temporarily the output to a new output. This functionality is typically used by the [`capture` statement](language.md#94-capture-variable-end).
- Caching of templates previously loaded by an `include` directive (see [`include` and `ITemplateLoader`](#include-and-itemplateloader) section )
- Various possible overrides to allow fine grained extensibility (evaluation of an expression, conversion to a string, enter/exit/step into a loop...etc.)

Note that a `TemplateContext` is not thread safe, so it is recommended to have one `TemplateContext` per thread.

[:top:](#runtime)
### The `ScriptObject`

The `ScriptObject` is a special implementation of a `Dictionary<string, object>` that runtime properties and functions accessible to a template:

- **Accessing as regular dictionary objects**:

  ```C#<commit_msg>Fix links in runtime.md<commit_after># Runtime

This document describes the runtime API to manipulate scriban text templating.

Scriban provides a **safe runtime**, meaning it doesn't expose any .NET objects that haven't been made explicitly available to a Template. 

The runtime is composed of two main parts:

- The **parsing/compiler** infrastructure that is responsible for parsing a text template and build a runtime representation of it (we will call this an `Abstract Syntax Tree`)
- The **rendering/evaluation** infrastructure that is responsible to render a compiled template to a string. We will see also that we can evaluate expressions without rendering.

The scriban runtime was designed to provide an easy, powerful and extensible infrastructure. For example, we are making sure that nothing in the runtime is using a static, so that you can correctly overrides all the behaviors of the runtime.

- [Parsing a template](#parsing-a-template)
  - [Parsing modes](#parsing-modes)
- [Rendering a template](#rendering-a-template)
  - [Overview](#overview)
  - [The <code>TemplateContext</code> execution model](#the-templatecontext-execution-model)
  - [The <code>ScriptObject</code>](#the-scriptobject)
  - [The stack of <code>ScriptObject</code>](#the-stack-of-scriptobject)
- [Advanced usages](#advanced-usages)
  - [Member renamer](#member-renamer)
  - [Include and Template Loader](#include-and-itemplateloader)
  - [The Lexer and Parser](#the-lexer-and-parser)
  - [Abstract Syntax Tree](#abstract-syntax-tree)
  - [Extending <code>TemplateContext</code>](#extending-templatecontext)
  - [<code>ScriptObject</code> advanced usages](#scriptobject-advanced-usages)
    - [Advanced custom functions](#advanced-custom-functions)
    - [Hyper custom functions<code>IScriptCustomFunction</code>](#hyper-custom-functionsiscriptcustomfunction)

[:top:](#runtime)
## Parsing a template

The `Scriban.Template` class is a main entry point to easily parse a template and renders it. The action of parsing consist of compiling the template to a faster runtime representation, suitable later for rendering the template.

This class is mostly a user friendly frontend to the underlying classes used to parse a template. See [The Lexer and Parser](#TODO) section for advanded usages.

The `Template.Parse ` method is a convenient method to parse a template from a string and returns the compiled Template:

```c#
var inputTemplateAsText = "This is a {{ name }} template";

// Parse the template
var template = Template.Parse(inputTemplateAsText);

// Check for any errors
if (template.HasErrors)
{
    foreach(var error in template.Messages)
    {
        Console.WriteLine(error);
    }
    return; // or throw...etc.
}
```

The returned `Template` object has the following relevant properties:

- `ScriptPage Page {get;}` that contains the compiled template to a root Abstract Syntax Tree (AST). From this object you can navigate through all the statements parsed from the template if necessary. See the section about the [Abstract Syntax Tree](#abstract-syntax-tree)
- `bool HasErrors {get;}` to check if the parsed template has any errors. In that case, the `ScriptPage Page` property is null.
- `List<LogMessage> Messages {get;}` contains the list of warning and error messages while parsing the template.

If you are using the `Template.Parse` method, it is important to verify `HasErrors` is `false`, otherwise you will get a null `ScriptPage` object from the `Template.Page` property.

The parse method can take an additional argument `sourceFilePath` used when reporting syntax errors, typically used to associate a template file read from the disk or an editor and you want to report the exact error to the user.

```c#
// Parse the template
var template = Template.Parse(File.ReadAllText(filePath), filePath);
```

> Note that the `sourceFilePath` is not used for accessing the disk (it could be a logical path to a zip file, or the name of tab opened in an editor...etc.). It is only a logical name that is used when reporting errors, but also you will see with the include directive and the setup of the [Template Loader](#include-and-itemplateloader) that this value can be used to perform an include operation in the relative context to the template path being processed.

[:top:](#runtime)
### Parsing modes

By default, when parsing a template, the template is expected to have mixed content of text and scriban code blocks enclosed by `{{` and `}}`. But you can modify the way a template is parsed by passing a `LexerOptions` to the `Template.Parse` method.

The parsing mode is defined by the `LexerOptions.Mode` property which is `ScriptMode.Default` by default (i.e. mixed text and code).

But you can also parse a template that contains directly scripting code (without enclosing `{{` `}}`), in that case, you can use the `ScriptMode.ScriptOnly` mode.

For example illustrate how to use the `ScriptOnly` mode:

```c#
// Create a template in ScriptOnly mode
var lexerOptions = new LexerOptions() { Mode = ScriptMode.ScriptOnly };
// Notice that code is not enclosed by `{{` and `}}`
var template = Template.Parse("y = x + 1; y;", lexerOptions: lexerOptions);
// Renders it with the specified parameter
var result = template.Evaluate(new {x = 10});
// Prints 11
Console.WriteLine(result);
```

> Note: As we will see in the following section about rendering, you can also avoid rendering a script only mode by evaluating the template instead of rendering. 

[:top:](#runtime)
## Rendering a template

### Overview

In order to render a template, you need pass a context for the variables, objects, functions that will be accessed by the template.

In the following examples, we have a variable `name` that is used by the template:

```c#
var inputTemplateAsText = "This is a {{ name }} template";

// Parse the template
var template = Template.Parse(inputTemplateAsText);

// Renders the template with the variable `name` exposed to the template
var result = template.Render(new { name = "Hello World"});

// Prints the result: "This is a Hello World template"
Console.WriteLine(result);
```

As we can see, we are passing an anonymous objects that has this field/property `name` and by calling the Render method, the template is executed with this data model context.

While passing an anonymous object is nice for a hello world example, it is not always enough for more advanced data model scenarios. 

In this case, you want to use more directly the `TemplateContext` (used by the method `Template.Render(object)`) and a `ScriptObject` which are both at the core of scriban rendering architecture to provide more powerful constructions & hooks of the data model exposed (variables but also functions...etc.).

[:top:](#runtime)
### The `TemplateContext` execution model

The `TemplateContext` provides:

- **an execution context** when evaluating a template. The same instance can be used with many different templates, depending on your requirements.
- A **stack of `ScriptObject`** that provides the actual variables/functions accessible to the template, accessible through `Template.PushGlobal(scriptObj)` and `Template.PopGlobal()`. Why a stack and how to use this stack is described below.
- The **text output** when evaluating a template, which is accessible through the `Template.Output` property as a `StringBuilder` but because you can have nested rendering happening, it is possible to use `Template.PushOutput()` and `Template.PopOutput()` to redirect temporarily the output to a new output. This functionality is typically used by the [`capture` statement](language.md#94-capture-variable--end).
- Caching of templates previously loaded by an `include` directive (see [`include` and `ITemplateLoader`](#include-and-itemplateloader) section )
- Various possible overrides to allow fine grained extensibility (evaluation of an expression, conversion to a string, enter/exit/step into a loop...etc.)

Note that a `TemplateContext` is not thread safe, so it is recommended to have one `TemplateContext` per thread.

[:top:](#runtime)
### The `ScriptObject`

The `ScriptObject` is a special implementation of a `Dictionary<string, object>` that runtime properties and functions accessible to a template:

- **Accessing as regular dictionary objects**:

  ```C#
{'fraction_non_alphanumeric': 0.08600072472520835, 'fraction_numerical': 0.0008455127430849137, 'mean_word_length': 4.777739009071877, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2170442', 'n_tokens_mistral': 4274, 'n_tokens_neox': 4098, 'n_words': 2130}