<commit_before><commit_msg>* Initial commit of Overview document.  - Still needs filling out and prettying up<commit_after><?xml version='1.0' encoding='UTF-8'?>
<book xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink"
  xmlns:xi="http://www.w3.org/2001/XInclude" xmlns:svg="http://www.w3.org/2000/svg"
  xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:html="http://www.w3.org/1999/xhtml"
  xmlns:db="http://docbook.org/ns/docbook" version="5.0">
  <info>
    <title>Fabric Engine Overview</title>
    <copyright>
      <year>{{FABRIC_COPYRIGHT_YEARS}}</year>
      <holder>{{FABRIC_COMPANY_NAME}}</holder>
    </copyright>
    <releaseinfo>{{FABRIC_NAME}} Version {{FABRIC_VERSION}}</releaseinfo>
  </info>
  <chapter>
    <title>Introduction</title>
    <para>
      Fabric Engine is a high performance processing engine that integrates with dynamic languages, exposing a interface for defining multi-threaded native code operations. Fabric Engine enables high-performance applications to be built using dynamic languages by providing an environment where processor-intensive tasks can be offloaded and processed using multi-threaded native code.
    </para>
    <section>
      <title>Ease Of Use</title>
      <para>
        Fabric has been designed for developers comfortable building tools and services using dynamic languages. Developers working with Fabric do not require experience in high performance software development, or knowledge of complex languages such as C++. 
      </para>
      <para>
        Developers working with dynamic languages can continue to work using the same tools they are already familiar with and run their applications or services in the same way. Existing applications and services can be modified to Incorporate Fabric Engine for handling compute intensive tasks. The majority of code in most applications is not performance critical and can continue to be written in the developers choice of language. No custom IDE is required, or compilation tool chain as Fabric integrates a compiler and loads source code contained in text files. 
      </para>
    </section>
    <section>
      <title>Dynamic Environment</title>
      <para>
        Fabric Engine is a completely dynamic development environment where all data structures, graph structures, and code are specified at run time, providing all the benefits of dynamic languages. Fabric Engine can be integrated with any programming language, and is initially integrated with JavaScript and Python, with support for other languages arriving over time. 
      </para>
      <para>
        This paradigm of development enables a wider range of developers to build high performance applications, by significantly lowering the barrier to entry.
      </para>
    </section>
    <section>
      <title>Open Source</title>
      <para>
        The Fabric Engine source code has been released under the AGLPV3 Open Source Licensing Agreement, with commercial licenses available on request. Developers have full access to all the source code used in Fabric, allowing all core systems to be reviewed, and modified if required. This makes integrating Fabric Engine into custom applications, or scripting languages a straight forward process with no black box code to hinder learning.
      </para>
    </section>
  </chapter>
  <chapter>
    <title>The KL Language</title>
    <para>
      KL is a custom language used to define operators in Fabric Engine. The KL language combines the benefits of high performance languages like C++, with the benefits of dynamic languages like JavaScript and Python.
    </para>
    <section>
      <title>Ease of Use</title>
      <para>
        KL is a language intended to be used in the context of dynamic languages. Developers familiar with higher level languages such as Python or JavaScript are able to learn the simple rules of KL quickly and become productive. The name ‘KL’ means ‘Kernel Language’, and refers to the scope of the language. KL is for writing kernels, which are small stateless blocks of code with an entry function. The syntax of KL is based on a the syntax of JavaScript, while introducing syntax critical to performance such as strong typing. This means that all variables in KL source code have a type, such as ‘Scalar’ or ‘Integer’ and this must be specified in the source code. 
      </para>
      <para>
        The KL language is ‘pointer-less’ meaning that users cannot use or manipulate pointers to system memory. This restriction simplifies usage of KL significantly and makes it impossible for KL to create issues such as memory leaks. All heap memory allocation is managed by the Fabric Engine core, abstracting the details of memory management from the user.
      </para>
    </section>
    <section>
      <title>Dynamic Compilation</title>
      <para>
        Fabric Engine integrates the LLVM compiler and uses it to compile the KL source code used in operators. The KL source code is compiled by Fabric as the dynamic language constructs each operator. No custom IDE tools are required to work with Fabric, so developers can continue to use whatever editors they currently use while working with Fabric Engine. Applications can use dynamic compilation to create applications that have self modifying behavior.
      </para>
    </section>
    <section>
      <title>Cross Platform</title>
      <para>
        Due to dynamic compilation of the KL source code to machine code, the dynamic application remains completely dynamic and portable across operating systems and various CPU architectures and devices, while not compromising on performance. Fabric Engine currently supports all major operating systems, and support for a wider range or CPU architectures in development.
      </para>
    </section>
    <section>
      <title>Performance</title>
      <para>
        KL is a strongly typed language like C, and therefore can be optimized in the same way during compilation. The resulting native machine code executes as fast as C or C++, and is integrated into the application dynamically. A benefit of dynamic compilation is that the compilation process can take into account the architecture of the clients CPU, generating optimal machine code.
      </para>
    </section>
    <section>
      <title>Security</title>
      <para>
        Fabric Engine can be run in a range of environments including client side web applications. The KL language is constrained in such a way that it makes it impossible for malicious applications to hijack a running application using Fabric and used it to penetrate the browsers security. This makes it safe to run Fabric Engine in either client side JavaScript applications, or in online web services. 
      </para>
    </section>
  </chapter>
  <chapter>
    <title>Registered Types</title>
    <para>
      Fabric Engine provides a method for defining the types that are used in the KL operators and dependency graph nodes. Once a custom type has been registered, the data type can be used to populate nodes and used in KL operators. This type system enables the types used in the KL source code to be defined before compilation.
    </para>
    <para>
      Using a small number of basic atomic types, users can register custom types that combine these types creating new complex types. Types can be defined using existing complex types enabling the definition of complex nested data structures. Types can have KL functions associated with them enabling object method syntax to be used with the type in KL. 
    </para>
  </chapter>
  <chapter>
    <title>Dependency Graph</title>
    <para>
      Today high-performance applications must leverage modern multi-core CPUs and GPUs to fully exploit the available computation resources. The number of cores in a CPU continues to increase, while at the same time GPUs continue to gain power and flexibility.
    </para>
    <para>
      The dependency graph model abstracts the concepts of thread management from the user, while enabling the construction of highly scalable multi-threaded applications.
    </para>
    <section>
      <title>Multi-threaded Evaluation</title>
      <para>
        Nodes are constructed using the dynamic language, and dependencies are defined between nodes. Operators are constructed and applied to the nodes in the graph. This combination of nodes, dependencies, and KL operators describes a complete workload that is made up of many tasks that can be distributed across available compute resources.
      </para>
    </section>
    <section>
      <title>Task Based Parallelism</title>
      <para>
        Each node in the dependency graph can have operators applied. The operators define how the data should be processed, and when it is propagated through the graph. The binding of an operator to a node represents a task to be executed. During evaluation of the graph the Fabric Engine core can evaluate node simultaneously that do not have dependencies on each other. 
      </para>
    </section>
    <section>
      <title>Data Based Parallelism</title>
      <para>
        Each node in the dependency graph can have operators applied. The operators define how the data should be processed, and when it is propagated through the graph. The binding of an operator to a node represents a task to be executed. During evaluation of the graph the Fabric Engine core can evaluate node simultaneously that do not have dependencies on each other. 
      </para>
    </section>
    <section>
      <title>Data Based Parallelism</title>
      <para>
        Each node in the dependency graph can be sliced, meaning that each member contained in a node is duplicated according to the number of node slices. This enables nodes to store large quantities of data across many slices, defining homogeneous data sets. Operators bound to the node can be invoked for each slice in parallel, enabling data based parallelism.
      </para>
    </section>
    <section>
      <title>Dynamic Graph Manipulation</title>
      <para>
        The Fabric Engine core systems can be modified at runtime, enabling the behavior of a running application to change based on such things as user input, or network events. In between evaluations of the core, the dynamic language can add or remove data, nodes, or operators. The structure of the graph can be changed causing different behavior. After modifications, a validation is performed, before execution continues at full speed.
      </para>
    </section>
  </chapter>
  <chapter>
    <title>Event Graph</title>
    <para>
      The Event graph is used to sequence the execution of a set of operators thought the construction of a tree structure. While the dependency graph is used for multi-threaded evaluation, the event graph is used for single-threaded evaluation. The event tree is built using a combination of a single event node, with a tree of event handlers arranges in a tree structure below it. Evaluation always starts at the event node, and traverses the tree in a depth first fashion evaluating operators during both descent and ascent. 
    </para>
    <section>
      <title>Rendering</title>
      <para>
        The event graph is typically used to build rendering pipelines that draw to the screen using OpenGL. Each viewport in Fabric provides a custom event node that is fired whenever the viewport needs to be rendered, and this causes all of the operators in the tree to be evaluated in the sequence defined by the structure of the tree. 
      </para>
      <para>
        The event graph tree structure and sequential evaluation enables complex rendering configurations to be built using the dynamic language such as Python or JavaScript. The resulting tree evaluates quickly and can be modified at runtime.
      </para>
    </section>
  </chapter>
  <chapter>
    <title>Map Reduce</title>
    <para>
      Fabric Engine provides a generic map-reduce framework that can be used to create recursively-parallel operations on large data sets. Map reduce is an ideal paradigm for solving problems that are solved recursively, or require a reduction step such as counting elements in large data sets. 
The map-reduce framework can be used in isolation, without construction a dependency graph, and this can simplify the development of applications that do not maintain state, such as web services. 
    </para>
    <para>
      the Fabric Engine implementation of map-reduce expands on the concepts of the classic map reduce model, providing a more flexible tool. By allowing the results of a map reduce call do be used to drive further map reduce calls, problems that are difficult to parallelise become solvable in this multi-threaded context.
    </para>
  </chapter>
  <chapter>
    <title>Extensions</title>
    <para>
      Extensions are libraries loaded at initialization time that can define libraries of KL functions, and/or bind C++ code libraries and expose their interfaces to KL. The extension system is used to integrate existing libraries of code with Fabric Engine applications. Fabric Engine comes packaged with a wide range of extensions that provide a range of functionality from interfacing to Microsoft’s Kinect Camera, to integration of the Bullet Physics Engine.
    </para>
    <section>
      <title>Integration of Existing Code Libraries</title>
      <para>
        Fabric Provides OpenGL bindings via an extension utilizing the GLEW OpenGL wrapper system. Similarly, any API such as Microsoft DirectX, or NVidia CUDA could be exposed via custom extensions.
      </para>
      <para>
        All source code for the provided extensions is available in the Fabric Engine repository, and can be used as the basis of a users custom extensions.
      </para>
    </section>
  </chapter>
  <chapter>
    <title>Meta Graphs</title>
    <para>
      The dependency graph, event graph, and operators provide a collection of tools for constructing a wide range of high performance applications. Often however, the API needs to be abstracted to provide interfaces specific to a given domain, such as image processing, or 3d graphics. An abstraction layer can be written in the dynamic language that provides higher level functionality to developers. The abstraction layer may automate the construction of graphs, and simplify common tasks. 
    </para>
    <para>
      Fabric Engine provides two such meta-graphs used for 3d graphics. A Python and Qt based Scene Graph targeted at industries such as high-end digital content creation for the Film, VFX, and games industry. Theses meta graphs provide an interface users familiar with other scene graphs will recognise, making it possible to construct scenes in just a few lines of code.
    </para>
    <section>
      <title>PythonQt Scene Graph</title>
      <para>
        The PytQy scene graph exposes an interface for building scenes that can be rendered using a custom Qt OpenGL widget. Qt is used for all GUI, layout and window management.
      </para>
      <para>
        The Scene graph provides systems for Cameras, Materials, Geometry, and data loading, and simulation. Using the PyQt scenegraph users can quickly build sophisticated tools and use all of the available Python and Qt technologies available.
      </para>
      <section>
        <title>Data Loading</title>
        <para>
          A wide variety of data formats are supported by the Python scene graph, including Autodesk FBX, Wavefront OBJ, LIDAR, and many image and video formats. The files are are opened and parsed by the extension, returning the parsed data to the calling KL operator code. 
        </para>
      </section>
      <section>
        <title>Simulation</title>
        <para>
          The Python Scene Graph provides a flexible framework for defining custom simulations. The time management of Fabric applications is defined in Python, enabling complex behaviours such as sub-frame evaluation, or any time step. The Python Scene Graph provides basic acceleration structures, and can be customised according to specific use cases.
        </para>
      </section>
      <section>
        <title>Rendering</title>
        <para>
        The SceneGraph comes with support for forward based rendering, deferred rendering, and volume rendering. An XML based material system uses GLSL shaders to define rendering, and can utilize all the features of a modern graphics card. The data driven material system, written entirely in Python can be modified by users according to thier own requirements.
        </para>
      </section>
    </section>
    <section>
      <title>JavaScript Scene Graph</title>
      <para>
        The JavaScript scene graph similar to the Python scene graph, with a reduced scope intended for content distribution online, or via HTML apps. 
      </para>
      <para>
        The JavaScript scene graph is useful for rendering complex data sets, or securely displaying sensitive data online. The JavaScript Fabric Scene Graph supports loading and parsing of binary formats, and provides interfaces to generate custom binary files that encode valuable asset data.
      </para>
      <section>
        <title>Data Loading</title>
        <para>
          The Fabric NPAPI plugin provides interfaces for opening and writing files on the users local machine. This enables web applications to write out files from online hosted web applications. Fabric protects the users security by only exposing this interface via user validated actions such as a file open dialog. 
        </para>
      </section>
      <section>
        <title>Rendering</title>
        <para>
          The JavaScript scene graph features the same XML based Material system as the Python Scene Graph and can also utilize all available features of the users GPU. Through Fabric, users running modern graphics cards can use the latest OpenGL 4.0 features, such as tessellating, and geometry instancing.
        </para>
      </section>
    </section>
  </chapter>
</book>
{'fraction_non_alphanumeric': 0.04997778764993336, 'fraction_numerical': 0.001943580630830742, 'mean_word_length': 3.552325581395349, 'pattern_counts': {'":': 0, '<': 220, '<?xml version=': 1, '>': 220, 'https://': 0, 'lorem ipsum': 0, 'www.': 5, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3628030', 'n_tokens_mistral': 4183, 'n_tokens_neox': 3977, 'n_words': 2438}