<commit_before>=========================
CML (Cache Meta Language)
=========================

---------------
Module: mod_cml
---------------

:Author: <NAME>
:Date: $Date: 2004/11/03 22:26:05 $
:Revision: $Revision: 1.2 $

:abstract:
  CML is a Meta language to describe the dependencies of a page at one side and building a page from its fragments on the other side using LUA.
  
.. meta::
  :keywords: lighttpd, cml, lua
  
.. contents:: Table of Contents

Description
===========

CML (Cache Meta Language) wants to solves several problems:

 * dynamic content needs caching to perform
 * checking if the content is dirty inside of the application is usually more expensive than sending out the cached data
 * a dynamic page is usually fragmented and the fragments have different livetimes
 * the different fragements can be cached independently

Cache Decision
--------------

A simple example should show how to a content caching the very simple way in PHP. 

jan.kneschke.de has a very simple design:

 * the layout is taken from a template in templates/jk.tmpl
 * the menu is generated from a menu.csv file
 * the content is coming from files on the local directory named content-1, content-2 and so on

The page content is static as long non of the those tree items changes. A change in the layout 
is affecting all pages, a change of menu.csv too, a change of content-x file only affects the
cached page itself.

If we model this in PHP we get: ::

  <?php

  ## ... fetch all content-* files into $content
  $cachefile = "/cache/dir/to/cached-content";

  function is_cachable($content, $cachefile) {
    if (!file_exists($cachefile)) {
      return 0;
    } else {
      $cachemtime = filemtime($cachefile);
    }

    foreach($content as $k => $v) {
      if (isset($v["file"]) && 
          filemtime($v["file"]) > $cachemtime) {
        return 0;
      }
    }

    if (filemtime("/menu/menu.csv") > $cachemtime) {
      return 0;
    }
    if (filemtime("/templates/jk.tmpl") > $cachemtime) {
      return 0;
    }
  }
  
  if (is_cachable(...), $cachefile) {
    readfile($cachefile);
    exit();
  } else {
    # generate content and write it to $cachefile
  }
  ?>

Quite simple. No magic involved. If the one of the files is new than the cached
content, the content is dirty and has to be regenerated. 

Now let take a look at the numbers:

 * 150 req/s for a Cache-Hit
 * 100 req/s for a Cache-Miss

As you can see the increase is not as good as it could be. The main reason as the overhead
of the PHP interpreter to start up (a byte-code cache has been used here).

Moving these decisions out of the PHP script into a server module will remove the need
to start PHP for a cache-hit. 

To transform this example into a CML you need 'index.cml' in the list of indexfiles 
and the following index.cml file: ::

  output_content_type = "text/html"

  b = request["DOCUMENT_ROOT"]
  cwd = request["CWD"]

  output_include = { b .. "_cache.html" }

  trigger_handler = "index.php"

  if file_mtime(b .. "../lib/php/menu.csv") > file_mtime(cwd .. "_cache.html") or
     file_mtime(b .. "templates/jk.tmpl")   > file_mtime(cwd .. "_cache.html") or
     file_mtime(b .. "content.html")        > file_mtime(cwd .. "_cache.html") then
     return CACHE_MISS
  else 
     return CACHE_HIT
  end

Numbers again:

 * 4900 req/s for Cache-Hit
 *  100 req/s for Cache-Miss

Content Assembling
------------------

Sometimes the different fragment are already generated externally. You have to cat them together: ::

  <?php
  readfile("head.html");
  readfile("menu.html");
  readfile("spacer.html");
  readfile("db-content.html");
  readfile("spacer2.html");
  readfile("news.html");
  readfile("footer.html");
  ?> 

We we can do the same several times faster directly in the webserver. 

Don't forget: Webserver are built to send out static content, that is what they can do best.

The index.cml for this looks like: ::

  output_content_type = "text/html"
  
  cwd = request["CWD"]
   
  output_include = { cwd .. "head.html", 
                     cwd .. "menu.html",
                     cwd .. "spacer.html",
                     cwd .. "db-content.html",
                     cwd .. "spacer2.html",
                     cwd .. "news.html",
                     cwd .. "footer.html" }
  
  return CACHE_HIT

Now we get about 10000 req/s instead of 600 req/s.

Power Magnet
------------

Next to all the features about Cache Decisions CML can do more. Starting
with lighttpd 1.4.9 a power-magnet was added which attracts each request
and allows you to manipulate the request for your needs.  

We want to display a maintainance page by putting a file in a specified<commit_msg>fixed the wrong typo :)<commit_after>=========================
CML (Cache Meta Language)
=========================

---------------
Module: mod_cml
---------------

:Author: <NAME>
:Date: $Date: 2004/11/03 22:26:05 $
:Revision: $Revision: 1.2 $

:abstract:
  CML is a Meta language to describe the dependencies of a page at one side and building a page from its fragments on the other side using LUA.
  
.. meta::
  :keywords: lighttpd, cml, lua
  
.. contents:: Table of Contents

Description
===========

CML (Cache Meta Language) wants to solves several problems:

 * dynamic content needs caching to perform
 * checking if the content is dirty inside of the application is usually more expensive than sending out the cached data
 * a dynamic page is usually fragmented and the fragments have different livetimes
 * the different fragements can be cached independently

Cache Decision
--------------

A simple example should show how to a content caching the very simple way in PHP. 

jan.kneschke.de has a very simple design:

 * the layout is taken from a template in templates/jk.tmpl
 * the menu is generated from a menu.csv file
 * the content is coming from files on the local directory named content-1, content-2 and so on

The page content is static as long non of the those tree items changes. A change in the layout 
is affecting all pages, a change of menu.csv too, a change of content-x file only affects the
cached page itself.

If we model this in PHP we get: ::

  <?php

  ## ... fetch all content-* files into $content
  $cachefile = "/cache/dir/to/cached-content";

  function is_cachable($content, $cachefile) {
    if (!file_exists($cachefile)) {
      return 0;
    } else {
      $cachemtime = filemtime($cachefile);
    }

    foreach($content as $k => $v) {
      if (isset($v["file"]) && 
          filemtime($v["file"]) > $cachemtime) {
        return 0;
      }
    }

    if (filemtime("/menu/menu.csv") > $cachemtime) {
      return 0;
    }
    if (filemtime("/templates/jk.tmpl") > $cachemtime) {
      return 0;
    }
  }
  
  if (is_cachable(...), $cachefile) {
    readfile($cachefile);
    exit();
  } else {
    # generate content and write it to $cachefile
  }
  ?>

Quite simple. No magic involved. If the one of the files is new than the cached
content, the content is dirty and has to be regenerated. 

Now let take a look at the numbers:

 * 150 req/s for a Cache-Hit
 * 100 req/s for a Cache-Miss

As you can see the increase is not as good as it could be. The main reason as the overhead
of the PHP interpreter to start up (a byte-code cache has been used here).

Moving these decisions out of the PHP script into a server module will remove the need
to start PHP for a cache-hit. 

To transform this example into a CML you need 'index.cml' in the list of indexfiles 
and the following index.cml file: ::

  output_contenttype = "text/html"

  b = request["DOCUMENT_ROOT"]
  cwd = request["CWD"]

  output_include = { b .. "_cache.html" }

  trigger_handler = "index.php"

  if file_mtime(b .. "../lib/php/menu.csv") > file_mtime(cwd .. "_cache.html") or
     file_mtime(b .. "templates/jk.tmpl")   > file_mtime(cwd .. "_cache.html") or
     file_mtime(b .. "content.html")        > file_mtime(cwd .. "_cache.html") then
     return CACHE_MISS
  else 
     return CACHE_HIT
  end

Numbers again:

 * 4900 req/s for Cache-Hit
 *  100 req/s for Cache-Miss

Content Assembling
------------------

Sometimes the different fragment are already generated externally. You have to cat them together: ::

  <?php
  readfile("head.html");
  readfile("menu.html");
  readfile("spacer.html");
  readfile("db-content.html");
  readfile("spacer2.html");
  readfile("news.html");
  readfile("footer.html");
  ?> 

We we can do the same several times faster directly in the webserver. 

Don't forget: Webserver are built to send out static content, that is what they can do best.

The index.cml for this looks like: ::

  output_contenttype = "text/html"
  
  cwd = request["CWD"]
   
  output_include = { cwd .. "head.html", 
                     cwd .. "menu.html",
                     cwd .. "spacer.html",
                     cwd .. "db-content.html",
                     cwd .. "spacer2.html",
                     cwd .. "news.html",
                     cwd .. "footer.html" }
  
  return CACHE_HIT

Now we get about 10000 req/s instead of 600 req/s.

Power Magnet
------------

Next to all the features about Cache Decisions CML can do more. Starting
with lighttpd 1.4.9 a power-magnet was added which attracts each request
and allows you to manipulate the request for your needs.  

We want to display a maintainance page by putting a file in a specified
{'fraction_non_alphanumeric': 0.12020460358056266, 'fraction_numerical': 0.010230179028132993, 'mean_word_length': 2.716831683168317, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5745451', 'n_tokens_mistral': 2922, 'n_tokens_neox': 2731, 'n_words': 1141}