<commit_before>.. -*- RST -*-

.. _tutorial-pastebin:

==========
 Pastebin
==========

As a demonstration of using `RESTAS`_ for creation of a full-featured
web application I want to show you how to construct a simple pastebin service.

.. note:: Note

  From `Wikipedia <http://en.wikipedia.org/wiki/Pastebin>`_:
  A pastebin is a web application which allows its users to upload snippets
  of text, usually samples of source code, for public viewing. It is very
  popular in IRC  channels where pasting large amounts of text is considered
  bad etiquette. A vast number of pastebins exist on the Internet, suiting
  a number of different needs and provided features tailored towards
  the crowd they focus on most.

Pastebin is a simple idea that demonstrates the basic
principles of building `RESTAS`_ applications. Apart from this, it is
a real application used by the `lisper.ru`_ site.  You can see it in
action `here <http://lisper.ru/apps/format/>`_.

The app must have the following abilities:

* A form for creating a new element.

* A list of all registered elements, with a way to view them after clicking
  on an element.

* A view for displaying every element.

* Syntax highlighting.

Apart from this, pastebin is aimed for using in a context of different sites and
hereby must have flexible means of configuration:

* Use the authentication system used by the site.

* Use the data storage mechanism provided by the site.

* Provide customisation for rendering pages.

Logical structure
=================

This application is built with classical MVC model and has distinct
separation of model, controller, and view.

Customisation of the module is done with special (dynamic) variables
(for details about customising modules with dynamic variables
see section :ref:`manual-modules`).

Pastebin data can be stored in a number of ways. We define aninterface
and a special variable :fun:`*storage*`, with which an outside
application can specify an implementation for storing data. For
development and debugging a developer needs to have a functioning
model, which is implemented by a trivial class :class:`memory-storage`
providing a way of storing data in the memory. Such an approach lets
you develop the application without worrying about the structure of a
database until you need it.

Usually sites have an unified authentication scheme which should be used in
all their components. Therefore the component under development shouldn't
know anything about the used authentication scheme. In fact, for proper
functioning it needs to know only the username. Consequently, it is enough
to introduce just one special variable :var:`*colorize-user-function*`
that will store a function for figuring out the username.

In order to provide flexible customisation of page view for
separation of logic and view the component uses the special variable
:var:`*default-render-method*` created by `RESTAS`_ and storing the object
in charge of representation (you will find it in more details in the section
:ref:`manual-view`). And for definition of page templates we use
`cl-closure-template`_ library.
 
Physical structure
==================

To simplify understanding of the code structure the following division into files is
used:

* **defmodule.lisp** - contains definition of the module and, also, special variables
  for controlling the behaviour of the component.

* **storage.lisp** - contains definition of the interface for used data model and
  a simple implementation of this model.

* **drawer.lisp** - interface and implementation of the default drawer object
  that is used as the value of :var:`*default-render-method*` special variable.

* **drawer.tmpl** - file with templates in `cl-closure-template`_ format

* **routes.lisp** - the central part of the application where URL scheme and
  request handlers are defined.

Implementation
==============

defmodule.lisp
--------------

.. code-block:: common-lisp

  (restas:define-module #:restas.colorize
    (:use #:cl #:iter)
    (:export ..))

  (in-package #:restas.colorize)

  (defvar *max-on-page* 10)

  (defvar *storage* nil)

  (defvar *colorize-user-function* #'(lambda () "anonymous"))

  (defun colorize-user ()
    (if *colorize-user-function*
        (funcall *colorize-user-function*)))

Here we define the new module **#:restas.colorize** (for economy of space
the list of exported symbols is omitted) and introduce several special variables
for using them in module customisation (variables :var:`*max-on-page*`
and :var:`*colorize-user-function*` come with default values) and also the
:fun:`colorize-user` helper function is defined for figuring out the username.

storage.lisp
------------

Definition of the :var:`*storage*` object interface:

.. code-block:: common-lisp

  (defgeneric storage-count-notes (storage)
    (:documentation "Total number of notes"))

  (defgeneric storage-list-notes (storage offset limit)
    (:documenation "List of notes starting with offset, but not more than limit"))

  (defgeneric storage-get-note (storage id)
    (:documentation "Get the note by id"))

  (defgeneric storage-add-note (storage note)
    (:documentation "Add new note"))

Helper class for storing information about a record:

.. code-block:: common-lisp

  (defclass note ()
    ((id :initarg :id :initform nil :accessor note-id)
     (date :initarg :date :initform nil :accessor note-date)
     (author :initarg :author :initform nil :accessor note-author)
     (title :initarg :title :initform nil :accessor note-title)
     (lang :initarg :lang :initform nil :accessor note-lang)
     (code :initarg :code :initform nil :accessor note-code)))

To implement an in-memory storage engine it's enough to just cite the class
definition:

.. code-block:: common-lisp

  (defclass memory-storage ()
    ((notes :initform nil)
     (last-id :initform 0)))

Setting default value for :var:`*storage*` variable:

.. code-block:: common-lisp

  (setf *storage* (make-instance 'memory-storage))

routes.lisp
-----------

In this file, processed URLs and the operation logic are defined.
Because all view logic is handled by :var:`*default-render-method*`
object, route handlers should just gather and prepare data for
processing them with function :fun:`restas:render-object`.
The view logic, as I said before, is implemented on the basis of
`cl-closure-template`_ library which uses *plist* as input data format
(you can read about using *property list* in more detail at
`here <http://lisper.ru/pcl/practical-a-simple-database>`_), so route handlers
generate data in this format.

A couple of helper functions for translating record information to *plist* format:

.. code-block:: common-lisp

  (defun note-plist/short (note)
    (list :href (restas:genurl 'view-note :id (note-id note))
          :date (local-time:format-timestring nil (note-date note))
          :title (note-title note)
          :author (note-author note)))

  (defun note-plist (note)
    (list* :title (note-title note)
           :code (note-code note)
           :lang (note-lang note)
           (note-plist/short note)))

In this code, the following call is of the most interest:

.. code-block:: common-lisp

  (restas:genurl 'view-note :id (note-id note))

It generates a URL for viewing a record basing on the route name ('view-node) and :id
parameters.

.. note:: None

  In route-based systems "manual" creation of URLs is not a common practice,
  because automatic generation considerably increases flexibility of an
  application and prevents many potential problems. Among other issues,
  in this case the base URL, which is an address where the module
  is mounted, is unknown at the module development stage.

Application entry point:

.. code-block:: common-lisp

  (restas:define-route main ("")
    (restas:redirect 'list-notes))

This route processes a request to the base URL where the module is mounted,
and simply redirects it to the page with the record list. For redirection
:fun:`restas:redirect` function is used which, like :fun:`restas:genurl`,
processes the name of a route (and a set of parameters, but 'list-notes route
doesn't have them) so there is no need to actually specify the URL.

For displaying the record list we use a paginated representation, with
the number of records on a single page specified by
:var:`*max-on-page*` (this variable was introduced in
**defmodule.lisp**). So, apart from the record list itself you need to
gather information about overall number of records, and also provide<commit_msg>fix typo in documentaion<commit_after>.. -*- RST -*-

.. _tutorial-pastebin:

==========
 Pastebin
==========

As a demonstration of using `RESTAS`_ for creation of a full-featured
web application I want to show you how to construct a simple pastebin service.

.. note:: Note

  From `Wikipedia <http://en.wikipedia.org/wiki/Pastebin>`_:
  A pastebin is a web application which allows its users to upload snippets
  of text, usually samples of source code, for public viewing. It is very
  popular in IRC  channels where pasting large amounts of text is considered
  bad etiquette. A vast number of pastebins exist on the Internet, suiting
  a number of different needs and provided features tailored towards
  the crowd they focus on most.

Pastebin is a simple idea that demonstrates the basic
principles of building `RESTAS`_ applications. Apart from this, it is
a real application used by the `lisper.ru`_ site.  You can see it in
action `here <http://lisper.ru/apps/format/>`_.

The app must have the following abilities:

* A form for creating a new element.

* A list of all registered elements, with a way to view them after clicking
  on an element.

* A view for displaying every element.

* Syntax highlighting.

Apart from this, pastebin is aimed for using in a context of different sites and
hereby must have flexible means of configuration:

* Use the authentication system used by the site.

* Use the data storage mechanism provided by the site.

* Provide customisation for rendering pages.

Logical structure
=================

This application is built with classical MVC model and has distinct
separation of model, controller, and view.

Customisation of the module is done with special (dynamic) variables
(for details about customising modules with dynamic variables
see section :ref:`manual-modules`).

Pastebin data can be stored in a number of ways. We define aninterface
and a special variable :fun:`*storage*`, with which an outside
application can specify an implementation for storing data. For
development and debugging a developer needs to have a functioning
model, which is implemented by a trivial class :class:`memory-storage`
providing a way of storing data in the memory. Such an approach lets
you develop the application without worrying about the structure of a
database until you need it.

Usually sites have an unified authentication scheme which should be used in
all their components. Therefore the component under development shouldn't
know anything about the used authentication scheme. In fact, for proper
functioning it needs to know only the username. Consequently, it is enough
to introduce just one special variable :var:`*colorize-user-function*`
that will store a function for figuring out the username.

In order to provide flexible customisation of page view for
separation of logic and view the component uses the special variable
:var:`*default-render-method*` created by `RESTAS`_ and storing the object
in charge of representation (you will find it in more details in the section
:ref:`manual-view`). And for definition of page templates we use
`cl-closure-template`_ library.
 
Physical structure
==================

To simplify understanding of the code structure the following division into files is
used:

* **defmodule.lisp** - contains definition of the module and, also, special variables
  for controlling the behaviour of the component.

* **storage.lisp** - contains definition of the interface for used data model and
  a simple implementation of this model.

* **drawer.lisp** - interface and implementation of the default drawer object
  that is used as the value of :var:`*default-render-method*` special variable.

* **drawer.tmpl** - file with templates in `cl-closure-template`_ format

* **routes.lisp** - the central part of the application where URL scheme and
  request handlers are defined.

Implementation
==============

defmodule.lisp
--------------

.. code-block:: common-lisp

  (restas:define-module #:restas.colorize
    (:use #:cl #:iter)
    (:export ..))

  (in-package #:restas.colorize)

  (defvar *max-on-page* 10)

  (defvar *storage* nil)

  (defvar *colorize-user-function* #'(lambda () "anonymous"))

  (defun colorize-user ()
    (if *colorize-user-function*
        (funcall *colorize-user-function*)))

Here we define the new module **#:restas.colorize** (for economy of space
the list of exported symbols is omitted) and introduce several special variables
for using them in module customisation (variables :var:`*max-on-page*`
and :var:`*colorize-user-function*` come with default values) and also the
:fun:`colorize-user` helper function is defined for figuring out the username.

storage.lisp
------------

Definition of the :var:`*storage*` object interface:

.. code-block:: common-lisp

  (defgeneric storage-count-notes (storage)
    (:documentation "Total number of notes"))

  (defgeneric storage-list-notes (storage offset limit)
    (:documenation "List of notes starting with offset, but not more than limit"))

  (defgeneric storage-get-note (storage id)
    (:documentation "Get the note by id"))

  (defgeneric storage-add-note (storage note)
    (:documentation "Add new note"))

Helper class for storing information about a record:

.. code-block:: common-lisp

  (defclass note ()
    ((id :initarg :id :initform nil :accessor note-id)
     (date :initarg :date :initform nil :accessor note-date)
     (author :initarg :author :initform nil :accessor note-author)
     (title :initarg :title :initform nil :accessor note-title)
     (lang :initarg :lang :initform nil :accessor note-lang)
     (code :initarg :code :initform nil :accessor note-code)))

To implement an in-memory storage engine it's enough to just cite the class
definition:

.. code-block:: common-lisp

  (defclass memory-storage ()
    ((notes :initform nil)
     (last-id :initform 0)))

Setting default value for :var:`*storage*` variable:

.. code-block:: common-lisp

  (setf *storage* (make-instance 'memory-storage))

routes.lisp
-----------

In this file, processed URLs and the operation logic are defined.
Because all view logic is handled by :var:`*default-render-method*`
object, route handlers should just gather and prepare data for
processing them with function :fun:`restas:render-object`.
The view logic, as I said before, is implemented on the basis of
`cl-closure-template`_ library which uses *plist* as input data format
(you can read about using *property list* in more detail at
`here <http://lisper.ru/pcl/practical-a-simple-database>`_), so route handlers
generate data in this format.

A couple of helper functions for translating record information to *plist* format:

.. code-block:: common-lisp

  (defun note-plist/short (note)
    (list :href (restas:genurl 'view-note :id (note-id note))
          :date (local-time:format-timestring nil (note-date note))
          :title (note-title note)
          :author (note-author note)))

  (defun note-plist (note)
    (list* :title (note-title note)
           :code (note-code note)
           :lang (note-lang note)
           (note-plist/short note)))

In this code, the following call is of the most interest:

.. code-block:: common-lisp

  (restas:genurl 'view-note :id (note-id note))

It generates a URL for viewing a record basing on the route name ('view-node) and :id
parameters.

.. note:: Note

  In route-based systems "manual" creation of URLs is not a common practice,
  because automatic generation considerably increases flexibility of an
  application and prevents many potential problems. Among other issues,
  in this case the base URL, which is an address where the module
  is mounted, is unknown at the module development stage.

Application entry point:

.. code-block:: common-lisp

  (restas:define-route main ("")
    (restas:redirect 'list-notes))

This route processes a request to the base URL where the module is mounted,
and simply redirects it to the page with the record list. For redirection
:fun:`restas:redirect` function is used which, like :fun:`restas:genurl`,
processes the name of a route (and a set of parameters, but 'list-notes route
doesn't have them) so there is no need to actually specify the URL.

For displaying the record list we use a paginated representation, with
the number of records on a single page specified by
:var:`*max-on-page*` (this variable was introduced in
**defmodule.lisp**). So, apart from the record list itself you need to
gather information about overall number of records, and also provide
{'fraction_non_alphanumeric': 0.09175711352302728, 'fraction_numerical': 0.0003520093869169845, 'mean_word_length': 4.091397849462366, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7364822', 'n_tokens_mistral': 4687, 'n_tokens_neox': 4496, 'n_words': 2196}