<commit_before>= Guide to using Annalist
<NAME>
v0.1.15, 2015-08-04
:toc: macro
:toc-title:

This document provides an initial guide to using Annalist.  It assumes Annalist has been installed or deployed, and provides an overview ofconcepts involved and operations that can be performed using Annalist.

*Table of Contents*

toc::[]

== Concepts

[[annalist-concepts]]
image::figures/annalist-concepts.png[alt="Annalist concepts"]

=== Site

A site corresponds to an Annalist installation.  It may contain any number of data collections.

=== Collections

A *collection* is a grouping of data under a single administrative control (i.e. access control is applied at the level of a collection;  there is no fine-grained access control within a collection.)

=== Data records

A *data record* is a web-accessible unit of user data.   Each data record has an entity identifier and a type identifier;  the combination of collection identifier, type identifier and entity identifier uniquely identify a data record within a site.  Combined with the site's URI, this invormation allows for any data record to have a unique identity on the World Wide Web, which is fundamental to Annalist's support for linked data.

A data record may have any number of properties; new properties may be added at any time to any data record.  Each property is identified by a URI or CURIE (often referred to as URIs).  There may be multiple values associated with any property URI.  This representation of data records with arbitrary properties is at the heart of Annalist's flexibility, and is based on the RDF and JSON-LD data models.  By not imposing any predefined structure, the content of any data record can be varied as needs dictate.  Presented structure is defined, as needed, by views on a data record (see below).

The data type of a property value is not formally constrained.  Multiple instances of the property may have values with different data types.  In practice, the data type associated with a property is determined by the field type (see below) used in the views though which data is entered and/or displayed, and these will generally present and interpret the property value as a particular data type.  If different data types do appear for a single property URI, the view- and list-fields used should be able to handle them appropriately.


=== Record types

*Record types* provide a simple, coarse-grained way of distinguishing different types of record from a user perspective, and associating with different content presentation as appropriate.  Each data record has exactly one data type, which is used directly to help organize the underlying data storage.  The intent is that data types are used to ceate a top-level organization of different kinds of data within a collection.  Finer-grained and cross-cutting type distinctions may be made though properties on each data record.

In keeping with the idea that properties can be added to any data record at any tine, the description of a record type does not include any structural information.  Rather, it includes descriptive information, and association with default view and list presentations (other view and list presentations can be used, but must be explicitly selected).


=== Record views

A *record view* describes a selection and organization of properties to be used when displaying a single data record.  As such, a view establishes a presentation of a particular kind of structure on the data.  Different views may present different structures over the same underlying data.  A view is defined with a number of fields (see below, and also [View fields in Annalist](view-field-types.md)).


=== List views

A *list view* describes a selection and organization of data records and properties to be used for displaying multiple data records.  They are used to create summary displays from which individual data records may be selected.  A list is defined with a data record selector and a number of fields (see below).

Associated with each collection is a default list view, which is displayed when a collection is initially accessed.


=== Fields

Record and list views are build up using *fields*, which associate a data record property URI with a field renderer and a stored data value type.  A field render formats data for presentation (essentially as a fragment of HTML), and interprets user input as a storable data value.

The available field renderers are defined by Annalist, and the range of rendering options is expected to increase as the software is developed.  Currently, there are three main types of renderer:  single-line text (Text), multiline text (TextArea) and enumeration, pwhich presents a selection of identifiers of some other defined type.

For more information, see [View fields in Annalist](view-field-types.md).


=== Field groups

Some types of field (notably those using `RepeatGroup` and `RepeatGroupRow` render types) make reference to a *Field group* (sometimes just called a *Group*).  A field group is a separately-defined list of fields that may be incorporated into a record view.

Field groups are central to the handling of fields that may appear multiple times in a view.  For example, to define a bibliography record which may contain multiple author descriptions, the fields describing an author are defined as a frield group, and the bibliography record can refer to tghis field group using an appropriate render type.


=== Entities

Data records, types, views, lists, fields and more are all handled internally as *entities*, which are always associated with a data type.  Data types, views, lists, fields and groups are associated with the special data types `_type`, `_view`, `_list`, `_field` and `_group` respectively.  Using these predefined types, their descriptions can be processed using the same facilities that are used to create, modify and view user data records.

Additional special data types may be added as new features are implemented.  A convention has been adopted that the special data types have type identifiers that beging with an underscore (`'_'`), so to avoid clashes user data types and entities should be created with idnetifiers that do not start with an underscore.

Entities are stored as flat files in JSON (JSON-LD) format.


=== Site-wide data

A site has been described as consisting of a number of collections containing user-defined entities (data records, data types, views, lists, etc).

There is also provision for site-wide entities, which is used mainly for the predefined Annalist special types (`_type`, '_view`, `_list`, etc., and also for some predefined instances of these), but which can be used for any data.  All site-wide values are available to all collections in a site.  If a collection creates a local entity that is also available as a site-wide entity, the local value takes precedence for that collection. 

Site-wide data is fixed for an installation:  it cannot be created or modified through the Annalist user interface.  Certain operations may appear to modify a site-wide value (e.g. editing a default record view): these cause the site-wide dscription to be copied into the collection and updated there.  Thus, the view `default_view`


== Accessing Annalist

The following instructions assume Annalist service running on host `annalist.example.org`.  If a different host is used, replace `annalist.example.org` with the name or IP address of the host that is running the Annalist server.

. Browse to annalist server at `http://annalist.example.org:8000`
+
  An empty list of collections should be displayed, along with some help text:
+
  image::screenshots/Front-page-initial.png[alt="Initial front page"]

. Select the *Login* item from the top menu bar:
+
  image::screenshots/Login-initial.png[alt="Initial login page"]

. Log in to Annalist using one of the procedures described belo.]

. After logged in, click the *Home* link on the to menu bar to return to the front page:
+
  image::screenshots/Front-page-admin.png[alt="Initial front page after login"]
+
  Note that the front page now shows text entry boxes and a button for creating a new collection.


=== Log in with OpenID Connect credentials

The intended means of Annalist user authentication is via a third party identity provider (IDP) service:

. Enter a local username (which is how Annalist will identify you, independently of any third party authentication account you may use), select an IDP (e.g. 'Google') and click *Login*.

. If you are not already logged in to the IDP you will be asked to login via their site.  Then the IDP will ask your permission to disclose basic identifying information (email, first name and last name) to Annalist.  This step is skipped if you have completed these actions previously.

. If this is an existing Annalist account, and the email from the IDP matches the Annalist account email, you will be logged in.  If the username given does not match an existing Annalist account, a new account is created with the appropriate details and you are logged in to it.  Being logged in does not necessarily mean you have permissions to access Annalist data;  it simply means that Annalist has an indicator of who you are.

(@@NOTE: Currently, the authorization system is not fully implememted, and all authenticated users have full access to the Annalist data, but that will change before the first non-prototype software release.  See https://github.com/gklyne/annalist/issues/11)[Annalist issue 11])


=== Log in with local user credentials

(See below for creating local user credentials)

. Select the "Local user credentials: *login*" link at the bottom of the page:
+
  image::screenshots/Login-django.png[alt="Django login page"]

. Enter a previously created username and password

. Click on the *Login* button


=== Creating local user credentials

The Annalist software installation includes a step to create an admin user:

----
    annalist-manager createadminuser
----

The username and password created at this step can be used as local credentials to log in to Annalist as described above.

Having logged in using the admin username and password, the *Admin* link in the page footer can be used to create additional local users via the local administrative interface (which is implemented in the underlying Django web application framework).  When logged in to Annalist using this account, the *Admin* link in the footer of most Annalist pages will allow new user accounts to be created via the Django admin interface.  More documentation about using this admin interface can be found in the http://www.djangobook.com/en/2.0/chapter06.html[The Django Admin Site], which is Chapter 6 of http://www.djangobook.com/en/2.0/index.html[The Django Book].


=== Access control and user permissions

Being logged in to Annalist does not, of itself, grant permission to access or modify any Annalist data - except in a very broad sense covered by `_default_user_perms` (see below).  That is controlled separately by _user permission_ records, which may be created for any collection ovcerv which the user is granted access permissions.

When a user is logged in to Annalist (using OpenID or local credentials), their session is associated with a user identifier (user_id) and an email address, as well as some descriptive information.  The user_id alone is not a secure basis for access control, as Annalist allows anyone to create a new user_id through the OpenID based login process.  Rather, it is assumed that the associated email address has been properly checked as associated with the logged-in user:  this verification is something that should be handled by any OpenID identity provider (such as Google) worthy of being used as such.  Annalist uses the combination of user_id and email address as a reliable authenticated identifier, with which access permissions are associated.  Different user_ids may bve associated with the same email address, possibly corresponding to different roles of the authenticated user;  each such combination is associated with its own set of permissions. (Technically, Annalist internally uses a mailto: URI as an authenticated user identifier, and uses different forms of URI for some special cases.)

Permissions granted to a user consist of a list of tokens denoting different access rights; e.g. VIEW, CREATE, UPDATE, DELETE, etc.  Permissions are applied at the level of a collection;  a user may have different permissions in different collections.  The permisisons granted form part of the collection metadata, and as such are replicated if an entire collection is copied or moved to a different service.  It is also possible to associate permissions with an entire Annalist site, which means they are applied in any collection for which more specific permissions are not granted;  the user interface for creating site-wide permissions currently takes the form of a limited set of `annalist_manager` commands, and is intended to be used for setting up site-wide defaults and administrative access.

Built-in Annalist permission tokens include:

`ADMIN` - required to create or view user permissions in a collection.  The creator of a collection is automatically granted `ADMIN` permissions over that collection, so they can assign permissions in that collection for other users.  This permission at site level also allows creation and deletion of collections.

`CONFIG` - required to change the structure of a collection: to create and/or modify record types, views, lists, etc.

`CREATE` - required to create new data in a collection.

`UPDATE` - required to edit data records in a collection.

`VIEW` - required to view or read data records in a collection.

`DELETE` - required to remove data records in a collection.

`CREATE_COLLECTION` - this permission, or `ADMIN`, is required at site level to create a new collection.

`DELETE_COLLECTION` - this permission, or `ADMIN`, is required at site level to remove an existing collection.

As noted, permissions (currently) apply to the entire content of an annalist Collection; i.e. a collection is the level of granularity at which access control is enforced.  (The internal design of the authorization system allows for new user-defined tokens to be introduced, and potentially to be associated with record types.  This could be used to provide sub-collection granularity of access control - e.g. to restrict access to a record type that may contain particularly sensitive data.)

==== Creating user permissions in a collection

A logged in user with ADMIN permissions over a collection may create further collection-level user permissions from the "List users" view of a collection, which is accessed from the default collection view by selecting `User_list` from the `List view` dropdown, then clicking the `View` button:

  image::screenshots/List-users-example.png[alt="List users view"]

A new user permission record can created from here by clicking the `New` button, which displays a form into which user details may be entered:

  image::screenshots/Edit-user-example.png[alt="User permissions editing view"]

This form, works like most other Annalist data forms: enter details and click `Save` to create a new user permissions record.

A similar pattern applies for copying, editing or deleting existing user permissions.

==== Setting site-wide permissions

Annalist site-wide permissions are set using the annalist_manager tool, and cannot be set through the web interface.  The relevant commands are:

----
    annalist_manager createadminuser [ username [ email [ firstname [ lastname ] ] ] ] [ CONFIG ]
    annalist_manager updateadminuser [ username ] [ CONFIG ]
    annalist_manager setdefaultpermissions [ permissions ] [ CONFIG ]
    annalist_manager setpublicpermissions [ permissions ] [ CONFIG ]
    annalist_manager deleteuser [ username ] [ CONFIG ]
----

For more information about any command, enter:
<commit_msg>Adjust formatting<commit_after>= Guide to using Annalist
<NAME>
v0.1.15, 2015-08-04
:toc: macro
:toc-title:

This document provides an initial guide to using Annalist.  It assumes Annalist has been installed or deployed, and provides an overview ofconcepts involved and operations that can be performed using Annalist.

*Table of Contents*

toc::[]

== Concepts

[[annalist-concepts]]
image::figures/annalist-concepts.png[alt="Annalist concepts"]

=== Site

A site corresponds to an Annalist installation.  It may contain any number of data collections.

=== Collections

A *collection* is a grouping of data under a single administrative control (i.e. access control is applied at the level of a collection;  there is no fine-grained access control within a collection.)

=== Data records

A *data record* is a web-accessible unit of user data.   Each data record has an entity identifier and a type identifier;  the combination of collection identifier, type identifier and entity identifier uniquely identify a data record within a site.  Combined with the site's URI, this invormation allows for any data record to have a unique identity on the World Wide Web, which is fundamental to Annalist's support for linked data.

A data record may have any number of properties; new properties may be added at any time to any data record.  Each property is identified by a URI or CURIE (often referred to as URIs).  There may be multiple values associated with any property URI.  This representation of data records with arbitrary properties is at the heart of Annalist's flexibility, and is based on the RDF and JSON-LD data models.  By not imposing any predefined structure, the content of any data record can be varied as needs dictate.  Presented structure is defined, as needed, by views on a data record (see below).

The data type of a property value is not formally constrained.  Multiple instances of the property may have values with different data types.  In practice, the data type associated with a property is determined by the field type (see below) used in the views though which data is entered and/or displayed, and these will generally present and interpret the property value as a particular data type.  If different data types do appear for a single property URI, the view- and list-fields used should be able to handle them appropriately.


=== Record types

*Record types* provide a simple, coarse-grained way of distinguishing different types of record from a user perspective, and associating with different content presentation as appropriate.  Each data record has exactly one data type, which is used directly to help organize the underlying data storage.  The intent is that data types are used to ceate a top-level organization of different kinds of data within a collection.  Finer-grained and cross-cutting type distinctions may be made though properties on each data record.

In keeping with the idea that properties can be added to any data record at any tine, the description of a record type does not include any structural information.  Rather, it includes descriptive information, and association with default view and list presentations (other view and list presentations can be used, but must be explicitly selected).


=== Record views

A *record view* describes a selection and organization of properties to be used when displaying a single data record.  As such, a view establishes a presentation of a particular kind of structure on the data.  Different views may present different structures over the same underlying data.  A view is defined with a number of fields (see below, and also [View fields in Annalist](view-field-types.md)).


=== List views

A *list view* describes a selection and organization of data records and properties to be used for displaying multiple data records.  They are used to create summary displays from which individual data records may be selected.  A list is defined with a data record selector and a number of fields (see below).

Associated with each collection is a default list view, which is displayed when a collection is initially accessed.


=== Fields

Record and list views are build up using *fields*, which associate a data record property URI with a field renderer and a stored data value type.  A field render formats data for presentation (essentially as a fragment of HTML), and interprets user input as a storable data value.

The available field renderers are defined by Annalist, and the range of rendering options is expected to increase as the software is developed.  Currently, there are three main types of renderer:  single-line text (Text), multiline text (TextArea) and enumeration, pwhich presents a selection of identifiers of some other defined type.

For more information, see [View fields in Annalist](view-field-types.md).


=== Field groups

Some types of field (notably those using `RepeatGroup` and `RepeatGroupRow` render types) make reference to a *Field group* (sometimes just called a *Group*).  A field group is a separately-defined list of fields that may be incorporated into a record view.

Field groups are central to the handling of fields that may appear multiple times in a view.  For example, to define a bibliography record which may contain multiple author descriptions, the fields describing an author are defined as a frield group, and the bibliography record can refer to tghis field group using an appropriate render type.


=== Entities

Data records, types, views, lists, fields and more are all handled internally as *entities*, which are always associated with a data type.  Data types, views, lists, fields and groups are associated with the special data types `_type`, `_view`, `_list`, `_field` and `_group` respectively.  Using these predefined types, their descriptions can be processed using the same facilities that are used to create, modify and view user data records.

Additional special data types may be added as new features are implemented.  A convention has been adopted that the special data types have type identifiers that beging with an underscore (`'_'`), so to avoid clashes user data types and entities should be created with idnetifiers that do not start with an underscore.

Entities are stored as flat files in JSON (JSON-LD) format.


=== Site-wide data

A site has been described as consisting of a number of collections containing user-defined entities (data records, data types, views, lists, etc).

There is also provision for site-wide entities, which is used mainly for the predefined Annalist special types (`_type`, '_view`, `_list`, etc., and also for some predefined instances of these), but which can be used for any data.  All site-wide values are available to all collections in a site.  If a collection creates a local entity that is also available as a site-wide entity, the local value takes precedence for that collection. 

Site-wide data is fixed for an installation:  it cannot be created or modified through the Annalist user interface.  Certain operations may appear to modify a site-wide value (e.g. editing a default record view): these cause the site-wide dscription to be copied into the collection and updated there.  Thus, the view `default_view`


== Accessing Annalist

The following instructions assume Annalist service running on host `annalist.example.org`.  If a different host is used, replace `annalist.example.org` with the name or IP address of the host that is running the Annalist server.

. Browse to annalist server at `http://annalist.example.org:8000`
+
An empty list of collections should be displayed, along with some help text:
+
image::screenshots/Front-page-initial.png[alt="Initial front page"]

. Select the *Login* item from the top menu bar:
+
image::screenshots/Login-initial.png[alt="Initial login page"]

. Log in to Annalist using one of the procedures described belo.]

. After logged in, click the *Home* link on the to menu bar to return to the front page:
+
image::screenshots/Front-page-admin.png[alt="Initial front page after login"]
+
Note that the front page now shows text entry boxes and a button for creating a new collection.


=== Log in with OpenID Connect credentials

The intended means of Annalist user authentication is via a third party identity provider (IDP) service:

. Enter a local username (which is how Annalist will identify you, independently of any third party authentication account you may use), select an IDP (e.g. 'Google') and click *Login*.

. If you are not already logged in to the IDP you will be asked to login via their site.  Then the IDP will ask your permission to disclose basic identifying information (email, first name and last name) to Annalist.  This step is skipped if you have completed these actions previously.

. If this is an existing Annalist account, and the email from the IDP matches the Annalist account email, you will be logged in.  If the username given does not match an existing Annalist account, a new account is created with the appropriate details and you are logged in to it.  Being logged in does not necessarily mean you have permissions to access Annalist data;  it simply means that Annalist has an indicator of who you are.

(@@NOTE: Currently, the authorization system is not fully implememted, and all authenticated users have full access to the Annalist data, but that will change before the first non-prototype software release.  See https://github.com/gklyne/annalist/issues/11)[Annalist issue 11])


=== Log in with local user credentials

(See below for creating local user credentials)

. Select the "Local user credentials: *login*" link at the bottom of the page:
+
image::screenshots/Login-django.png[alt="Django login page"]

. Enter a previously created username and password

. Click on the *Login* button


=== Creating local user credentials

The Annalist software installation includes a step to create an admin user:

----
    annalist-manager createadminuser
----

The username and password created at this step can be used as local credentials to log in to Annalist as described above.

Having logged in using the admin username and password, the *Admin* link in the page footer can be used to create additional local users via the local administrative interface (which is implemented in the underlying Django web application framework).  When logged in to Annalist using this account, the *Admin* link in the footer of most Annalist pages will allow new user accounts to be created via the Django admin interface.  More documentation about using this admin interface can be found in the http://www.djangobook.com/en/2.0/chapter06.html[The Django Admin Site], which is Chapter 6 of http://www.djangobook.com/en/2.0/index.html[The Django Book].


=== Access control and user permissions

Being logged in to Annalist does not, of itself, grant permission to access or modify any Annalist data - except in a very broad sense covered by `_default_user_perms` (see below).  That is controlled separately by _user permission_ records, which may be created for any collection ovcerv which the user is granted access permissions.

When a user is logged in to Annalist (using OpenID or local credentials), their session is associated with a user identifier (user_id) and an email address, as well as some descriptive information.  The user_id alone is not a secure basis for access control, as Annalist allows anyone to create a new user_id through the OpenID based login process.  Rather, it is assumed that the associated email address has been properly checked as associated with the logged-in user:  this verification is something that should be handled by any OpenID identity provider (such as Google) worthy of being used as such.  Annalist uses the combination of user_id and email address as a reliable authenticated identifier, with which access permissions are associated.  Different user_ids may bve associated with the same email address, possibly corresponding to different roles of the authenticated user;  each such combination is associated with its own set of permissions. (Technically, Annalist internally uses a mailto: URI as an authenticated user identifier, and uses different forms of URI for some special cases.)

Permissions granted to a user consist of a list of tokens denoting different access rights; e.g. VIEW, CREATE, UPDATE, DELETE, etc.  Permissions are applied at the level of a collection;  a user may have different permissions in different collections.  The permisisons granted form part of the collection metadata, and as such are replicated if an entire collection is copied or moved to a different service.  It is also possible to associate permissions with an entire Annalist site, which means they are applied in any collection for which more specific permissions are not granted;  the user interface for creating site-wide permissions currently takes the form of a limited set of `annalist_manager` commands, and is intended to be used for setting up site-wide defaults and administrative access.

Built-in Annalist permission tokens include:

`ADMIN` - required to create or view user permissions in a collection.  The creator of a collection is automatically granted `ADMIN` permissions over that collection, so they can assign permissions in that collection for other users.  This permission at site level also allows creation and deletion of collections.

`CONFIG` - required to change the structure of a collection: to create and/or modify record types, views, lists, etc.

`CREATE` - required to create new data in a collection.

`UPDATE` - required to edit data records in a collection.

`VIEW` - required to view or read data records in a collection.

`DELETE` - required to remove data records in a collection.

`CREATE_COLLECTION` - this permission, or `ADMIN`, is required at site level to create a new collection.

`DELETE_COLLECTION` - this permission, or `ADMIN`, is required at site level to remove an existing collection.

As noted, permissions (currently) apply to the entire content of an annalist Collection; i.e. a collection is the level of granularity at which access control is enforced.  (The internal design of the authorization system allows for new user-defined tokens to be introduced, and potentially to be associated with record types.  This could be used to provide sub-collection granularity of access control - e.g. to restrict access to a record type that may contain particularly sensitive data.)

==== Creating user permissions in a collection

A logged in user with ADMIN permissions over a collection may create further collection-level user permissions from the "List users" view of a collection, which is accessed from the default collection view by selecting `User_list` from the `List view` dropdown, then clicking the `View` button:

image::screenshots/List-users-example.png[alt="List users view"]

A new user permission record can created from here by clicking the `New` button, which displays a form into which user details may be entered:

image::screenshots/Edit-user-example.png[alt="User permissions editing view"]

This form, works like most other Annalist data forms: enter details and click `Save` to create a new user permissions record.

A similar pattern applies for copying, editing or deleting existing user permissions.

==== Setting site-wide permissions

Annalist site-wide permissions are set using the annalist_manager tool, and cannot be set through the web interface.  The relevant commands are:

----
    annalist_manager createadminuser [ username [ email [ firstname [ lastname ] ] ] ] [ CONFIG ]
    annalist_manager updateadminuser [ username ] [ CONFIG ]
    annalist_manager setdefaultpermissions [ permissions ] [ CONFIG ]
    annalist_manager setpublicpermissions [ permissions ] [ CONFIG ]
    annalist_manager deleteuser [ username ] [ CONFIG ]
----

For more information about any command, enter:

{'fraction_non_alphanumeric': 0.04615964328330088, 'fraction_numerical': 0.001695660365509012, 'mean_word_length': 4.54053583855254, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5260116', 'n_tokens_mistral': 7563, 'n_tokens_neox': 7313, 'n_words': 4721}