<commit_before># Data Manager

The Data Manager service runs as part of the Observing Database at each summit, interacting with
the Gemini archive and FITS storage servers to report dataset status and synchronize QA state.  This
document provides a user-focused look at what the Data Manager does and how it works, and provides
insight into understanding the status messages that are presented in the OT.


## Overview

To understand the Data Manager, it is helpful to put it in context with the FITS storage server,
public archive and Observing Tool.  The picture below represents the software systems involved and
their interactions at a basic level.

![](images/dataman.png)

The Data Manager is colocated with the Observing Database so it can easily read and update any
program.  It is charged with keeping the QA state in sync between ODB records and summit storage.
This is important because QA state plays a role in time accounting, the results of which are
presented in both the Observing Tool and various reports.

The FITS storage server handles all interactions with the FITS files themselves.  It keeps the
FITS files on disk in sync with its own internal representation and maintains the dataset in sync
with the version available in the public archive.  It is important to underscore the difference
between the FITS file itself and the record of it maintained in the FITS storage server.  The two
can be momentarily out of sync while the storage server is busy handling an update.  This topic
comes up again in the following discussion of QA state updates and when detailing the status
messages displayed in the Observing Log UI.

Finally the OT presents a user interface that reveals the current status of datasets in the system
and allows the user to update the QA state.  The OT communicates with the ODB / Data Manager
directly and leaves database record maintenance to the Data Manager.


### Updating QA State

There are two ways that a user can modify QA states:

1. Use the OT and its Observing Log GUI.
2. Work directly with FITS files in the summit storage system.  

We'll cover the second alternative below in the discussion on polling.  Using the Observing Log GUI
in the OT works just like updating anything else in the science program.  Namely, the user edits the
QA state by selecting the desired value in the provided widget and then stores the change in the
ODB via the program synchronization feature.  Until changes are synchronized with the database,
they will not be forwarded on to the FITS Storage server.

The Data Manager monitors the ODB for updates to QA state.  When it detects a change, it posts
a corresponding request to the FITS storage server.  The storage server in turn immediately accepts
the request and updates the dataset file on disk, or else responds with an error.  Acceptance of an
update request is a promise on the part of the storage server to eventually update its internal 
records to match.  Until that promise is fulfilled, a query for the current QA state of the
dataset will continue to report the value it had *before* the update took place.  Presumably at
some later point, the storage server records are updated and the modified dataset is forwarded on to
the public archive.  Throughout the process, the Data Manager keeps the dataflow status up-to-date
in the database.

### Polling

Neither the Data Manager nor the FITS storage server provide events when dataset records are
updated.  For this reason, current dataset status is always updated via polling.  The Data Manager
polls the storage server and applies any necessary changes to ODB records.  The OT polls the
current dataset status in the database and updates the local copy of the program that it is
editing.

In the OT, the poll period depends upon user privileges and program status.  In particular the
polling rates are:

<table>
  <caption>OT/ODB Poll Periods</caption>
  <tr><th>Program Status</th><th>Privileges</th><th>Poll Period</th></tr>
  <tr><td>Active</td><td>Staff</td><td>10 seconds</td></tr>
  <tr><td>Active</td><td>Non-Staff</td><td>1 hour</td></tr>
  <tr><td>Inactive</td><td>Staff</td><td>5 minutes</td></tr>
  <tr><td>Inactive</td><td>Non-Staff</td><td>1 day</td></tr>
</table>

To keep the network load to a minimum, the OT transmits a hash of the expected poll results.
The ODB checks the hash against the current poll results and if nothing has changed, returns
a minimal result.

The Data Manager polls both the local FITS storage server and the public archive for dataset status
information. Like the OT, the poll period is variable.  The polling rates are:

<table>
  <caption>Data Manager/Archive Poll Periods</caption>
  <tr><th>Category</th><th>Poll Period</th></tr>
  <tr><td>Tonight's Datasets</td><td>1 minute</td></tr>
  <tr><td>This Week's Datasets</td><td>15 minutes</td></tr><commit_msg>REL-2407: clarify OT polling period in doc<commit_after># Data Manager

The Data Manager service runs as part of the Observing Database at each summit, interacting with
the Gemini archive and FITS storage servers to report dataset status and synchronize QA state.  This
document provides a user-focused look at what the Data Manager does and how it works, and provides
insight into understanding the status messages that are presented in the OT.


## Overview

To understand the Data Manager, it is helpful to put it in context with the FITS storage server,
public archive and Observing Tool.  The picture below represents the software systems involved and
their interactions at a basic level.

![](images/dataman.png)

The Data Manager is colocated with the Observing Database so it can easily read and update any
program.  It is charged with keeping the QA state in sync between ODB records and summit storage.
This is important because QA state plays a role in time accounting, the results of which are
presented in both the Observing Tool and various reports.

The FITS storage server handles all interactions with the FITS files themselves.  It keeps the
FITS files on disk in sync with its own internal representation and maintains the dataset in sync
with the version available in the public archive.  It is important to underscore the difference
between the FITS file itself and the record of it maintained in the FITS storage server.  The two
can be momentarily out of sync while the storage server is busy handling an update.  This topic
comes up again in the following discussion of QA state updates and when detailing the status
messages displayed in the Observing Log UI.

Finally the OT presents a user interface that reveals the current status of datasets in the system
and allows the user to update the QA state.  The OT communicates with the ODB / Data Manager
directly and leaves database record maintenance to the Data Manager.


### Updating QA State

There are two ways that a user can modify QA states:

1. Use the OT and its Observing Log GUI.
2. Work directly with FITS files in the summit storage system.  

We'll cover the second alternative below in the discussion on polling.  Using the Observing Log GUI
in the OT works just like updating anything else in the science program.  Namely, the user edits the
QA state by selecting the desired value in the provided widget and then stores the change in the
ODB via the program synchronization feature.  Until changes are synchronized with the database,
they will not be forwarded on to the FITS Storage server.

The Data Manager monitors the ODB for updates to QA state.  When it detects a change, it posts
a corresponding request to the FITS storage server.  The storage server in turn immediately accepts
the request and updates the dataset file on disk, or else responds with an error.  Acceptance of an
update request is a promise on the part of the storage server to eventually update its internal 
records to match.  Until that promise is fulfilled, a query for the current QA state of the
dataset will continue to report the value it had *before* the update took place.  Presumably at
some later point, the storage server records are updated and the modified dataset is forwarded on to
the public archive.  Throughout the process, the Data Manager keeps the dataflow status up-to-date
in the database.

### Polling

Neither the Data Manager nor the FITS storage server provide events when dataset records are
updated.  For this reason, current dataset status is always updated via polling.  The Data Manager
polls the storage server and applies any necessary changes to ODB records.  The OT polls the
current dataset status in the database and updates the local copy of the program that it is
editing.

In the OT, the poll period depends upon user privileges and whether the program is currently being
displayed in an OT window.  In particular the polling rates are:

<table>
  <caption>OT/ODB Poll Periods</caption>
  <tr><th>Displayed?</th><th>Privileges</th><th>Poll Period</th></tr>
  <tr><td>Yes</td><td>Staff</td><td>10 seconds</td></tr>
  <tr><td>Yes</td><td>Non-Staff</td><td>1 hour</td></tr>
  <tr><td>No</td><td>Staff</td><td>5 minutes</td></tr>
  <tr><td>No</td><td>Non-Staff</td><td>1 day</td></tr>
</table>

Displayed means that the program is open in an OT window, not just loaded in its history and
certainly not just in the local database.  Any program not currently visible in an OT window
is counted as not displayed.

To keep the network load to a minimum, the OT transmits a hash of the expected poll results.
The ODB checks the hash against the current poll results and if nothing has changed, returns
a minimal result.

The Data Manager polls both the local FITS storage server and the public archive for dataset status
information. Like the OT, the poll period is variable.  The polling rates are:

<table>
  <caption>Data Manager/Archive Poll Periods</caption>
  <tr><th>Category</th><th>Poll Period</th></tr>
  <tr><td>Tonight's Datasets</td><td>1 minute</td></tr>
  <tr><td>This Week's Datasets</td><td>15 minutes</td></tr>
{'fraction_non_alphanumeric': 0.05106638630219285, 'fraction_numerical': 0.0024031240612796636, 'mean_word_length': 4.443051771117166, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 133, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6191996', 'n_tokens_mistral': 2571, 'n_tokens_neox': 2464, 'n_words': 1530}