<commit_before>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Ckpt-Server}
Installing a Checkpoint Server}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The Checkpoint Server is a repository for checkpoint files.
Using checkpoint servers reduces the disk requirements of submitting
machines in the pool since they no longer need to store checkpoint
files locally.
Checkpoint server machines should have a large amount of disk space
available, and should have a fast connection to machines in the Condor
pool.

If your spool directories are on a network file system, then
checkpoint files will make two trips over the network, one between the
submission machine and the execution machine and a second between the
submit machine and the network file server.
If you install a checkpoint server and configure it to use the
server's local disk, the checkpoint will travel only once over the
network, between the execution machine and the checkpoint server.
You may also obtain checkpointing network performance benefits by
using multiple checkpoint servers, as discussed below.

\Note It is a good idea to pick very stable machines for your checkpoint
servers.
If the checkpoint servers crash, the Condor system will continue to
operate, though poorly.  
While the Condor system will recover from a checkpoint server crash
as best it can, there are two problems that can (and will) occur:
\begin{enumerate}

\item If the checkpoint server is not functioning, when jobs need to
checkpoint, they cannot do so.
The jobs will keep trying to contact the checkpoint server, backing
off exponentially in the time they wait between attempts.
Normally, jobs only have a limited time to checkpoint before they are
kicked off the machine.
So, if the server is down for a long period of time, chances are that
you'll lose a lot of work by jobs being killed without writing a
checkpoint. 

\item When the jobs wish to start, if their checkpoints cannot be
retrieved from 
the checkpoint server, they will either have to be restarted from
scratch, or the job will wait for the server to come back on-line.
You can control this behavior with the
\Macro{MAX\_DISCARDED\_RUN\_TIME} parameter in your config file (see
section~\ref{sec:Checkpoint-Server-Config-File-Entries} on
page~\pageref{sec:Checkpoint-Server-Config-File-Entries} for details).
Basically, this represents the maximum amount of CPU time you're
willing to discard by starting a job over from scratch if the
checkpoint server isn't responding to requests.

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Prepare-Ckpt-Server}
Preparing to Install a Checkpoint Server} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Because of the problems that exist if your pool is configured to use a
checkpoint server and that server is down, it is advisable to shut
your pool down before doing any maintenance on your checkpoint
server.  
See section~\ref{sec:Pool-Management} on
page~\pageref{sec:Pool-Management} for details on how to do that. 

When modifying the checkpoint server configuration of a submission
machine, you must make sure there are no jobs currently in the queue
on that machine.
If you have jobs in your queues, with checkpoint files on the local
spool directories of your submit machines, those jobs will never run
if your submit machines are configured to use a checkpoint server and
the checkpoint files cannot be found on the server.  
You can either remove jobs from your queues or let them complete
before you configure those submission machines with non-empty job
queues.
However, you may proceed and install the checkpoint server,
configuring only those submission machines with empty queues and
postponing the configuration of submission machines with non-empty job
queues until the queues are empty.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Install-Ckpt-Server-Module}
Installing the Checkpoint Server Module} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To install a checkpoint server, download the appropriate binary
contrib module for the platform(s) your server will run on.
When you uncompress and untar the file, you'll have a directory that
contains a \File{README}, \File{ckpt\_server.tar}, and so on.
The \File{ckpt\_server.tar} acts much like the \File{release.tar} file
from a main release.
This archive contains these files:
\begin{verbatim}
        sbin/condor_ckpt_server
        sbin/condor_cleanckpts
        etc/examples/condor_config.local.ckpt.server
\end{verbatim}
These are all new files, not found in the main release, so you can
safely untar the archive directly into your existing release
directory. 
\File{\condor{ckpt\_server}} is the checkpoint server binary.
\File{\condor{cleanckpts}} is a script that can be periodically run to
remove stale checkpoint files from your server.  
Normally, the checkpoint server cleans all old files by itself.  
However, in certain error situations, stale files can be left that are
no longer needed. 
So, you may want to put a cron job in place that calls
\Condor{cleanckpts} every week or so, just to be safe.
The example config file is described below.

Once you have unpacked the contrib module, you have a few more steps
you must complete.
Each will be discussed in their own section:
\begin{enumerate}
\item Configure the checkpoint server.
\item Spawn the checkpoint server.
\item Configure your pool to use the checkpoint server.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Configure-Ckpt-Server}
Configuring a Checkpoint Server} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

There are a few settings you must place in the local config file of
your checkpoint server.  
The file \File{etc/examples/condor\_config.local.ckpt.server} contains
all such settings, and you can just insert it into the local
configuration file of your checkpoint server machine. 

There is one setting that you must customize, and that is
\Macro{CKPT\_SERVER\_DIR}.  
\label{param:CkptServerDir}
The \Macro{CKPT\_SERVER\_DIR} defines where your checkpoint files
should be located. 
It is better if this is on a very fast local file system (preferably a
RAID). 
The speed of this file system will have a direct impact on the speed
at which your checkpoint files can be retrieved from the remote
machines. 

The other optional settings are:
\begin{description}<commit_msg>remove duplicate definition of \label{param:CkptServerDir}<commit_after>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Ckpt-Server}
Installing a Checkpoint Server}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The Checkpoint Server is a repository for checkpoint files.
Using checkpoint servers reduces the disk requirements of submitting
machines in the pool since they no longer need to store checkpoint
files locally.
Checkpoint server machines should have a large amount of disk space
available, and should have a fast connection to machines in the Condor
pool.

If your spool directories are on a network file system, then
checkpoint files will make two trips over the network, one between the
submission machine and the execution machine and a second between the
submit machine and the network file server.
If you install a checkpoint server and configure it to use the
server's local disk, the checkpoint will travel only once over the
network, between the execution machine and the checkpoint server.
You may also obtain checkpointing network performance benefits by
using multiple checkpoint servers, as discussed below.

\Note It is a good idea to pick very stable machines for your checkpoint
servers.
If the checkpoint servers crash, the Condor system will continue to
operate, though poorly.  
While the Condor system will recover from a checkpoint server crash
as best it can, there are two problems that can (and will) occur:
\begin{enumerate}

\item If the checkpoint server is not functioning, when jobs need to
checkpoint, they cannot do so.
The jobs will keep trying to contact the checkpoint server, backing
off exponentially in the time they wait between attempts.
Normally, jobs only have a limited time to checkpoint before they are
kicked off the machine.
So, if the server is down for a long period of time, chances are that
you'll lose a lot of work by jobs being killed without writing a
checkpoint. 

\item When the jobs wish to start, if their checkpoints cannot be
retrieved from 
the checkpoint server, they will either have to be restarted from
scratch, or the job will wait for the server to come back on-line.
You can control this behavior with the
\Macro{MAX\_DISCARDED\_RUN\_TIME} parameter in your config file (see
section~\ref{sec:Checkpoint-Server-Config-File-Entries} on
page~\pageref{sec:Checkpoint-Server-Config-File-Entries} for details).
Basically, this represents the maximum amount of CPU time you're
willing to discard by starting a job over from scratch if the
checkpoint server isn't responding to requests.

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Prepare-Ckpt-Server}
Preparing to Install a Checkpoint Server} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Because of the problems that exist if your pool is configured to use a
checkpoint server and that server is down, it is advisable to shut
your pool down before doing any maintenance on your checkpoint
server.  
See section~\ref{sec:Pool-Management} on
page~\pageref{sec:Pool-Management} for details on how to do that. 

When modifying the checkpoint server configuration of a submission
machine, you must make sure there are no jobs currently in the queue
on that machine.
If you have jobs in your queues, with checkpoint files on the local
spool directories of your submit machines, those jobs will never run
if your submit machines are configured to use a checkpoint server and
the checkpoint files cannot be found on the server.  
You can either remove jobs from your queues or let them complete
before you configure those submission machines with non-empty job
queues.
However, you may proceed and install the checkpoint server,
configuring only those submission machines with empty queues and
postponing the configuration of submission machines with non-empty job
queues until the queues are empty.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Install-Ckpt-Server-Module}
Installing the Checkpoint Server Module} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To install a checkpoint server, download the appropriate binary
contrib module for the platform(s) your server will run on.
When you uncompress and untar the file, you'll have a directory that
contains a \File{README}, \File{ckpt\_server.tar}, and so on.
The \File{ckpt\_server.tar} acts much like the \File{release.tar} file
from a main release.
This archive contains these files:
\begin{verbatim}
        sbin/condor_ckpt_server
        sbin/condor_cleanckpts
        etc/examples/condor_config.local.ckpt.server
\end{verbatim}
These are all new files, not found in the main release, so you can
safely untar the archive directly into your existing release
directory. 
\File{\condor{ckpt\_server}} is the checkpoint server binary.
\File{\condor{cleanckpts}} is a script that can be periodically run to
remove stale checkpoint files from your server.  
Normally, the checkpoint server cleans all old files by itself.  
However, in certain error situations, stale files can be left that are
no longer needed. 
So, you may want to put a cron job in place that calls
\Condor{cleanckpts} every week or so, just to be safe.
The example config file is described below.

Once you have unpacked the contrib module, you have a few more steps
you must complete.
Each will be discussed in their own section:
\begin{enumerate}
\item Configure the checkpoint server.
\item Spawn the checkpoint server.
\item Configure your pool to use the checkpoint server.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Configure-Ckpt-Server}
Configuring a Checkpoint Server} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

There are a few settings you must place in the local config file of
your checkpoint server.  
The file \File{etc/examples/condor\_config.local.ckpt.server} contains
all such settings, and you can just insert it into the local
configuration file of your checkpoint server machine. 

There is one setting that you must customize, and that is
\Macro{CKPT\_SERVER\_DIR}.  
The \Macro{CKPT\_SERVER\_DIR} defines where your checkpoint files
should be located. 
It is better if this is on a very fast local file system (preferably a
RAID). 
The speed of this file system will have a direct impact on the speed
at which your checkpoint files can be retrieved from the remote
machines. 

The other optional settings are:
\begin{description}
{'fraction_non_alphanumeric': 0.12434998869545558, 'fraction_numerical': 0.0, 'mean_word_length': 5.137835337650324, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1854972', 'n_tokens_mistral': 3297, 'n_tokens_neox': 3120, 'n_words': 1770}