<commit_before>\chapter{Iterative methods}
\label{sec:iterative_methods}

The following sections deal with the methods of Iterative Boltzmann Inversion
(\ibi), Inverse Monte Carlo (\imc), and Relative Entropy (\re).

In general, \ibi, \imc, and \re are implemented within the same
framework. Therefore, most settings and parameters of those methods are similar
and thus described in a general section (see
sec. \ref{sec:iterative_methods_imc}). Further information on iterative methods
follows in the next chapters, in particular on the \ibi, \imc, and \re methods.

\begin{figure}[h]
   \centering
   \includegraphics[width=7cm]{usage/fig/flow_ibi.eps}
   \caption{\label{fig:flow_ibi}Flowchart to perform iterative Boltzmann inversion.}
\end{figure}

\section{Iterative workflow control}
\label{sec:iterative_workflow}

\begin{figure}[t]
   \centering
   \includegraphics[width=7cm]{functionality/fig/flowchart.eps}
   \caption{\label{fig:flowchart}Block-scheme of the workflow control for the iterative methods. The most time-consuming parts are marked in red.}
\end{figure}

Iterative workflow control is essential for the \ibi, \imc, and \re methods.

The general idea of iterative workflow is sketched in fig.~\ref{fig:flowchart}. During the global initialization the initial guess for the coarse-grained potential is calculated from the reference function or converted from a given potential guess into the internal format. The actual iterative step starts with an iteration initialization. It searches for possible checkpoints and copies and converts files from the previous step and the base directory. Then, the simulation run is prepared by converting potentials into the format required by the external sampling program and the actual sampling is performed.

After sampling the phasespace, the potential update is calculated. Often, the update requires postprocessing, such as smoothing, interpolation, extrapolation or fitting to an analytical form.

Finally, the new potential is determined and postprocessed. If the iterative process continues, the next iterative step will start to initialize.

\subsubsection*{How to start:}

The first thing to do is generate reference distribution functions. These might come from experiments or from atomistic simulations. To get reasonable results out of the iterative process, the reference distributions should be of good quality (little noise, etc).

\votca can create initial guesses for the coarse-grained
potentials by boltzmann inverting the distribution function. If a custom initial
guess for an interaction shall be used instead, the table can be provided in
\textit{$<$interaction$>$.pot.in}. As already mentioned, \votca automatically creates potential tables to run a simulation. However, it does not know how to run a coarse-grained simulation. Therefore, all files needed to run a coarse-grained simulation, except for the potentials that are iteratively refined, must be provided and added to the \hyperlink{\cgref{inverse.filelist}}{filelist} in the settings \xml-file. If an atomistic topology and a mapping definition are present, \votca offers tools to assist the setup of a  coarse-grained topology (see chapter \ref{sec:usage:cgrun}).

To get an overview of how input files look like, it is suggested to take a look at one of the tutorials provided on \votcaweb.

In what follows we describe how to set up the iterative coarse-graining, run the main script, continue the run, and add customized scripts.

\subsection{Preparing the run}
\label{sec:preparing_the_run}
To start the first iteration, one has to prepare the input for the sampling program. This means that all files for running a coarse-grained simulation must be present and described in a separate \xml file, in our case \texttt{settings.xml} (see sec. \ref{sec:setting_files} for details). An extract from this file is given below. The only exception are tabulated potentials, which will be created and updated by the script in the course of the iterative process.

The input files include: target distributions, initial guess (optional) and a list of interactions to be iteratively refined. As a target distribution, any table file can be given (e.g. \gromacs output from \texttt{g\_rdf}). The program automatically takes care to resample the table to the correct grid spacing according to the options provided in \texttt{settings.xml}.

The initial guess is normally taken as a potential of mean force and is generated by Boltzmann-inversion of the corresponding distribution function. It is written in \texttt{step\_000/<name>.pot.new}. If you want to manually specify the initial guess for a specific interaction, write the potential table to a file called \texttt{<name>.pot.in} in the folder where you plan to run the iterative procedure.

A list of interactions to be iteratively refined has to be given in the options file. As an example, the \texttt{setting.xml} file for a propane is shown in listing~\ref{list:settings}. For more details,  see the full description of all options in ref.~\ref{sec:ref_options}.
\begin{figure}
\centering
\framebox[\textwidth]{\lstinputlisting{functionality/settings.xml}
}
\caption{\texttt{settings.xml} file specifies interactions to be refined, grid spacings, sampling engine, and the iterative method. The complete file can be found in the \texttt{propane/ibm} tutorial. 
\label{list:settings}
}
\end{figure}

\subsection{Starting the iterative process}
\label{sec:starting_iterative_process}
After all input files have been set up, the run can be started by
\begin{verbatim}
  csg_inverse --options settings.xml
\end{verbatim}

Each iteration is stored in a separate directory, named \texttt{step\_<iteration>}. \texttt{step\_000} is a special folder which contains the initial setup. For each new iteration, the files required to run the CG simulation (as specified in the config file) are copied to the current working directory. The updated potentials  are copied from the last step, \texttt{step\_<n-1>/<interaction>.pot.new}, and used as the new working potentials \texttt{step\_<n>/<interaction>.pot.cur}.

After the run preparation, all potentials are converted into the format of the sampling program and the simulation starts. Once the sampling has finished, analysis programs generate new distributions, which are stored in \texttt{<interaction>.dist.new}, and new potential updates, stored in \texttt{<interaction>.dpot.new}. 

Before adding the update to the old potential, it can be processed in the \texttt{post\_update} step. For each script that is specified in the postupdate, \texttt{<interaction>.dpot.new} is renamed  to \texttt{<interaction>.dpot.old} and stored in \texttt{<interaction>.dpot.<a-number>} before the processing script is called. Each processing script  uses the current potential update \texttt{<interaction>.dpot.cur} and writes the processed update to \texttt{<interaction>.dpot.new}. As an example, a pressure correction is implemented as a postupdate script within this framework.

After all postupdate scripts have been called, the update is added to the potential and the new potential \texttt{<interaction>.pot.new} is written. Additional post-processing of the potential can be performed in the \texttt{post\_add} step which is analogous to the \texttt{post\_update} step except for a potential instead of an update.

To summarize, we list all standard output files for each iterative step:

\begin{tabular}{ll}
\texttt{*.dist.new} & distribution functions of the current step \\
\texttt{*.dpot.new} & the final potential update, created by \texttt{calc\_update} \\
\texttt{*.dpot.<number>} & for each postupdate script, the \texttt{.dpot.new} is saved and a new one\\
&is created\\
\texttt{*.pot.cur} & the current potential used for the actual run\\
\texttt{*.pot.new} & the new potential after the add step \\
\texttt{*.pot.<number>} & same as \texttt{dpot.<number>} but for \texttt{post\_add}
\end{tabular}

If a sub-step fails during the iteration, additional information can be found in the log file. The name of the log file is specified in the steering \xml file.

\subsection{Restarting and continuing}
The interrupted or finished iterative process can be restarted either by extending a finished run or by restarting the interrupted run. When the script \prog{csg_inverse} is called, it automatically checks for a file called \texttt{done} in the current directory. If this file is found, the program assumes that the run is finished. To extend the run, simply increase \cgopt{inverse.iterations_max} in the settings file and remove the file called \texttt{done}. After that, \prog{csg_inverse} can be restarted, which will automatically recognize existing steps and continue after the last one.

If the iteration was interrupted, the script \prog{csg_inverse} might not be able to restart on its own. In this case, the easiest solution is to delete the last step and start again. The script will then repeat the last step and continue. However, this method is not always practical since sampling and analysis might be time-consuming and the run might have only crashed due to some inadequate post processing option. To avoid repeating the entire run, the script \prog{csg_inverse} creates a file with restart points and labels already completed steps such as simulation, analysis, etc. The file name is specified in the option \cgopt{inverse.restart_file}. If specific actions should be redone, one can simply remove the corresponding lines from this file. Note that a file \texttt{done} is also created in each folder for those steps which have been successfully finished.


% #####################################################################################################################
% #####################################################################################################################

\section{Iterative Boltzmann Inversion}
\subsection{Input preparation}
This section describes the usage of \ibi, implemented within the scripting framework described in the previous section \ref{sec:iterative_workflow}. It is suggested to get a basic understanding of this framework before proceeding.

An outline of the workflow for performing \ibi is given in \fig{fig:flow_ibi}.

To specify Iterative Boltzmann Inversion as algorithm in the script, add \texttt{ibi} in the \texttt{method} section of the \xml setting file as shown below.

\begin{lstlisting}
  <cg>
    ...
    <inverse>
      <method>ibi</method>
    </inverse>
  </cg>
\end{lstlisting}


% #####################################################################################################################
% #####################################################################################################################

\section{Inverse Monte Carlo}
\label{sec:iterative_methods_imc}
In this section, additional options are described to run \imc coarse graining. The usage of \imc is similar to the one of \ibi and understanding the use of the scripting framework described in chapter~\ref{sec:iterative_workflow} is necessary.

\textbf{WARNING: multicomponent \imc is still experimental!}

\subsection{General considerations}
In comparison to \ibi, \imc needs significantly more statistics to calculate the potential update\cite{Ruehle:2009.a}. It is advisable to perform smoothing on the potential update. Smoothing can be performed as described in \sect{ref:ibi:optimize}. In addition, \imc can lead to problems related to finite size: for methanol, an undersized system proved to lead to a linear shift in the potential\cite{Ruehle:2009.a}. It is therefore always necessary to check that the system size is sufficiently large and that runlength csg smoothing iterations are well balanced.

\subsection{Additional mapping for statistics}
The program \prog{csg_stat} is used for evaluating the \imc matrix. Although the matrix only acts on the coarse-grained system here, it still needs a mapping file to work. This will improve with one of the next releases to simplify the setup. The mapping file needs to be a one to one mapping of the coarse grained system, e.g. for coarse graining \spce water, the mapping file looks as follows:
\begin{lstlisting}
  </cg_molecule>
    <name>SOL</name> 
    <ident>SOL</ident>
    <topology>
      <cg_beads>
        <cg_bead>
          <name>CG</name>
          <type>CG</type>
          <mapping>A</mapping>
          <beads>
            1:SOL:CG 
          </beads>
        </cg_bead>
      </cg_beads>
    </topology>
    <maps>
      <map>
        <name>A</name>
        <weights>1</weights>
      </map>
    </maps>
  </cg_molecule>
\end{lstlisting}

\subsection{Correlation groups}
Unlike \ibi, \imc also takes cross-correlations of interactions into account in order to calculate the update. However, it might not always be beneficial to evaluate cross-correlations of all pairs of interactions. By specifying \interopt{inverse.imc.group}, \votca allows to define groups of interactions, amongst which cross-correlations are taken into account, where \interopt{inverse.imc.group} can be any name.

\begin{lstlisting}
  <non-bonded>
    <name>CG-CG</name>
    <type1>CG</type1>
    <type2>CG</type2>
    ...
    <imc>
      <group>solvent</group>
   </imc>
  </non-bonded>
  <non-bonded>
\end{lstlisting}

\subsection{Regularization}

To use the regularized version of IMC a $\lambda$ value $>0$ has to be specified by setting \interopt{inverse.imc.reg}. 
If set to $0$ (default value) the unregularized version of IMC is applied.
\begin{lstlisting}
 <non-bonded>
   <name>CG-CG</name>
   <type1>CG</type1>
   <type2>CG</type2>
    ...
   <inverse>
     <imc>
       <reg>300</reg><commit_msg>IMC: remove obsolete mapping section<commit_after>\chapter{Iterative methods}
\label{sec:iterative_methods}

The following sections deal with the methods of Iterative Boltzmann Inversion
(\ibi), Inverse Monte Carlo (\imc), and Relative Entropy (\re).

In general, \ibi, \imc, and \re are implemented within the same
framework. Therefore, most settings and parameters of those methods are similar
and thus described in a general section (see
sec. \ref{sec:iterative_methods_imc}). Further information on iterative methods
follows in the next chapters, in particular on the \ibi, \imc, and \re methods.

\begin{figure}[h]
   \centering
   \includegraphics[width=7cm]{usage/fig/flow_ibi.eps}
   \caption{\label{fig:flow_ibi}Flowchart to perform iterative Boltzmann inversion.}
\end{figure}

\section{Iterative workflow control}
\label{sec:iterative_workflow}

\begin{figure}[t]
   \centering
   \includegraphics[width=7cm]{functionality/fig/flowchart.eps}
   \caption{\label{fig:flowchart}Block-scheme of the workflow control for the iterative methods. The most time-consuming parts are marked in red.}
\end{figure}

Iterative workflow control is essential for the \ibi, \imc, and \re methods.

The general idea of iterative workflow is sketched in fig.~\ref{fig:flowchart}. During the global initialization the initial guess for the coarse-grained potential is calculated from the reference function or converted from a given potential guess into the internal format. The actual iterative step starts with an iteration initialization. It searches for possible checkpoints and copies and converts files from the previous step and the base directory. Then, the simulation run is prepared by converting potentials into the format required by the external sampling program and the actual sampling is performed.

After sampling the phasespace, the potential update is calculated. Often, the update requires postprocessing, such as smoothing, interpolation, extrapolation or fitting to an analytical form.

Finally, the new potential is determined and postprocessed. If the iterative process continues, the next iterative step will start to initialize.

\subsubsection*{How to start:}

The first thing to do is generate reference distribution functions. These might come from experiments or from atomistic simulations. To get reasonable results out of the iterative process, the reference distributions should be of good quality (little noise, etc).

\votca can create initial guesses for the coarse-grained
potentials by boltzmann inverting the distribution function. If a custom initial
guess for an interaction shall be used instead, the table can be provided in
\textit{$<$interaction$>$.pot.in}. As already mentioned, \votca automatically creates potential tables to run a simulation. However, it does not know how to run a coarse-grained simulation. Therefore, all files needed to run a coarse-grained simulation, except for the potentials that are iteratively refined, must be provided and added to the \hyperlink{\cgref{inverse.filelist}}{filelist} in the settings \xml-file. If an atomistic topology and a mapping definition are present, \votca offers tools to assist the setup of a  coarse-grained topology (see chapter \ref{sec:usage:cgrun}).

To get an overview of how input files look like, it is suggested to take a look at one of the tutorials provided on \votcaweb.

In what follows we describe how to set up the iterative coarse-graining, run the main script, continue the run, and add customized scripts.

\subsection{Preparing the run}
\label{sec:preparing_the_run}
To start the first iteration, one has to prepare the input for the sampling program. This means that all files for running a coarse-grained simulation must be present and described in a separate \xml file, in our case \texttt{settings.xml} (see sec. \ref{sec:setting_files} for details). An extract from this file is given below. The only exception are tabulated potentials, which will be created and updated by the script in the course of the iterative process.

The input files include: target distributions, initial guess (optional) and a list of interactions to be iteratively refined. As a target distribution, any table file can be given (e.g. \gromacs output from \texttt{g\_rdf}). The program automatically takes care to resample the table to the correct grid spacing according to the options provided in \texttt{settings.xml}.

The initial guess is normally taken as a potential of mean force and is generated by Boltzmann-inversion of the corresponding distribution function. It is written in \texttt{step\_000/<name>.pot.new}. If you want to manually specify the initial guess for a specific interaction, write the potential table to a file called \texttt{<name>.pot.in} in the folder where you plan to run the iterative procedure.

A list of interactions to be iteratively refined has to be given in the options file. As an example, the \texttt{setting.xml} file for a propane is shown in listing~\ref{list:settings}. For more details,  see the full description of all options in ref.~\ref{sec:ref_options}.
\begin{figure}
\centering
\framebox[\textwidth]{\lstinputlisting{functionality/settings.xml}
}
\caption{\texttt{settings.xml} file specifies interactions to be refined, grid spacings, sampling engine, and the iterative method. The complete file can be found in the \texttt{propane/ibm} tutorial. 
\label{list:settings}
}
\end{figure}

\subsection{Starting the iterative process}
\label{sec:starting_iterative_process}
After all input files have been set up, the run can be started by
\begin{verbatim}
  csg_inverse --options settings.xml
\end{verbatim}

Each iteration is stored in a separate directory, named \texttt{step\_<iteration>}. \texttt{step\_000} is a special folder which contains the initial setup. For each new iteration, the files required to run the CG simulation (as specified in the config file) are copied to the current working directory. The updated potentials  are copied from the last step, \texttt{step\_<n-1>/<interaction>.pot.new}, and used as the new working potentials \texttt{step\_<n>/<interaction>.pot.cur}.

After the run preparation, all potentials are converted into the format of the sampling program and the simulation starts. Once the sampling has finished, analysis programs generate new distributions, which are stored in \texttt{<interaction>.dist.new}, and new potential updates, stored in \texttt{<interaction>.dpot.new}. 

Before adding the update to the old potential, it can be processed in the \texttt{post\_update} step. For each script that is specified in the postupdate, \texttt{<interaction>.dpot.new} is renamed  to \texttt{<interaction>.dpot.old} and stored in \texttt{<interaction>.dpot.<a-number>} before the processing script is called. Each processing script  uses the current potential update \texttt{<interaction>.dpot.cur} and writes the processed update to \texttt{<interaction>.dpot.new}. As an example, a pressure correction is implemented as a postupdate script within this framework.

After all postupdate scripts have been called, the update is added to the potential and the new potential \texttt{<interaction>.pot.new} is written. Additional post-processing of the potential can be performed in the \texttt{post\_add} step which is analogous to the \texttt{post\_update} step except for a potential instead of an update.

To summarize, we list all standard output files for each iterative step:

\begin{tabular}{ll}
\texttt{*.dist.new} & distribution functions of the current step \\
\texttt{*.dpot.new} & the final potential update, created by \texttt{calc\_update} \\
\texttt{*.dpot.<number>} & for each postupdate script, the \texttt{.dpot.new} is saved and a new one\\
&is created\\
\texttt{*.pot.cur} & the current potential used for the actual run\\
\texttt{*.pot.new} & the new potential after the add step \\
\texttt{*.pot.<number>} & same as \texttt{dpot.<number>} but for \texttt{post\_add}
\end{tabular}

If a sub-step fails during the iteration, additional information can be found in the log file. The name of the log file is specified in the steering \xml file.

\subsection{Restarting and continuing}
The interrupted or finished iterative process can be restarted either by extending a finished run or by restarting the interrupted run. When the script \prog{csg_inverse} is called, it automatically checks for a file called \texttt{done} in the current directory. If this file is found, the program assumes that the run is finished. To extend the run, simply increase \cgopt{inverse.iterations_max} in the settings file and remove the file called \texttt{done}. After that, \prog{csg_inverse} can be restarted, which will automatically recognize existing steps and continue after the last one.

If the iteration was interrupted, the script \prog{csg_inverse} might not be able to restart on its own. In this case, the easiest solution is to delete the last step and start again. The script will then repeat the last step and continue. However, this method is not always practical since sampling and analysis might be time-consuming and the run might have only crashed due to some inadequate post processing option. To avoid repeating the entire run, the script \prog{csg_inverse} creates a file with restart points and labels already completed steps such as simulation, analysis, etc. The file name is specified in the option \cgopt{inverse.restart_file}. If specific actions should be redone, one can simply remove the corresponding lines from this file. Note that a file \texttt{done} is also created in each folder for those steps which have been successfully finished.


% #####################################################################################################################
% #####################################################################################################################

\section{Iterative Boltzmann Inversion}
\subsection{Input preparation}
This section describes the usage of \ibi, implemented within the scripting framework described in the previous section \ref{sec:iterative_workflow}. It is suggested to get a basic understanding of this framework before proceeding.

An outline of the workflow for performing \ibi is given in \fig{fig:flow_ibi}.

To specify Iterative Boltzmann Inversion as algorithm in the script, add \texttt{ibi} in the \texttt{method} section of the \xml setting file as shown below.

\begin{lstlisting}
  <cg>
    ...
    <inverse>
      <method>ibi</method>
    </inverse>
  </cg>
\end{lstlisting}


% #####################################################################################################################
% #####################################################################################################################

\section{Inverse Monte Carlo}
\label{sec:iterative_methods_imc}
In this section, additional options are described to run \imc coarse graining. The usage of \imc is similar to the one of \ibi and understanding the use of the scripting framework described in chapter~\ref{sec:iterative_workflow} is necessary.

\textbf{WARNING: multicomponent \imc is still experimental!}

\subsection{General considerations}
In comparison to \ibi, \imc needs significantly more statistics to calculate the potential update\cite{Ruehle:2009.a}. It is advisable to perform smoothing on the potential update. Smoothing can be performed as described in \sect{ref:ibi:optimize}. In addition, \imc can lead to problems related to finite size: for methanol, an undersized system proved to lead to a linear shift in the potential\cite{Ruehle:2009.a}. It is therefore always necessary to check that the system size is sufficiently large and that runlength csg smoothing iterations are well balanced.

\subsection{Correlation groups}
Unlike \ibi, \imc also takes cross-correlations of interactions into account in order to calculate the update. However, it might not always be beneficial to evaluate cross-correlations of all pairs of interactions. By specifying \interopt{inverse.imc.group}, \votca allows to define groups of interactions, amongst which cross-correlations are taken into account, where \interopt{inverse.imc.group} can be any name.

\begin{lstlisting}
  <non-bonded>
    <name>CG-CG</name>
    <type1>CG</type1>
    <type2>CG</type2>
    ...
    <imc>
      <group>solvent</group>
   </imc>
  </non-bonded>
  <non-bonded>
\end{lstlisting}

\subsection{Regularization}

To use the regularized version of IMC a $\lambda$ value $>0$ has to be specified by setting \interopt{inverse.imc.reg}. 
If set to $0$ (default value) the unregularized version of IMC is applied.
\begin{lstlisting}
 <non-bonded>
   <name>CG-CG</name>
   <type1>CG</type1>
   <type2>CG</type2>
    ...
   <inverse>
     <imc>
       <reg>300</reg>
{'fraction_non_alphanumeric': 0.1048974814031643, 'fraction_numerical': 0.0023411244949590303, 'mean_word_length': 4.757391304347826, 'pattern_counts': {'":': 0, '<': 131, '<?xml version=': 0, '>': 133, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 20}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2984379', 'n_tokens_mistral': 6855, 'n_tokens_neox': 6532, 'n_words': 3314}