<commit_before>\pageId{browserRequirements}

\section*{MathML Semantic Up-Conversion}

To write about...

\begin{itemize}
\item Failure handling
\item Conventions followed
\end{itemize}

\subsection*{Presentation MathML Enhancement}

The first part of the up-conversion process takes the rather flat MathML normally
output by SnuggleTeX and creates ``enhanced'' presentation MathML that displays in the
same way whilst having a structure that is more amenable to further processing,
such as conversion to Content MathML and other forms.

Generally speaking, the improvements made are as follows:

\begin{itemize}
\item Precedence of infix operators is inferred, as described below, using
\verb|<mrow/>| to house inferred arguments.
\item Implicit multiplications are inferred, using \verb|<mo>&InvisibleTimes;</mo>|
to represent this in the resulting MathML.
\item Applications of known functions like \verb|sin| are inferred, using
\verb|<mo>&ApplyFunction;</mo>| to represent this.
\end{itemize}

\textbf{TODO:} Talk about conventions followed.

\begin{tabular}{|c|l|}
\hline
Test & Result \\
\hline
Infix $,$ & Grouped into a \verb|<mfenced>| with empty opener and closer \\
Infix $\vee$ & Associative Grouping \\
Infix $\wedge$ & Associative Grouping \\
Infix relation operator(s) & Associative Grouping (all at same level) \\
Infix $\cup$ & Associative Grouping \\
Infix $\cap$ & Associative Grouping \\
Infix $\setminus$ & Left-associative Grouping \\
Infix $+$ & Associative Grouping \\
Infix $-$ & Left-associative Grouping \\
Infix $*$, $\times$ and $\cdot$ & Associative Grouping (all at same level) \\
Infix $/$ and $\div$ & Left-associative Grouping \\
Space characters & Treated as explicit multiplication, grouped associatively \\
Any infix operator in unary context & Operator ``appplied'' by wrapping in \verb|<mrow/>| \\
No Infix Operator present & Apply into subgroups (as defined below) and apply implicit product \\
Atom & Kept as-is \\
\hline
\end{tabular}

\subsubsection*{No Infix Operator Handling}

This is split into subgroups starting with elements satisfying
any of the following:

\begin{itemize}
\item The first sibling in a group
\item The first sibling following an \verb|<mfenced/>|
\item The first of one or more prefix operator or function siblings
\item The first non-postfix operator after one or more postfix operator siblings
\end{itemize}

For example, the LaTeX source:

\begin{verbatim}
xy \sin 2x \sin\cos 2x!
\end{verbatim}

would identify 3 subgroups: \verb|xy|, \verb|\sin 2x| and \verb|\sin\cos 2x!|.

The rules for handling these subgroups are listed below. The results of these
are turned into an implicit product using \verb|<mo>&InvisibleTimes;</mo>|.

\subsubsection*{Subgroup Handling}

\begin{itemize}
\item Firstly, any prefix functions (e.g. \verb|\sin|) and operators (e.g. \verb|\lnot|)
are applied recursively from left to right to everything following.

So, \verb|\sin\cos 2ax!| is handled as it if were \verb|\sin(\cos(2ax!))|.

\item Once all prefix functions and operators have been applied, any postfix operators
attached to whatever is remaining are applied from right to left. (Note however
that the factorial operator is handled specially so that it only gets applied
to the last token, so that \verb|2ax!| is treated as \verb|2a(x!)|, which fits in
with common conventions.)

\item Finally, everything between the prefix and postfix parts are treated as an
implicit multiplication.
\end{itemize}

For example, the result of this work on \verb|\sin\cos 2ax!| is the equivalent of
\verb|\sin(\cos(2 \cdot a \cdot x!))|.

\subsection*{Conversion to Content MathML}

The second step in the up-conversion process takes the enhanced Presentation MathML
and attempts to convert it into Content MathML.

\subsubsection*{Configurable Assumptions}

\begin{itemize}
\item Whether $e$ should be treated as the exponential number or not.
\item Whether $i$ should be treated as the imaginary number.
\item Whether $\pi$ should be treated as the number $3.141592\ldots$.
\item Whether $(\ldots)$ should be assumed to delimit a vector or not.
\item Whether $[\ldots]$ should be assumed to delimit a list or not.
\item Whether $\{\ldots\}$ should be assumed to delimit a set or not.
\end{itemize}

\subsubsection*{Supported Operators}

\begin{tabular}{|c|c|c|}
\hline
LaTeX operator & Application & Content MathML element \\
\hline
\verb|\vee| & n-ary & \verb|or| \\
\verb|\wedge| & n-ary & \verb|and| \\
Any mix of relation operators & binary, applied in adjacent pairs & See below \\
\verb|\cup| & n-ary & \verb|union| \\
\verb|\cap| & n-ary & \verb|intersect| \\
\verb|\setminus| & binary & \verb|setdiff| \\
\verb|+| & unary or n-ary & \verb|plus| \\
\verb|-| & unary or binary & \verb|minus| \\
Any multiplication & n-ary & \verb|times| \\
Any division & binary & \verb|times| \\
\verb|!| & unary & \verb|factorial| \\
\hline
\end{tabular}

\begin{itemize}
\item
Operators may be left ``unapplied'', e.g. a raw input of \verb|+|
would result in \verb|<plus/>| with no enclosing
\verb|<apply/>|.

\item
Failures will be registered if an operator is used in an inappropriate
context.
\end{itemize}

\subsubsection*{Supported Relation Operators}

\begin{tabular}{|c|c|}
\hline
LaTeX operator & Content MathML element \\
\hline
\verb|=| & \verb|<eq/>| \\
\verb|\not=| & \verb|<neq/>| \\
\verb|<| & \verb|<lt/>| \\
\verb|\not<| & \verb|<not>...<lt/>...</not>| \\
\verb|>| & \verb|<gt/>| \\
\verb|\not>| & \verb|<not>...<gt/>...</not>| \\
\verb|\leq| & \verb|<leq/>| \\
\verb|\not\leq| & \verb|<not>...<leq/>...</not>| \\
\verb|\geq| & \verb|<geq/>| \\
\verb|\not\geq| & \verb|<not>...<geq/>...</not>| \\
\verb|\equiv| & \verb|<equivalent/>| \\
\verb|\not\equiv| & \verb|<not>...<equivalent/>...</not>| \\
\verb|\approx| & \verb|<approx/>| \\
\verb|\not\approx| & \verb|<not>...<approx/>...</not>| \\
\verb.|. & \verb|<factorof/>| \\
\verb.\not|. & \verb|<not>...<factorof/>...</not>| \\
\verb|\in| & \verb|<in/>| \\
\verb|\not\in| & \verb|<notin/>| \\
\hline
\end{tabular}

\subsubsection*{Supported Functions}

\begin{tabular}{|c|c|c|c|}
\hline
LaTeX function & Arity & Invertible & Content MathML element \\
\hline
\verb|\sin| & unary & Yes & \verb|sin| or \verb|arcsin| \\
\verb|\min| & n-ary & No & \verb|min| \\
$\ldots$ & & & \\
\hline
\end{tabular}
<commit_msg>Fleshed out a bit more. Needs more thought but it's a start!<commit_after>\pageId{browserRequirements}

\section*{MathML Semantic Up-Conversion}

To write about...

\begin{itemize}
\item Failure handling
\item Conventions followed
\end{itemize}

\subsection*{Presentation MathML Enhancement}

The first part of the up-conversion process takes the rather flat MathML normally
output by SnuggleTeX and creates ``enhanced'' presentation MathML that displays in the
same way whilst having a structure that is more amenable to further processing,
such as conversion to Content MathML and other forms.

Generally speaking, the improvements made are as follows:

\begin{itemize}
\item Precedence of infix operators is inferred, as described below, using
\verb|<mrow/>| to house inferred arguments.
\item Implicit multiplications are inferred, using \verb|<mo>&InvisibleTimes;</mo>|
to represent this in the resulting MathML.
\item Applications of known functions like \verb|sin| are inferred, using
\verb|<mo>&ApplyFunction;</mo>| to represent this.
\end{itemize}

\textbf{TODO:} Talk about conventions followed.

\begin{tabular}{|c|l|}
\hline
Test & Result \\
\hline
Infix $,$ & Grouped into a \verb|<mfenced>| with empty opener and closer \\
Infix $\vee$ & Associative Grouping \\
Infix $\wedge$ & Associative Grouping \\
Infix relation operator(s) & Associative Grouping (all at same level) \\
Infix $\cup$ & Associative Grouping \\
Infix $\cap$ & Associative Grouping \\
Infix $\setminus$ & Left-associative Grouping \\
Infix $+$ & Associative Grouping \\
Infix $-$ & Left-associative Grouping \\
Infix $*$, $\times$ and $\cdot$ & Associative Grouping (all at same level) \\
Infix $/$ and $\div$ & Left-associative Grouping \\
Space characters & Treated as explicit multiplication, grouped associatively \\
Any infix operator in unary context & Operator ``appplied'' by wrapping in \verb|<mrow/>| \\
No Infix Operator present & Apply into subgroups (as defined below) and apply implicit product \\
Atom & Kept as-is \\
\hline
\end{tabular}

\subsubsection*{No Infix Operator Handling}

This is split into subgroups starting with elements satisfying
any of the following:

\begin{itemize}
\item The first sibling in a group
\item The first sibling following an \verb|<mfenced/>|
\item The first of one or more prefix operator or function siblings
\item The first non-postfix operator after one or more postfix operator siblings
\end{itemize}

For example, the LaTeX source:

\begin{verbatim}
xy \sin 2x \sin\cos 2x!
\end{verbatim}

would identify 3 subgroups: \verb|xy|, \verb|\sin 2x| and \verb|\sin\cos 2x!|.

The rules for handling these subgroups are listed below. The results of these
are turned into an implicit product using \verb|<mo>&InvisibleTimes;</mo>|.

\subsubsection*{Subgroup Handling}

\begin{itemize}
\item Firstly, any prefix functions (e.g. \verb|\sin|) and operators (e.g. \verb|\lnot|)
are applied recursively from left to right to everything following.

So, \verb|\sin\cos 2ax!| is handled as it if were \verb|\sin(\cos(2ax!))|.

\item Once all prefix functions and operators have been applied, any postfix operators
attached to whatever is remaining are applied from right to left. (Note however
that the factorial operator is handled specially so that it only gets applied
to the last token, so that \verb|2ax!| is treated as \verb|2a(x!)|, which fits in
with common conventions.)

\item Finally, everything between the prefix and postfix parts are treated as an
implicit multiplication.
\end{itemize}

For example, the result of this work on \verb|\sin\cos 2ax!| is the equivalent of
\verb|\sin(\cos(2 \cdot a \cdot x!))|.

\subsection*{Conversion to Content MathML}

The second step in the up-conversion process takes the enhanced Presentation MathML
and attempts to convert it into Content MathML.

\subsubsection*{Configurable Assumptions}

\begin{itemize}
\item Whether \verb|e| should be treated as the exponential number or not.
\item Whether \verb|i| should be treated as the imaginary number.
\item Whether \verb|\pi| should be treated as the number $3.141592\ldots$.
\item Whether \verb|( ... )|verb should be assumed to delimit a vector or not.
\item Whether \verb|[ ... ]| should be assumed to delimit a list or not.
\item Whether \verb|\{ ... \}| should be assumed to delimit a set or not.
\end{itemize}

\subsubsection*{Supported Operators}

\begin{tabular}{|c|c|c|}
\hline
LaTeX operator & Application & Content MathML element \\
\hline
\verb|+| & unary or n-ary & \verb|plus| \\
\verb|-| & unary or binary & \verb|minus| \\
Any multiplication & n-ary & \verb|times| \\
Any division & binary & \verb|times| \\
\verb|\vee| & n-ary & \verb|or| \\
\verb|\wedge| & n-ary & \verb|and| \\
\verb|\cup| & n-ary & \verb|union| \\
\verb|\cap| & n-ary & \verb|intersect| \\
\verb|\setminus| & binary & \verb|setdiff| \\
\verb|\lnot| & unary (prefix) & \verb|not| \\
\verb|!| & unary (postfix) & \verb|factorial| \\
Any mix of relation operators & binary, applied in adjacent pairs & See below \\
\hline
\end{tabular}

\begin{itemize}
\item
Operators may be left ``unapplied'', e.g. a raw input of \verb|+|
would result in \verb|<plus/>| with no enclosing
\verb|<apply/>|.

\item
Failures will be registered if an operator is used in an inappropriate
context.
\end{itemize}

\subsubsection*{Supported Relation Operators}

\begin{tabular}{|c|c|}
\hline
LaTeX operator & Content MathML element \\
\hline
\verb|=| & \verb|<eq/>| \\
\verb|\not=| & \verb|<neq/>| \\
\verb|<| & \verb|<lt/>| \\
\verb|\not<| & \verb|<not>...<lt/>...</not>| \\
\verb|>| & \verb|<gt/>| \\
\verb|\not>| & \verb|<not>...<gt/>...</not>| \\
\verb|\leq| & \verb|<leq/>| \\
\verb|\not\leq| & \verb|<not>...<leq/>...</not>| \\
\verb|\geq| & \verb|<geq/>| \\
\verb|\not\geq| & \verb|<not>...<geq/>...</not>| \\
\verb|\equiv| & \verb|<equivalent/>| \\
\verb|\not\equiv| & \verb|<not>...<equivalent/>...</not>| \\
\verb|\approx| & \verb|<approx/>| \\
\verb|\not\approx| & \verb|<not>...<approx/>...</not>| \\
\verb.|. & \verb|<factorof/>| \\
\verb.\not|. & \verb|<not>...<factorof/>...</not>| \\
\verb|\in| & \verb|<in/>| \\
\verb|\not\in| & \verb|<notin/>| \\
\hline
\end{tabular}

\begin{itemize}
\item
  Relation operators will be paired up earlier in the up-conversion
  process. Where there are two or more relations together, each pairing
  becomes an operand of an enclosing logical \verb|and| when
  converting to Content MathML. So a LaTeX input like \verb|1<x \leq 2| will
  result in the same output as \verb|(1<x) \land (x \leq 2)|.

\item
  Note that this pairing is still done for inputs like \verb|a=b=c|,
  even though they could also have been converted to an $n$-ary application of
  the \verb|<eq/>| operator.
\end{itemize}

\subsubsection*{Supported Functions}

\begin{tabular}{|c|c|c|c|}
\hline
LaTeX function & Arity & Invertible & Content MathML element \\
\hline
\verb|\sin| & unary & Yes & \verb|sin| or \verb|arcsin| \\
\verb|\cos| & unary & Yes & \verb|cos| or \verb|arccos| \\
\verb|\tan| & unary & Yes & \verb|tan| or \verb|arctan| \\
\verb|\sec| & unary & Yes & \verb|csc| or \verb|arcsec| \\
\verb|\cot| & unary & Yes & \verb|cot| or \verb|arccot| \\
\verb|\sin| & unary & Yes & \verb|sinh| or \verb|arcsinh| \\
\verb|\cos| & unary & Yes & \verb|cosh| or \verb|arccosh| \\
\verb|\tan| & unary & Yes & \verb|tanh| or \verb|arctanh| \\
\verb|\sec| & unary & Yes & \verb|sech| or \verb|arcsech| \\
\verb|\csc| & unary & Yes & \verb|csch| or \verb|arccsch| \\
\verb|\cot| & unary & Yes & \verb|coth| or \verb|arccoth| \\
\verb|\arcsin| & unary & No & \verb|arcsin| \\
\verb|\arccos| & unary & No & \verb|arccos| \\
\verb|\arctan| & unary & No & \verb|arctan| \\
\verb|\arcsec| & unary & No & \verb|arcsec| \\
\verb|\arccsc| & unary & No & \verb|arccsc| \\
\verb|\arccot| & unary & No & \verb|arccot| \\
\verb|\arcsinh| & unary & No & \verb|arcsinh| \\
\verb|\arccosh| & unary & No & \verb|arccosh| \\
\verb|\arctanh| & unary & No & \verb|arctanh| \\
\verb|\arcsech| & unary & No & \verb|arcsech| \\
\verb|\arccsch| & unary & No & \verb|arccsch| \\
\verb|\arccoth| & unary & No & \verb|arccoth| \\
\verb|\ln| & unary & No & \verb|ln| \\
\verb|\log| & unary & No & \verb|log| \\
\verb|\exp| & unary & No & \verb|exp| \\
\verb|\det| & unary & No & \verb|determinant| \\
\verb|\gcd| & n-ary & No & \verb|gcd| \\
\verb|\lcm| & n-ary & No & \verb|lcm| \\
\verb|\max| & n-ary & No & \verb|max| \\
\verb|\min| & n-ary & No & \verb|min| \\
\verb|\Re| & n-ary & No & \verb|imaginary| \\
\verb|\Im| & n-ary & No & \verb|real| \\
$\ldots$ & & & \\
\hline
\end{tabular}

\begin{itemize}
\item
  For all functions, constructs like \verb|\cos^n x| is
  interpreted as ``cosine $x$ raised to the power of $n$''
  where $n$ is either an identifier or a number greater than
  or equal to $1$.

\item
  For functions listed as Invertible in the table above,
  the up-conversion process interprets constructs like
  \verb|\sin^{-1} x| as the ``inverse sin of $x$''
  and would result in \verb|<apply><arcsin/><ci>x</ci></apply>|.
  A failure will be noted if constructs like these are
  used on functions which do not support this.

\item
  For the $\log$ function, an input like \verb|\log_a x|
  is interpreted as ``logarithm to base $a$ of $x$''.
\end{itemize}

\subsubsection*{Supported Symbols}

\begin{tabular}{|c|c|}
\hline
LaTeX symbol & Content MathML interpretation \\
\hline
\verb|\emptyset| & \verb|<emptyset/>| \\
\verb|\infty| & \verb|<infinity/>| \\
\hline
\end{tabular}

\subsubsection*{Other Supported Constructs}

\begin{itemize}

\item
  Powers are supported in the expected way. The special case of \verb|e^x|
  is optionally treated as \verb|\exp x|.

\item
  Square and $n$th roots are supported using the familiar LaTeX
  constructs \verb|\sqrt{x}| and \verb|\sqrt[n]{x}|.

\item
  Subscripted identifiers like \verb|x_1|, \verb|x_{1,2}|
  and \verb|a_{x_y}| are supported and are kept as presentation MathML wrapped
  inside a \verb|<ci/>| container element.

\item
  SnuggleTeX includes a custom macro called \verb|\units| that can be used
  to denote that its argument represent units. The result of up-converting
  this is a \verb|<csymbol/>| element.

\end{itemize}

\subsection*{Conversion to Maxima Input}

\begin{itemize}
\item
  The Maxima input form is created from the Content MathML form. As a result,
  if the conversion to Content MathML fails then so will the conversion to
  Maxima.

\item
  Not all inputs that can be successfully converted to Content MathML can
  be further converted into Maxima input form.
\end{itemize}

\subsubsection*{Supported Identifiers}

MathML generally allows arbitrary Unicode characters to be used as identifier
names whereas Maxima only uses ASCII. In terms of LaTeX input, this means that
idenfiers input as \verb|\something| are only supported if we have identifier a
means to convert these into a Maxima input form.

The identifiers supported so far are:

\begin{tabular}{|c|c|}
\hline
LaTeX input & Maxima form \\
\hline
\verb|\alpha| & \%alpha \\
\verb|\beta| & \%beta \\
\verb|\gamma| & \%gamma \\
\verb|\delta| & \%delta \\
\verb|\epsilon| & \%epsilon \\
\verb|\zeta| & \%zeta \\
\verb|\eta| & \%eta \\
\verb|\theta| & \%theta \\
\verb|\iota| & \%iota \\
\verb|\kappa| & \%kappa \\
\verb|\lambda| & \%lambda \\
\verb|\mu| & \%mu \\
\verb|\nu| & \%nu \\
\verb|\xi| & \%xi \\
\verb|\pi| & \%pi \\
\verb|\rho| & \%rho \\
\verb|\sigma| & \%sigma \\
\verb|\tau| & \%tau \\
\verb|\upsilon| & \%upsilon \\
\verb|\phi| & \%phi \\
\verb|\chi| & \%chi \\
\verb|\psi| & \%psi \\
\verb|\omega| & \%omega \\
\verb|\Gamma| & \%Gamma \\
\verb|\Delta| & \%Delta \\
\verb|\Theta| & \%Theta \\
\verb|\Lambda| & \%Lambda \\
\verb|\Xi| & \%Xi \\
\verb|\Pi| & \%Pi \\
\verb|\Sigma| & \%Sigma \\
\verb|\Upsilon| & \%Upsilon \\
\verb|\Phi| & \%Phi \\
\verb|\Psi| & \%Psi \\
\verb|\Omega| & \%Omega \\
\hline
\hline
\end{tabular}

The LaTeX inputs \verb|e| and \verb|i| are converted to \verb|%e|
and \verb|%i| respectively if the Content MathML up-conversion
was configured to interpret them in this way.

\subsubsection*{Supported Functions}

All functions supported by the Content MathML up-conversion are supported
here, with the exception of \verb|\log| as Maxima only supports natural
logarithms.

\subsubsection*{Supported Operators}

All of the operators supported by the Content MathML process are
supported, with the exception of the following relations:

\begin{itemize}
  \item \verb|\equiv|
  \item \verb|\not\equiv|
  \item \verb|\approx|
  \item \verb|\not\approx|
  \item \verb.|.
  \item \verb.\not|.
  \item \verb|\in|
  \item \verb|\not\in|
\end{itemize}

Maxima does not allow ``unapplied'' operators symbols, so a LaTeX input
of the form \verb|+| generates a result of the form \verb|operator("+")|
as a placeholder for this. Similarly, \verb|\not=| results in
\verb|operator("not=")| for want of anything better.
The name of the resulting function is configurable.

\subsubsection*{Other Supported Constructs}

\begin{itemize}
\item
  Units entered using the special SnuggleTeX \verb|\units| macro, such as
  \verb|\units{kg}| generate a Maxima form like \verb|units("kg")|, which
  is similar to how we handle unapplied operators. (Again, the name of
  the resulting function is configurable.)

\item
  Subscripted identifiers are converted to a suitable Maxima form if
  deemed possible.
\end{itemize}
{'fraction_non_alphanumeric': 0.16126739108357876, 'fraction_numerical': 0.0022849598862597744, 'mean_word_length': 3.540110650069156, 'pattern_counts': {'":': 0, '<': 107, '<?xml version=': 0, '>': 105, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4153100', 'n_tokens_mistral': 6662, 'n_tokens_neox': 6460, 'n_words': 2268}