Q:

How do I typeset a table with a caption within a tabularx within another table with a caption?

Unfortunately, it keeps bugging me with forgotten \endgroup, float(s) lost, not in outer par mode and so on.
In other words, why can't I do something like this (the actual table is significantly more complex with rules and stuff):
\documentclass{article}

\usepackage{blindtext}
\usepackage{tabularx}

\begin{document}

\begin{table}
  \begin{tabularx}{\textwidth}{X}
    \begin{table}
      \begin{tabularx}{\textwidth}{X}
        \blindtext
      \end{tabularx}
      \caption{Inner table}
    \end{table}
  \end{tabularx}
  \caption{Outer table}
\end{table}

\end{document}

A:

There are two problems here:

Nested table environments: Not in outer par mode
I suggest dropping the inner table environment. (How should the concept of floating be applicable inside a tabular where the contents are fixed, anyway?)
Nested tabularx environments: 

Extra }, or forgotten \endgroup, 
Missing \endgroup inserted and
Missing } inserted

The solution is to enclose the inner tabularx by a pair of braces.
The the tabularx manual states:

tabular and tabular* environments may be nested with no restriction,
  however if one tabularx environment occurs inside another, then the inner
  one must be enclosed by { }.

Now the inner caption does work without problems and results in a enumerated caption:

Table 1: Inner table

If this is not preferred, one can use the starred version of \caption* for the inner caption from the caption package.

I also replaced \textwidth with \linewidth so that inner tabularx uses the correct remaining horizontal space (→ Difference between \textwidth, \linewidth and \hsize)
Further improvements  could be 

enclosing the inner tabularx and its caption inside the center environment or 
providing more vertical space before the inner tabularx, i.e. \\[2ex].

Code
\documentclass{article}

\usepackage{blindtext}
\usepackage{tabularx}

\begin{document}

\begin{table}
  \begin{tabularx}{\linewidth}{X}
     \blindtext \\
      {\begin{tabularx}{\linewidth}{X}
        \blindtext
      \end{tabularx}}
      \caption{Inner table}
     \\
     \blindtext
  \end{tabularx}
  \caption{Outer table}
\end{table}

\end{document}

Output

