Completely baffled by this question involving putting matrices in matrices This is homework, so only hints please.

Let $A\in M_{m\times m}(\mathbb{R})$
   , $B\in M_{n\times n}(\mathbb{R})$
   . Suppose there exist orthogonal matrices $P$
    and $Q$
    such that $P^{T}AP$
    and $Q^{T}BQ$
    are upper triangular. Let $C$
    be any $m\times n$ matrix. Then show that there is an orthogonal $R$
   , not depending on $C$
   , such that $$R^{T} \begin{bmatrix}A & C\\
0 & B
\end{bmatrix}R$$
   is upper triangular.

I haven't done any proofs before using matrices containing matrices in them. I have absolutely no idea where to begin. Any ideas?

The elements of a matrix must be add-able and multiply-able (and the addition and multiplication have to satisfy some field properties).
Matrices themselves are add-able and multiply-able (except that AB isn't necessarily BA), so they can be elements of another matrix.
$\begin {bmatrix} A & B \\ C & D \end{bmatrix} \begin {bmatrix} E & F \\ G & H \end{bmatrix}  = \begin {bmatrix} A E + B G & A F + B H \\ C E + D F & C G + D H \end{bmatrix}$ 
In your problem, we want to find
$\begin {bmatrix} X & Y \\ 0 & Z \end{bmatrix} = \begin {bmatrix} R_A & R_B \\ R_C & R_D\end{bmatrix}^T \begin {bmatrix} A & C \\ 0 & B \end{bmatrix} \begin {bmatrix} R_A &  R_B \\ R_C & R_D \end{bmatrix}$
where $X$ and $Z$ are upper triangular.
$ = \begin {bmatrix} R_A^T & R_C^T \\ R_B^T & R_D^T\end{bmatrix} \begin {bmatrix} A & C \\ 0 & B \end{bmatrix} \begin {bmatrix} R_A &  R_B \\ R_C & R_D \end{bmatrix}$
$ = \begin {bmatrix} R_A^T A & R_A^TC + R_C^T B \\ R_B^TA & R_B^TC + R_D^T B\end{bmatrix} \begin {bmatrix} R_A &  R_B \\ R_C & R_D \end{bmatrix}$
$ = \begin {bmatrix} R_A^T A R_A + R_A^TCR_C + R_C^T B R_C & R_A^T A R_B + R_A^TCR_D + R_C^T B R_D \\ R_B^T A R_A + R_B^TCR_C + R_D^T B R_C & R_B^T A R_B + R_B^TCR_D + R_D^T B R_D\end{bmatrix}$
So we want to chose R so that:
$\begin{align}
\text{Upper Triangular} &= R_A^T A R_A + R_A^TCR_C + R_C^T B R_C \\
\text{Anything} &= R_A^T A R_B + R_A^TCR_D + R_C^T B R_D \\
\text{Zero} &= R_B^T A R_A + R_B^TCR_C + R_D^T B R_C \\
\text{Upper Triangular} &= R_B^T A R_B + R_B^TCR_D + R_D^T B R_D \\
\\
\text{Given that:} \\
\text{Upper Triangular} &= P^TAP \\
\text{Upper Triangular} &= Q^TBQ \\
\end{align}$
You can see by looking at the equations now that you want some terms to be upper triangular and some to be zero.  Fit in $R_A = P$, $R_B = 0$, $R_C = 0$, and $R_D = Q$, you get:
$\begin{align}
\text{Upper Triangular} &= P^T A P + P^TC0 + 0^T B 0 \\
                        &= P^T A P \\
\text{Anything} &= P^T A 0 + P^TCQ + 0^T B Q \\
                &= P^TCQ \\
\text{Zero} &= 0^T A P + 0^TC0 + Q^T B 0 \\
            &= 0 \\
\text{Upper Triangular} &= 0^T A 0 + 0^TCQ + Q^T B Q \\
                        &= Q^T B Q \\
\end{align}$
$R = \begin {bmatrix} P & 0 \\ 0 & Q \end{bmatrix}$
