What does 'forms a right-handed set' mean? In a question I am reading, the following question appears.

What if $\vec{A},\vec{B}$, and $\vec{C}$ are mutually perpendicular and form a right-handed set?

What exactly does "form a right-handed set" mean? That $\hat{A}\times\hat{B}=\hat{C}$? Or does it mean the stronger equality $\vec{A}\times \vec{B}=\vec{C}$?

There are several equivalent definitions, but here is one that is often convenient:
The ordered triplet $(\vec A,\vec B,\vec C)$ of vectors in $\mathbb R^3$ is right called right-handed if $\det(\vec A,\vec B,\vec C)>0$, where $(\vec A,\vec B,\vec C)$ is the square matrix formed by taking the three vectors as columns.
(Note that the determinant is necessarily nonzero since you assumed the vectors to be orthogonal and (hopefully also) nonzero.)
Similarly the triplet is left-handed if the determinant is negative.
The triplet is linearly dependent if the determinant is zero.
This generalizes to any dimension: we can easily define what it means for a $n$-tuplet of vectors in $\mathbb R^n$ to be right-handed.
If you multiply each vector in the multiplet by the same invertible square matrix, handedness is preserved if the matrix has positive determinant and reversed if negative.
Some intuitive properties are easy to observe from this point of view:
If you interchange any two vectors in the multiplet, the handedness is reversed.
If you permute the vectors cyclically, handedness is preserved.
