What is the math notation for this type of function? A function that turns a real number into another real number can be represented like $f : \mathbb{R}\to \mathbb{R}$
What is the analogous way to represent a function that turns an unordered pair of elements of positive integers each in $\{1,...,n\}$ into a real number?  I guess it would almost be something like $$f : \{1,...,n\} \times \{1,...,n\} \to \mathbb{R}$$ but is there a better notation that is more concise and that has the unorderedness?

I would say that it might be best to preface your notation with a sentence explaining it, which will allow the notation itself to be more compact, and generally increase the understanding of the reader. For example, we could write:

Let $X=\{x\in\mathbb{N}\mid x\leq N\}$, and let $\sim$ be an equivalence relation on $X^2$ defined by $(a,b)\sim(c,d)$ iff either $a=c$ and $b=d$, or $a=d$ and $b=c$. Let $Y=X^2/\sim$, and let $f:Y\to\mathbb{R}$.

So, $Y$ can be thought of as the set of unordered pairs of positive integers up to $N$, and you can then proceed to use this notation every time you want to talk about such a function.
