why do we tend to use curly braces for the arguments to the min and max functions? Why is it that we denote the arguments of the min/max functions with curly braces, i.e. $\{$ and $\}$? My first thought would be that it's related to sets and infimum and supremum, but many times it no longer seems to relate to a set, i.e. when used as such: $\min_{1 \leq i \leq N}\{x_i\}$.

I would not use that notation.
I would either treat the argument of $\min$ as a set (with curly braces):
$$\min \{x_i\}_{i=1}^N,$$
or write the argument as a single number, parameterized by indices written below the $\min$:
$$\min_{1\leq i\leq N} x_i.$$
But I'm sure you can find any number of other notations in the literature, and as long as you succeed in clearly communicating the intended meaning, there is no single "right" notation.
