What is the mathematical notation for rounding a given number to the nearest integer? What is the mathematical notation for rounding a given number to the nearest integer? So like a mix between the floor and the ceiling function.

It might be too verbose, but something like the following is unlikely to be misinterpreted.
$$\mathrm{RoundToEven}(5.5) = 6$$
If you need another convention such as rounding to the nearest odd number, rounding towards infinity, or rounding towards negative infinity I'd define my own function and include some examples.
For instance:

Let $R \mathop: \mathbb{R} \to \mathbb{Z}$ denote the function that rounds each real number to the nearest integer, rounding ties towards negative infinity.
$$ R(-0.5) = -1 $$
$$ \left\{ R(-0.3)\;,\; R(0)\;,\; R(0.3)\;,\; R(0.5) \right\} = \left\{ 0 \right\} $$
$$ R(1.5) = 1 $$
$$ R(1.7) = 2 $$
