Upper bounds on the size of $\operatorname{Aut}(G)$ Any automorphism of a group $G$ is a bijection that fixes the identity, so an easy upper bound for the size of $\operatorname{Aut}(G)$ for a finite group $G$ is given by 
\begin{align*}\lvert\operatorname{Aut}(G)\rvert \leq (|G| - 1)! \end{align*}
This inequality is an equality for cyclic groups of orders $1$, $2$ and $3$ and also the Klein four-group $\mathbb{Z}_2 \times \mathbb{Z_2}$. I think it's reasonable to believe that they are the only groups with this property. The factorial $(|G| - 1)!$ is eventually huge. I searched through groups of order less than $100$ with GAP and found no other examples.
The problem can be reduced to the abelian case. We can check the groups of order $< 6$ by hand. Then if $|G| \geq 6$ and the equality holds, we have $\operatorname{Aut}(G) \cong S_{|G|-1}$. Now $\operatorname{Inn}(G)$ is a normal subgroup of $\operatorname{Aut(G)}$, and is thus isomorphic to $\{(1)\}$, $A_{|G|-1}$ or $S_{|G|-1}$. This is because $A_n$ is the only proper nontrivial normal subgroup of $S_n$ when $n \geq 5$. We can see that $(|G| - 1)!/2 > |G|$ and thus $\operatorname{Inn}(G) \cong G/Z(G)$ is trivial.
How to prove that there are no other groups for which the equality $\lvert\operatorname{Aut}(G)\rvert = (|G| - 1)!$ holds? Are any better upper bounds known for larger groups?

I believe this is an exercise in Wielandt's permutation groups book.
$\newcommand{\Aut}{\operatorname{Aut}}\newcommand{\Sym}{\operatorname{Sym}}\Aut(G) \leq \Sym(G\setminus\{1\})$ and so if $|\Aut(G)|=(|G|-1)!$, then $\Aut(G) = \Sym(G\setminus\{1\})$ acts $|G|-1$-transitively on the non-identity elements of G.  This means the elements of G are indistinguishable.  Heck even subsets of the same size (not containing the identity) are indistinguishable. I finish it below:

 In particular, every non-identity element of G has the same order, p, and G has no proper, non-identity characteristic subgroups, like $Z(G)$, so G is an elementary abelian p-group.  However, the automorphism group is $\newcommand{\GL}{\operatorname{GL}}\GL(n,p)$ which, for $p \geq 3, n\geq 2$, only acts at most $n-1$-transitively since it cannot send a basis to a non-basis.  The solutions of $p^n-1 \leq n-1, p \geq 3, n \geq 2$ are quite few: none.  Obviously $\GL(1,p)$ has order $p-1$ which is very rarely equal to $(p-1)!$, when $p=2, 3$.  $\GL(n,2)$ still can only act $n$-transitively if $2^n-1 > n+1$, since once a basis's image is specified, the other points are determined, and the solutions of $2^n-1 \leq n+1$ are also limited: $n=1,2$.  Thus the cyclic groups of order 1,2,3 and the Klein four group are the only examples.

