How many triangles can be formed by the vertices of a regular polygon of $n$ sides? 
How many triangles can be formed by the vertices of a regular polygon of $n$ sides? And how many if no side of the polygon is to be a side of any triangle ?

I have no idea where I should start to think. Can anyone give me some insight ?

Use Combination or Permutation

Consider a regular polygon with $n$ number of vertices $\mathrm{A_1, \ A_2,\ A_3, \ A_3, \ldots , A_{n-1}}$ & $\mathrm{A_{n}}$
Total number of triangles formed by joining the vertices of n-sided regular polygon $$N=\text{number of ways of selecting 3 vertices out of n}=\color{}{\binom{n}{3}}$$ $$N=\color{red}{\frac{n(n-1)(n-2)}{6}}$$
$\forall \ \ \color{blue}{n\geq 3}$
Consider a side $\mathrm{A_1A_2}$ of regular n-polygon. To get a triangle with only one side $A_1A_2$ common (As shown in figure-1 below)
 (figure-1)
Join the vertices $A_1$ & $A_2$ to any of $(n-4)$ vertices i.e. $A_4, \ A_5,\ A_6, \  \ldots \ A_{n-1}$ to get triangles with only one side common. Thus there are $(n-4)$ different triangles with only one side $A_1A_2$ common. Similarly, there are $(n-4)$ different triangles with only one side $A_2A_3$ common & so on. Thus there are $(n-4)$ different triangles with each of $n$ sides common.
Therefore, number of triangles $N_1$ having only one side common with that of the polygon $$N_1=\text{(No. of triangles corresponding to one side)}\text{(No. of sides)}=\color{blue}{(n-4)n}$$
 (figure-2)
Now, join the alternate vertices $A_1$ & $A_3$ by a straight (blue) line to get a triangle $A_1A_2A_3$ with two sides $A_1A_2$ & $A_2A_3$ common. Similarly, join alternate vertices $A_2$ & $A_4$ to get another triangle $A_2A_3A_4$ with two sides $A_2A_3$ & $A_3A_4$ common & so on (as shown in above figure-2). Thus there are $n$ pairs of alternate & consecutive vertices to get $n$ different triangles with two sides common (Above fig-2 shows $n$ st. lines of different colors to join alternate & consecutive vertices). Therefore, number of triangles $N_2$ having two sides common with that of the polygon $$N_2=\color{blue}{n}$$
If $N_0$ is the number of triangles having no side common with that of the polygon then we have $$N=N_0+N_1+N_2$$ $$N_0=N-N_1-N_2$$ $$=\binom{n}{3}-(n-4)n-n$$ $$=\color{}{\frac{n(n-1)(n-2)}{6}-n^2+3n}$$
$$N_0=\color{red}{\frac{n(n-4)(n-5)}{6}}$$
The above formula $(N_0)$ is valid for polygon having $n$ no. of the sides such that $ \ \ \color{blue}{n\geq 6}$
