What does the star symbol means in the binary operation a*b .... Ex: x*y=x+y+xy what does the star symbol means in binary operation algebra (logic) 
Does it refer to a certain rule
It means the operation defined on the RHS.

For example
$$2*3= 2+ 3+2\times 3=11$$
In binary logic $+$ refers to OR and $\cdot$ refers to AND.

You could also use alternate notation $x\lor y\lor(x\land y)$ but it is equivalent to simple $x\lor y$ or $x+y$ in the origninal notation.

Assuming it is really a logical operation, I do not see the purpose of defining $*$ if it is the same thing as $+$ or maybe this is th epurpose of the exercise to show they are the same rules.

Anyway, I sometimes saw $*$ being used for NAND i.e. $\lnot(x\land y)$ or $(xy)'$ in both notations.

But this is not the case here, are you certain about the definition of $x*y$ ?

https://en.wikipedia.org/wiki/Logic_gate