If I want to avoid quantifiers? In mathematics quantifiers always is used with restrictions $\forall x\in A$ etc and mathematicians often write:
$$x\in A\implies p(x)\;\;\;\text{instead of}\;\;\;\forall x\in A:p(x).$$ 
Is there a counterpart expression for $\exists x\in A: p(x)$ without quantifier?

Is writing like this logical unobjectionable? 

I'm interested in trying to replace 'predicate logic' in mathematics with a 'mathematical propositional algebra', that is, only use propositions of the type $x\in A\implies p(x)$. For such propositions $P_i$, there would be a "mathematical ring" with elements
$$\displaystyle
\bigoplus_i\bigwedge_j p_j^{n_i}.
$$
I guess it won't work but in the process I might learn something about logic and semantics.
The fact that the universal quantifier on $x$ is omitted doesn't mean it's not supposed to be there. The correct parsing of $\forall x\in A: p(x)$ is $\forall x(x\in A\rightarrow p(x))$.

For existential quantifier this would be $\exists x(x\in A\land p(x))$. You may feel it is okay to omit the outer quantifier, but you have to remember it's still there.
Considering the other answers, I conclude that it really is possible to exclude quantifiers with limitations of the form $x\in X$ ($\forall x\in X$ and $\exists x\in X$) without loss of information, when properly distinguishes between bounded and free variables. Obviously 

$\forall x\in X:P(x)\quad\Leftrightarrow\quad a\in X\Rightarrow P(a)$

and here there are no problems with the interpretation of the alternatives. Less obvious  is

$\exists x\in X:P(x)\quad\Leftrightarrow\quad a\in X\wedge P(a)$

but successive reversible transformations shows:

$\exists x\in X:P(x)\;\Leftrightarrow\;\neg\forall x\in X:\neg P(x)\;\Leftrightarrow\;
\neg(a\in X\Rightarrow\neg P(a))\;\Leftrightarrow\;\neg(a\notin X\vee\neg P(a))\;\Leftrightarrow\;$$(a\in X\wedge P(a))$.

Not all the steps are equally easy to interpret, but never the less they are logical equivalents and it is more a psychological issue that it may be difficult to interpret a specific step in the chain. But the logical machinery works consistent. If one is interested of the interpretation of the statement one have to transform the statement to a more readable form ($\mathbb N_m=\{x\in\mathbb N|x&gt;m\}$ and $\oplus$ is exclusive or below):

$1\oplus(\varepsilon\in\mathbb R_+)\oplus(\varepsilon\in\mathbb R_+)(m\in\mathbb N)
\oplus(\varepsilon\in\mathbb R_+)(m\in\mathbb N)(n\in\mathbb N_m)\oplus$
$(n\in\mathbb N_m)(|x-x_n|&lt;\varepsilon)\Leftrightarrow$
$(\varepsilon\notin\mathbb R_+)\oplus(\varepsilon\in\mathbb R_+)(m\in\mathbb N)
(1+(n\in\mathbb N_m))\oplus(n\in\mathbb N_m)(|x-x_n|&lt;\varepsilon)\Leftrightarrow$
$(\varepsilon\notin\mathbb R_+)\oplus(\varepsilon\in\mathbb R_+)(m\in\mathbb N)
(n\notin\mathbb N_m)\oplus(n\in\mathbb N_m)(|x-x_n|&lt;\varepsilon)\Leftrightarrow$
$(\varepsilon\notin\mathbb R_+)\vee (m\in\mathbb N\wedge n\notin\mathbb N_m)\vee
(m\in\mathbb N\wedge|x-x_n|&lt;\varepsilon)\Leftrightarrow$
$\varepsilon\in\mathbb R_+\Rightarrow m\in\mathbb N\wedge(n\in\mathbb N_m\Rightarrow |x-x_n|&lt;\varepsilon)\Leftrightarrow$
$\forall\epsilon\in\mathbb R_+\exists k\in\mathbb N:n\in\mathbb N_{k}\Rightarrow |x-x_n|&lt;\epsilon\Leftrightarrow \displaystyle \lim_{i\rightarrow\infty}x_i=x$.

Thus, the logic in (non meta) mathematics is substantially propositional logic.

In general predicate logic it is not always possible to eliminate quantifiers.