what is the usual topology on a vector space? I do not understand the topology of a Lie group clearly. Let $G$ be a Lie group and $T_eG$ be its tangent space at the identity $e \in G$. Why $Aut(T_eG)$ is an open subset of the vector space of endomorphisms of $T_eG$ (i.e. $End(T_eG)$)? What does "open" mean?

First, this has nothing to do with Lie groups or Lie algebras.  The only important part is that $T_eG$ is a vector space.
For any vector space $V$ (say, finite dimensional over the reals), the set $End(V)$ is naturally a finite dimensional vector space.  Hence, $End(V)$ is isomorphic to $\mathbb{R}^N$ for some $N$ (in fact, $N = $(dim$V)^2$).  Use any choice of isomorphism to topologize $End(V)$.  This choice of isomorphism is equivalent to choosing a basis of $V$.
Now, one has the determinant $det:End(V)\rightarrow\mathbb{R}$ which is given as a polynomials in the entries of the matrices in $End(V)$ (they are matrices after choosing a basis), and hence is continuous.
Since $det$ is continuous, $det^{-1}(\mathbb{R}-\{0\})$ is an open subset of $End(V)$.  But this subset is precisely $Aut(V)$, the invertible transformations from $V$ to $V$.
