Q:

How to say in correct tense

I want to say in English about some fact in the past, but it has an impact the present:

I have starting going to the gym last week? (I still go there)  

or

I have started going to the gym last week? (I still go there)

What about that I want to say about the same action, but that action began much earlier:

I have starting going to the gym two years ago? (I still go there)  

or

I have started going to the gym two years ago? (I still go there)

A:

The auxiliary have is always followed by a past participle, never any other form, so I have starting is never grammatical. 

I have started going to the gym.

is grammatical, and completely natural, if you started recently, and are still continuing.
For me, I would not use the present perfect with a specified time, so I would not say I have started going to the gym last week, but rather

I started going to gym last week. 

I would not count the form with have started as ungrammatical, but as less idiomatic. 
If you are talking about two years ago, I have started going to the gym is disctinctly odd. If I've just met somebody I haven't seen for ten years, I could say I've started going to the gym for any time in those ten years, as long as I'm still going there; but in any other context it wouldn't fit. 

