Q:

Prometheus - same metric with different labels is overridden before scraping

I'm consuming various gauge metrics from Kafka to Prometheus using custom python app. The metrics needs to be consumed in more and less the same time (milliseconds). I'm able to successfully scrap only the last metric from below example as the first three are immediately overwritten.
my_metric{labelA = "aaa", labelB = "111"} 8
my_metric{labelA = "aaa", labelB = "222"} 12
my_metric{labelA = "bbb", labelB = "111"} 7
**my_metric{labelA = "bbb", labelB = "222"} 15**

I can get all four metrics getting scrapped by setting them a unique metrics name e.g.:
my_metric_aaa_111{labelA = "aaa", labelB = "111"} 8

but this does not seem to be the best practice plus working with such metrics is very difficult in general later on in Grafana.
I can also push the metrics serially to be scrapped and lower the interval of scrapping in Prometheus config, but this is against the whole idea of the solution.
Apart from suggestions I'm completely unaware - is it possible to keep for Prometheus the same metric to be scrapped where only label values differ? The only discussion I found on this is here with no answer: https://github.com/prometheus/pushgateway/issues/65.
If the above is not possible, can I somehow merge/join/combine metrics name later on in Prometheus/Grafana to be able to work with them based on their labels? Meaning remove the unnecessary ending _aaa_111 in the example above to work back with everything as one metrics?

A:

You can change the Pushgateway job name, using a combination of the labels, or some unique value. So prometheus could scrape all your metrics, and they aren't overwriten by others.
In your case, if you export: my_metric{labelA = "aaa", labelB = "111"} 8 to a job called, some_job_aaa_111.
You can test pushing some metrics manually: echo "my_metric{labelA = \"aaa\", labelB = \"111\"} 8" | curl --data-binary @- http://localhost:9091/metrics/job/some_job_aaa_111
In Prometheus pushgateway you would see the next:

In Prometheus, the pushgateway job, becomes in a label that you can easily ignore, like the next output of Prometheus:

