Is it OK to add citations from low impact factor journals? For example, let us consider that I write a manuscript which is to be submitted to one of the prestigious journals such as PRL, PRB, etc. In this manuscript, is it OK to cite from some journals which are generally considered as not-so-prestigious journals? If I cite more papers from low impact factor journals, does it play a negative role?
Of course, you should cite any relevant work.

You can't ignore work just because it is in a journal that someone believes to be lesser. That doesn't mean the work is bad. 

As long you do a thorough literature review and avoid predatory journals, there should be no negative impact on your paper.