Citing books in scientific articles I'm writing an article that has something to do with the multipole expansion with irreducible tensor operators. The topic is old, and I only find books talking about it.

So is it OK to cite tertiary sources (books) in scientific journals? or must it be a primary source source article? Or does it depend on the journal?

The question I'm asking is general about citation rules, and not for this specific case, though this specific case with irreducible tensor operators is what I'm facing right now.
Yes, citing books is fine (including anything from research monographs to elementary textbooks).  The question you should ask yourself is what the citation is for.  What are you trying to convey to the reader?  For example, if you are trying to assign intellectual credit, then you should cite whoever originally made the discovery.  However, that's far from the only reason to give a citation.  If you are providing references to background or more detailed explanations, then all you need is a source that explains what you need, and sometimes a textbook will be clearer or more useful than a research paper.  If you are supplying a citation to back up a potentially controversial claim, then you need an authoritative source.

The specific topic you mention is mathematical, in which case the distinction between primary, secondary, and tertiary sources is not particularly important except for assigning credit.  As far as explanations go, any derivation will do.  Of course, the form of your citation should indicate your intention.  For example, if you cite a textbook, you can say something like "See [insert reference] for an exposition of the theory of irreducible tensor operators" to make sure it doesn't look like you are claiming this book is where the theory originated.

Of course other fields may attach far more weight to these distinctions.