To what extent should relevant background be discussed vs. just referenced? Is is perfectly appropriate to refer reviewers to other papers for understanding a concept or writer must include the concept in his paper too?
What should be done while keeping brevity, plagiarism and complexity of idea in mind?
It depends.


What do your readers need to know to appreciate the main result of your paper? If your paper builds on the ideas of others, these ideas must be stated (included explicitly) and accompanied by a citation. If you simply note in passing that there are other relevant ideas, reference may be enough. Having said that, as a reviewer, I will question and discourage an excessively long list of references [1-22]. Authors should do their work to identify and tell the reader what they can find in each paper separately, not dump the whole reading list on them.
How long is your paper? If you are writing a 100+ page manuscript, then brevity is not as important as if you were restricted to 1 or 2 pages for your extended abstract. 
Whom are you writing for? If this is a highly specialised journal, you can expect your readers to already know some sacred texts by heart, and a reference is sufficient. For a more general readership, you need to support your reader and tell them more , i.e. explain the main ideas and then refer to the texts.