We want to start working in a new research direction. How to proceed as the first student in the group? I'm a graduate (Master's) student in Engineering/Applied Physics (EE, specifically). A few months ago when I wanted to choose my specific research topic, my advisor got especially interested in one of my suggestions and, even though no one in our department had worked on that topic before, she insisted that I pick this as my dissertation topic (I think because of the novelty and hotness of the topic, besides my own interest). She told me that, starting the next year, she will ask the new students to work on this same topic. I accepted and started my studies.

Now since it's my first experience doing research as a member of a research group, I'm a little confused about how we (the group) should proceed about starting a whole new research direction. There are some new basic materials (like unfamiliar/unconventional math and physics for EE students) that are actually needed to be learned, besides the leading-edge research papers.

From my own perspective, this is my dream topic to work on as I'm going to get back into serious Physics after some unpleasant engineering years, and I think in long-term this is going to be great for me as a researcher, as I'm getting familiar with a field while still being in its dawn. For PhD, I'm planning to move to one of the leading research groups in the field. What are some advice for me, as the first student working on this new topic in the group with no one having any prior experience in it, to make the best outcome in short-term (publications, PhD admissions) and therefore being able to prove my competence in research?

(Also as a side-question, from the group's perspective: How do research groups, especially small ones, embark on a new research direction? Does it start with an enthusiastic student convincing the advisor to start working on the topic, or with an interested professor finding a enthusiastic student? What are the student's and the advisor's responsibilities in each case, as both have near zero experience in the field?)
For your question from the research groups perspective there is no definitive answer. Many things are possible and it depends on the group, the students, the field and whatnot. However, if you plan to enter such endeavor, you should discuss plans and responsibilities.

For a personal perspective: It is a lot of work to go into a new field. Read basics, but stay up to date. Talk a lot to people who work in the field to get a better sense, what the pressing questions, the important goals are and learn the jargon. Listen to talks on the topic, prepare well ad ask questions to get in discussion.

The bad news is, that nobody can guarantee any short-term outcome (unless you can somehow figure out some low-hanging fruits in the new field)…