What to do if most of my students have switched to a better lecturer's class? I teach a certain course which is part of the obligatory undergraduate curriculum. I teach two groups out of four; the other two groups are taught by two other lecturers. In one group I have about 20-30 students. The other group also started with the same number, but students left the group gradually, and now there remain only 5 or even less; the others go to the other lecturer who gives the same lesson in parallel. One reason is probably that I am a new teacher and still not very good at teaching. Another reason is that the other lecturer is very good. 

I feel that some of the 5 students remaining in my class do so only because they are being polite. I appreciate it very much since I put a lot of effort in preparing the lessons, but it is not very efficient for me or for them. So, I thought of suggesting to the remaining students that they move to the other lecturer's group. This way, they will have a better lecturer and I will have more time for research  (the other lecturer said it's OK with him). What might be the implications of such a step? 

UPDATE: Thanks a lot for all the feedback. While there are many great answers, the one thing that convinced me to keep going was the fact that this is the only way I can become a better teacher. So today I came to class as usual. For some reason, the number of students rised to 10.
You should discuss this with your department chair (the person who runs your department). While the teaching chair (or graduate/undergraduate chair) might seem more relevant, it is the chair of the department who you have to keep happy. Their opinion matters more than ours.

Describe the dwindling number of students as a concern, then see where the conversation goes.  Instead of offering your suggestion, invite your chair to offer their thoughts and suggestions first, before you launch into yours.  Who knows?  Perhaps they will suggest closing down your section on their own and you get what you want.  Their response probably also depends on the attractiveness of your research alternative versus their need to find instructors to cover sections they've already committed to offering.

Your chair may also be glad you invited the conversation, because they may have some feedback based on their own observations and conversations with your colleagues and students they would like to share with you.  They may be able to tell you how you could get better so you could one day be the instructor who draws away everyone else's students.

But I agree with and upvoted DBB's answer, that trying to dump the class could make you look bad and that if you care about getting good at teaching, you should simply "push through" and deliver the lectures.