Which is better for PhD application: research-based BS thesis somewhat related to interests, or extra courses + standard thesis in area of interest? I am studying applied math. My bachelor's thesis is coming up next semester and I've got a dilemma.

I was originally planning to do a standard project (most of them seem very soft to be honest) and then take an extra course (I got permission to take a PHD level course in information theory) or two simultaneously.

But then a project involving a fairly novel part of control theory popped up, novel enough that it might be possible to get something interesting published from the project. It seemed like a fairly interesting problem and the possibility of co-authoring a paper at this stage is very appealing as I have an eye on a PHD after my masters degree.

Control theory itself doesn't interest me that much, but the tangential mathematical subjects do (optimization, algorithms, information theory, etc). The problem is that it's probably going to be a lot more time consuming than a normal project and I don't know if I'd benefit more from it than just taking another course in an area I'm actually interested in.

I'm nervous it's going to be a stressful spring without much to show for it if we don't manage to get any interesting results at all. On the other hand, I suppose any research experience is positive even if it's not in my main interest area.

So, the question is: for a PhD application, should I complete a research-based thesis tangentially related to areas of interest, or normal thesis focused on my areas of interest plus additional courses in my areas of interest?

(edit: I'd probably drop a PDE course in favor of this project)
If your plans to pursue a PhD eventually are serious, I would definitely recommend the research-based option. You'll see this advice in many other places, but acquiring a PhD is very much the process of learning how to do research and proving that you can indeed do it. Even if the project is not directly in an area that you're very interested in, you'll have the opportunity to start practicing your research skills. The prospect of a publication is also something to keep in mind; an entering PhD student who has already published is viewed as already having proved to some degree that they're capable of quality research, and I've never met a faculty member anywhere who didn't appreciate that.

You're exactly right in that the research experience will be valuable to you even if you do not end up with a publication. Why? Primarily, if you do good work you've given your faculty collaborators some very good material to put in the reference letters you're going to want them to write when you apply to PhD programs. In fact, in your position I'd probably be thinking more about coming out of the experience with a strong reference than a publication.