Is dropping one of my major considered "bad" if I were to study related to that field? I'm a third year Physics &amp; Mathematics student, and through the years, I have done lots of self-study in semesters/winter breaks/summer breaks. 

Now the thing has come to a point that I can learn almost anything by myself as long as I have the necessary prerequisites for studying that subject.In fact, what I have observed in these last 2 years is that I barely learn anything in the lectures I take compare to my self-studies. That is why, if I didn't have to take exams, I would spend all my time self-studying the lectures I need to graduate, and more.

Now the problem is, I'm not taking not much (in fact not at all) benefit from my physics major, and it seems to me that it just wastes my time. That is why, I was thinking to drop of physics, and just graduate from my mathematics major.

However, if I were to study in a field related to physics, would this have negative impact in my future career [In fact, I want to study to in a field that is the intersection of Physics and Mathematics; so I'm not planning to go for pure mathematics]? I mean after all, I'm dropping out that major, but the only reason that I'm doing is so that I can have much more time to self-study by which I learn better.
Would this have negative impact in my future career?


Absolutely no one cares what you did in undergrad, except grad school admissions. Further, as Nate Eldredge said in the comments:


  I don't think it would so much be an issue that you dropped the physics major, as that you don't have a physics degree.


So, I think your question is equivalent to:


  Am I less likely to get into grad school for physics/math with only a math degree?


You're certainly less likely to get into physics grad school without a physics degree (though with enough coursework, it's not impossible). For math grad school, I'm less qualified to comment, but I strongly suspect that a dual degree would have more weight than one degree + coursework. So, yes, I think the second degree would help you get into grad school. Only you can weigh this reality against the advantages of self-studying. 

By the way, I also wanted to comment on this line:


  the only reason that I'm doing is so that I can have much more time to self-study by which I learn better.


This has been discussed elsewhere, the consensus being that while many passionate students feel this way, few truly do not benefit from a course's structure and a professor's experience. Just one other factor to consider in your decision.