Is it possible to be admitted to graduate programs in subjects one minored in in undergraduate? I'm currently studying electrical engineering and considering on getting both a physics and math minor. I'm wondering though, is it possible to get a Ph.D. or a Master's degree in Physics or Math when I only have the minor and not a real degree? 

Is it likely at all to be admitted into a graduate study with only a minor? I'm wondering because I don't know whether or not the minor is actually worth it if I won't be able to pursue a graduate degree later on with my credentials.

For reference, a math minor at my school typically means 3 extra upper-level math courses (300-400 level), and only 2 more physics courses (one in waves + light + thermodynamics; the other which samples different fields of modern physics, all required for the degree program)
In the U.S., for graduate work in mathematics, even a minimal major in mathematics is not really adequate preparation for a good graduate program in mathematics. Not enough coursework, which means insufficient exposure to higher-level ideas (assuming, as is apparently mostly the case, there's not much interaction with mathematics outside coursework). Even though various engineering programs require quite a bit of mathematics, in my observation the topics exactly do not get to fourth-year mathematics, so that exactly rigorous analysis, abstract algebra, point-set topology are not seen. The latter are a very minimal preparation for graduate work in mathematics of almost any sort. Still, if there's room for those courses in a "minor", then some minimum thresh-hold would be reached for appearing to be prepared for math grad school.