Is it unethical of me and can I get in trouble if a professor passes me based on an oral exam without attending class? I have been experiencing anxiety based on some recent events. During my last semester of college I enrolled in a language course in a language in which I was already fluent. I did this to enable quicker graduation.

After the first two weeks of the class, it became apparent I wasn't going to benefit at all from this course. Additionally, as I was soon graduating, I had other work to perform. I asked the professor if I could stop attending classes and simply turn in the homework by the end of the semester. I did this while speaking the language that she taught. She said "yes".

I didn't attend class for a month and a half, and didn't hear from the professor again. Towards the end of the semester, I emailed the professor to inquire what she wanted me to do to pass the course. She scolded me because she wanted me to attend at least some of the classes (I had misunderstood her in our first conversation). She then told me that despite that, it was OK, she would give me a passing grade (via oral exam), as "you are fluent in the language and shouldn't come to class as it will seem unfair to the other kids."

This appeared to me as if she would pass me based on merit alone. I was VERY shook up, and called her again the same week telling her that I feel like withdrawing and holding off graduation and just taking a different course next semester before starting grad school. She reassured me I'm all set and I need not worry.

Graduation happens, however I was still uneasy as I was preparing to start grad school. I felt like I cheated the system. I contacted her again, and again she told me not to worry. After the first week of grad shool, I called her again and demanded I take the midterm and final for the course in an attempt to ease my mind and worry. Scored 99 and 96 on them without studying.

I still feel really guilty for some reason, and I'm still trying to cope with this. Its honestly affecting my performance in school. MY worse nightmare is that my undergrad will find out and take away my degree and that this will affect my grad school as I needed a bachelors to enter. The professor made it seem as if she was taking a risk by letting me do what I did. She made it seem I was doing something illegal. She told me not to tell my classmates..

As such, my question is: If what happened was found out, would my grad school have grounds to kick me out? If my undergrad found out, can they pull me degree?

Edit: Thanks for the answers, I guess why I'm soo stressed about this is because, to my understanding, she made it obvious that I and the prof. were doing something against school policies, I knew about it and didn't think much about it.
I feel like you're overthinking this. What you did seems an improvement to me over taking, say, bowling to fill a last credit, which isn't uncommon in undergrad. The professor gave you your grade and you're off to grad school. There's no reason to revisit it, with yourself or anyone else.

Also, it isn't your responsibility, as a student, to audit how a professor grades a class.  They may have even consulted with others before letting you do this; you don't know.  Don't make assumptions - you did what you did, and the professor decided it was enough to pass the class.

Put this behind you and focus on grad school.