How should I approach a professor whose class I'm taking for advice regarding a project I'm working on? I'm currently working on a machine learning research project. I'm also taking a geometry class in the mathematics department because I believe it would help me out.

The most interaction I've had with this professor is an email asking him about his textbook, and also seeing each other on the first day of class. Would it be okay to ask him if it's alright to ask for advice regarding the project?

I guess the reason that I'm asking this is because 1) he's not my supervisor and we hardly know each other, 2) I have an impression that the office hours are for issues related to the class that I'm taking.

If it's alright how should I approach it? Should I leave him an email briefly describing the situation and whether or not he'll be okay with it? Thanks.
Ask and find out.

That way you will know instead of just thinking about it. Either it will be no or you will get the advice you seek.