Is it worth it going to academic conferences as undergraduate without any research to present? In particular, I would like to focus on CS Theory conferences like STOC or FOCS.

I am interested in the presented topics and, of course, it would be useful to learn more about it for potential graduate research. But on the other hand, one can always read the presented papers when they are published. It seems that the most important part of conferences is networking, however, I do not see myself being efficient in networking at this point yet.
If the conference is "affordable" to you in terms of time and money, go for it! It will be a great way to see more about how academia works. As others mention, there are often some workshops for students, and poster sessions are great ways to learn about new ideas and ask (even basic) questions. 

You may not get very much out of going to more than one conference without presenting, though. (If you love the first one and it inspires you to work hard on your research, maybe attending another without presenting anything would be a good thing, but much of the value for you at first will be in understanding how academics communicate with each other.)

Also, check if there are interesting conferences coming up in your region. Often local students can volunteer to help with registration desk or other key work during the conference in exchange for admission to some parts of the conference. This would be a way to get many of the benefits of attending the conference, without having to spend a lot of money to do so.