I've been reading that Google can actually penalize you if some spammy websites backlinks to you, which I found totally unfair as I have lots of websites linking to me that I've never heard of. And finding which links are good or bad seems to be an impossible job with GWT.Should I care about those links or not? Google is offering a tool to disavow links, does it value my time to use it? 
Remove backlinks from bad sources or not?
seo;google search console;backlinks
If you have Google Search Console (formerly Webmaster Tools), you'll be notified if your site is being penalised. No notification? Don't worry. In general, penalties will only apply to links that are, or are suspected to be a result of practices prohibited by Google guidelines, especially those pertaining to link schemes. I wouldn't recommend using the disavow tool unless you have an explicit notification that action has been taken against your site, and then only when other clean-up methods have been exhausted. 