Leave out degree on academic CV? Say I have done a BA, two MA's, and then a PhD (which, technically, includes another MA).

Could I then just not mention one of my MA's on my CV or on my profile on the faculty website? Will I have to list everything, all the time?
It needs to be on your CV when submitted for application to any position. Most institutions would find missing information to be misleading at best and fraudulent at worst. Explaining how you spent the years of your life between starting university and now is part of the purpose of your CV and relevant to the application process. Leaving things out makes it appear that you are trying to hide something, and people reviewing your application won't like that. 

If you don't want to highlight this degree on your website, there's likely no real harm since your website isn't part of your application to anything.