Q:

When to use "be worth VERBing" and when to use "be worthy VERBing"?

Consider this sentence:

This movie is worth watching.
This movie is worthy watching.

I am confused about these two words worth and worthy. When should you use “be worth VERBing” and when should you instead use “be worthy VERBing”?

A:

worthy V-ing is not acceptable at all, so I assume you really meant to ask about worthy of V-ing.
worthy of is usually used with a noun, rather than a verb, as in

The esteemed professor is worthy of respect.

This Google Ngram shows that worthy of watching is used a miniscule amount compared to worth watching.
worth V-ing is the normal way to express this, as per this definition:

2.1 Used to suggest that the specified course of action may be advisable:
  the company’s service schemes are worth checking out

