Q:

Is it mandatory to implement self.instances and self.prefetch methods for a Puppet provider

we are writing a Puppet module for a networked storage device. We are implementing custom types and providers and for one of the type we will have on an average around 40k objects. 
Getting these many resources thru pre-fetch and self.instances will be very performance intensive. Is it mandatory to implement self.instances and pre-fetch methods for a provider? What will we lose if we do not implement them?

A:

Is it mandatory to implement self.instances and pre-fetch methods for
  a provider?

No, it is not.

What will we lose if we do not implement them?

You'll not be able to use a Resources resource to purge unmanaged resources of the type in question.  But with so many resources, I'm inclined to think that you'll be actively managing only a few, and will not want to purge the others anyway.
You'll not be able to use the puppet resource command to enumerate all resources of your custom type.
You'll also need either to retrieve individual resource properties on demand, when the provider's getter methods are invoked, or else to track whether each individual resource's state has been retrieved and have each getter first load the state if needed.  Either is doable.
For comparison, consider File resources, which seem conceptually similar to what you have in mind.  Puppet makes no attempt to prefetch all File resources of a given system, and there's no practical way it could do.

