You're still not getting it.  Rejecting the CDF is to reject the authority that conferred his 
priesthood.  The same authority that put the CDF in place, is the one that "laid hands on him".  There is one Church.

Your comment though does point out that much work needs to be done to better form the lay about the Church.  Remarkable amount of work not done.