I could give a whole speech about how using a spell based on navigation systems that subtracts where everything is in an area from where they aren't but frankly without expressing it mathematically it even makes my head spin, I mean go back to the sentence I just said and tell me what I meant by that, you just can't. Anyways basically there is a big bubble of nothing where there should be something and all I need to do to find where I'm looking for is to find that big bubble of nothing. and right up from the graveyard is where I find it. I really should of guessed that but like all magics that affect the mind the fidelius get's a bit fiddly when you try to deduct the location of a place from existing knowledge. The only reason I'm able to find the Manor is because I'm not looking for the manor I'm looking for nothing. Literally. Is you head spinning yet?