Between the fence and the west-side glamour-wall are deadly wards; you will need ward-breakers to tear down those wards. Some of those wards keep the Fidelius Charm strong, and keep strong the glamour that shows you a brick wall when there is no wall there. But for only one of the wardstones can you be sure where it is.