Mary was, along with her elder brother Philip, older sister Susanna (1727–1822), and younger sister Margaret (1733-1752), a one-quarter heir to the roughly 250 sqmi "Highland Patent" of her father (later to become known as the Philipse Patent, and in time today's Putnam County of southeastern New York).