Brown was born in South Lee, Massachusetts, the son of Mary Tyler and Billings Brown, and grew up in Massachusetts and Connecticut.
 His was a New England merchant family.
He attended Monson Academy, Monson, MA and entered Yale College at 16.
There he was a member of Alpha Delta Phi fraternity and was elected to membership in Phi Beta Kappa.
He earned a Bachelor of Arts degree there in 1856.
 Among his undergraduate classmates were Chauncey Depew, later a U.S. Senator from New York, and David Josiah Brewer, who became Brown's colleague on the Supreme Court.
 Depew roomed across the hall from Brown for three years in Old North Middle Hall, and remembered "a feminine quality [about Brown] which led to his being called Henrietta, though there never was a more robust, courageous and decided man in meeting the problems of life[.]"
 After a yearlong tour of Europe, Brown studied law with Judge John H. Brockway in Ellington, Connecticut, but his refusal to participate in a local religious revival made life there unpleasant for him.
He left Ellington to pursue legal studies, with a year at Yale Law School, and a semester at Harvard Law School.
Brown was born in South Lee, Massachusetts, the son of Mary Tyler and Billings Brown, and grew up in Massachusetts and Connecticut.  His was a New England merchant family. He attended Monson Academy, Monson, MA and entered Yale College at 16. There he was a member of Alpha Delta Phi fraternity and was elected to membership in Phi Beta Kappa. He earned a Bachelor of Arts degree there in 1856.  Among his undergraduate classmates were Chauncey Depew, later a U.S. Senator from New York, and David Josiah Brewer, who became Brown's colleague on the Supreme Court.  Depew roomed across the hall from Brown for three years in Old North Middle Hall, and remembered "a feminine quality [about Brown] which led to his being called Henrietta, though there never was a more robust, courageous and decided man in meeting the problems of life[.]"  After a yearlong tour of Europe, Brown studied law with Judge John H. Brockway in Ellington, Connecticut, but his refusal to participate in a local religious revival made life there unpleasant for him. He left Ellington to pursue legal studies, with a year at Yale Law School, and a semester at Harvard Law School.
Henry Billings Brown