Before the arrival of Europeans, the mouths of streams that eroded gullies in the East River bluffs are conjectured to have been the sites of fishing camps used by the Lenape, whose controlled burns once a generation or so kept the dense canopy of oak–hickory forest open at ground level.

In the 19th century the farmland and market garden district of what was to be the Upper East Side was still traversed by the Boston Post Road and, from 1837, the New York and Harlem Railroad, which brought straggling commercial development around its one station in the neighborhood, at 86th Street, which became the heart of German Yorkville.
The area was defined by the attractions of the bluff overlooking the East River, which ran without interruption from James William Beekman's "Mount Pleasant", north of the marshy squalor of Turtle Bay, to Gracie Mansion, north of which the land sloped steeply to the wetlands that separated this area from the suburban village of Harlem.
Among the series of villas a Schermerhorn country house overlooked the river at the foot of present-day 73rd Street and another, Peter Schermerhorn's at 66th Street, and the Riker homestead was similarly sited at the foot of 75th Street.
By the mid-19th century the farmland had largely been subdivided, with the exception of the 150 acres (61 ha) of Jones's Wood, stretching from 66th to 76th Streets and from the Old Post Road (Third Avenue) to the river and the farmland inherited by James Lenox, who divided it into blocks of houselots in the 1870s, built his Lenox Library on a Fifth Avenue lot at the farm's south-west corner, and donated a full square block for the Presbyterian Hospital, between 70th and 71st Streets, and Madison and Park Avenues.
At that time, along the Boston Post Road taverns stood at the mile-markers, Five-Mile House at 72nd Street and Six-Mile House at 97th, a New Yorker recalled in 1893.

The fashionable future of the narrow strip between Central Park and the railroad cut was established at the outset by the nature of its entrance, in the southwest corner, north of the Vanderbilt family's favored stretch of Fifth Avenue from 50th to 59th Streets.
A row of handsome townhouses was built on speculation by Mary Mason Jones, who owned the entire block bounded by 57th and 58th Streets and Fifth and Madison.
In 1870 she occupied the prominent corner house at 57th and Fifth, though not in the isolation described by her niece, Edith Wharton, whose picture has been uncritically accepted as history, as Christopher Gray has pointed out.

It was her habit to sit in a window of her sitting room on the ground floor, as if watching calmly for life and fashion to flow northward to her solitary door... She was sure that presently the quarries, the wooden greenhouses in ragged gardens, the rocks from which goats surveyed the scene, would vanish before the advance of residences as stately as her own.

— Edith Wharton
Before the arrival of Europeans, the mouths of streams that eroded gullies in the East River bluffs are conjectured to have been the sites of fishing camps used by the Lenape, whose controlled burns once a generation or so kept the dense canopy of oak–hickory forest open at ground level.
In the 19th century the farmland and market garden district of what was to be the Upper East Side was still traversed by the Boston Post Road and, from 1837, the New York and Harlem Railroad, which brought straggling commercial development around its one station in the neighborhood, at 86th Street, which became the heart of German Yorkville. The area was defined by the attractions of the bluff overlooking the East River, which ran without interruption from James William Beekman's "Mount Pleasant", north of the marshy squalor of Turtle Bay, to Gracie Mansion, north of which the land sloped steeply to the wetlands that separated this area from the suburban village of Harlem. Among the series of villas a Schermerhorn country house overlooked the river at the foot of present-day 73rd Street and another, Peter Schermerhorn's at 66th Street, and the Riker homestead was similarly sited at the foot of 75th Street. By the mid-19th century the farmland had largely been subdivided, with the exception of the 150 acres (61 ha) of Jones's Wood, stretching from 66th to 76th Streets and from the Old Post Road (Third Avenue) to the river and the farmland inherited by James Lenox, who divided it into blocks of houselots in the 1870s, built his Lenox Library on a Fifth Avenue lot at the farm's south-west corner, and donated a full square block for the Presbyterian Hospital, between 70th and 71st Streets, and Madison and Park Avenues. At that time, along the Boston Post Road taverns stood at the mile-markers, Five-Mile House at 72nd Street and Six-Mile House at 97th, a New Yorker recalled in 1893.
The fashionable future of the narrow strip between Central Park and the railroad cut was established at the outset by the nature of its entrance, in the southwest corner, north of the Vanderbilt family's favored stretch of Fifth Avenue from 50th to 59th Streets. A row of handsome townhouses was built on speculation by Mary Mason Jones, who owned the entire block bounded by 57th and 58th Streets and Fifth and Madison. In 1870 she occupied the prominent corner house at 57th and Fifth, though not in the isolation described by her niece, Edith Wharton, whose picture has been uncritically accepted as history, as Christopher Gray has pointed out.
It was her habit to sit in a window of her sitting room on the ground floor, as if watching calmly for life and fashion to flow northward to her solitary door... She was sure that presently the quarries, the wooden greenhouses in ragged gardens, the rocks from which goats surveyed the scene, would vanish before the advance of residences as stately as her own.
— Edith Wharton
Upper East Side