Lyttelton married Lady Sarah Spencer, daughter of George John, 2nd Earl Spencer, on 4 March 1813; she was for a time governess to the children of Queen Victoria and a Lady of the Bedchamber, and died 13 April 1870.
They had three sons and two daughters:
The Honorable Caroline Lavinia Lyttelton (1 February 1816 – 8 April 1902)
George William Lyttelton, 4th Baron Lyttelton (31 March 1817 – 19 April 1876) he married Mary Glynne on 25 July 1839.
They have twelve children.
He remarried Sybella Harriet Clive on 10 June 1869.
They have three daughters
The Honorable Spencer Lyttelton (19 June 1818 – 4 February 1889) he married Henrietta Cornewall on 10 August 1848.
They have one son.

Reverend The Honorable William Lyttelton (3 April 1820 – 24 July 1884) he married Emily Pepys on 28 September 1854.
He remarried Constance Yorke on 5 February 1880.

The Honorable Lavinia Lyttelton (1821-3 October 1850) she married Reverend Henry Glynne  rector of Hawarden.
on 14 October 1843.
They have four children.
Lyttelton married Lady Sarah Spencer, daughter of George John, 2nd Earl Spencer, on 4 March 1813; she was for a time governess to the children of Queen Victoria and a Lady of the Bedchamber, and died 13 April 1870. They had three sons and two daughters:
The Honorable Caroline Lavinia Lyttelton (1 February 1816 – 8 April 1902)
George William Lyttelton, 4th Baron Lyttelton (31 March 1817 – 19 April 1876) he married Mary Glynne on 25 July 1839. They have twelve children. He remarried Sybella Harriet Clive on 10 June 1869. They have three daughters
The Honorable Spencer Lyttelton (19 June 1818 – 4 February 1889) he married Henrietta Cornewall on 10 August 1848. They have one son.
Reverend The Honorable William Lyttelton (3 April 1820 – 24 July 1884) he married Emily Pepys on 28 September 1854. He remarried Constance Yorke on 5 February 1880.
The Honorable Lavinia Lyttelton (1821-3 October 1850) she married Reverend Henry Glynne  rector of Hawarden. on 14 October 1843. They have four children.
William Lyttelton, 3rd Baron Lyttelton