Spencer married Lady Mary Beauclerk (4 December 1743 – 13 January 1812), daughter of Vere Beauclerk, 1st Baron Vere and sister of Aubrey Beauclerk, 5th Duke of St Albans, on 2 October 1762.
They had three sons.

Robert Spencer (circa 1764 – 1831)
John Spencer (21 December 1767 – 17 December 1831)
William Robert Spencer (9 January 1769 – 23 October 1834)
Lady Charles Spencer died in January 1812 aged 68.
Spencer survived her by eight years and died in June 1820, aged 80.
Spencer married Lady Mary Beauclerk (4 December 1743 – 13 January 1812), daughter of Vere Beauclerk, 1st Baron Vere and sister of Aubrey Beauclerk, 5th Duke of St Albans, on 2 October 1762. They had three sons.
Robert Spencer (circa 1764 – 1831)
John Spencer (21 December 1767 – 17 December 1831)
William Robert Spencer (9 January 1769 – 23 October 1834)
Lady Charles Spencer died in January 1812 aged 68. Spencer survived her by eight years and died in June 1820, aged 80.
Lord Charles Spencer