The only Prime Minister to serve four terms under that definition was William Ewart Gladstone (3 December 1868 -- 20 February 1874, 23 April 1880 -- 23 June 1885, 1 February 1886 -- 25 July 1886 and 15 August 1892 -- 5 March 1894).
who has been prime minister the most times
William Ewart Gladstone