R. J. Reynolds Tobacco Holdings Inc., the No. 2 United States cigarette maker, said yesterday that it would pay a quarterly dividend of 77.5 cents a share, less than analysts had expected. The cash dividend is the company's first since the tobacco business was separated from the RJR Nabisco Holdings Corporation last month. It will be payable Oct. 1 to shareholders as of Sept. 10. The payout equals $3.10 a share on an annual basis; analysts had forecast $3.23 to $3.60. R. J. Reynolds, of Winston-Salem, N.C., had told analysts that it would set the dividend at about half of estimated cash earnings for 1999. The shares fell 56.25 cents, to $27.50