In December 1925, Lee married Juanita Eugenia Waller.
On November 22, 1926, they had a son, Carl Vincent Canegata, who became actor Carl Lee.
The couple separated while their son was young, and they were amicably divorced in 1942.

In 1934, Lee began a love affair with publisher and peace activist Caresse Crosby, despite the threat of miscegenation laws.
They often had lunch in uptown New York in Harlem at the then-new restaurant "Franks", where they could maintain their secret relationship.
When Lee was performing in Washington, D.C., during the 1940s, the only restaurant in the city where they could eat together was an African restaurant named the Bugazi.
Crosby and Lee's intimate relationship continued into the mid-1940s.

In March 1951, Lee married Frances Pollack.
They remained together until he died just over a year later.
In December 1925, Lee married Juanita Eugenia Waller. On November 22, 1926, they had a son, Carl Vincent Canegata, who became actor Carl Lee. The couple separated while their son was young, and they were amicably divorced in 1942.
In 1934, Lee began a love affair with publisher and peace activist Caresse Crosby, despite the threat of miscegenation laws. They often had lunch in uptown New York in Harlem at the then-new restaurant "Franks", where they could maintain their secret relationship. When Lee was performing in Washington, D.C., during the 1940s, the only restaurant in the city where they could eat together was an African restaurant named the Bugazi. Crosby and Lee's intimate relationship continued into the mid-1940s.
In March 1951, Lee married Frances Pollack. They remained together until he died just over a year later.
Canada Lee