Williams was born in Georgia and moved to Chicago with his grandmother at the age of three. He reunited with his mother, who taught him to play the piano and would take him to the symphony. At a young age, Williams acquired tuberculosis, which slowed him down as a teenager. However, later he was able to perform at social events and even form his own gospel vocal quartet, The Jubilee Boys.
By the end of the 1930s, he had made the transition to the Chicago club scene and appeared with orchestras led by Jimmie Noone and Les Hite during. In 1941, he toured with Coleman Hawkins to Memphis, Tennessee. In 1943, he performed in Boston with the Lionel Hampton Orchestra. After leaving the Basie band, Williams had a successful career as a soloist at festivals, in clubs, and on television. He and Basie remained on good terms and he regularly appeared with the Basie orchestra when time allowed.
Williams sometimes worked as an actor, and in 1985, took the role of “Grandpa Al” Hanks in The Cosby Show. Williams also appeared several times on Sesame Street in the 1980s and early 1990s. Williams worked until his death in 1999.