Joseph was born in Shreveport as one of five children to Joseph W. and Earlene V. Thomas Walker. His mother was only "thirteen or fourteen years old when I was born," he says. She left to work in California as a dietician at the University of California when Joseph was a small child. His grandmother raised him in a shotgun house with an outhouse in the yard. He walked more than five miles to church. Joseph moved with the family to the country and attended the one-room Shady Grove School where his father was principal. He returned to Shreveport in the seventh grade, and later attended Central Colored High School, then Booker T. Washington. To make extra money he shined shoes, a nickel per pair. Joseph learned how to repair shoes and went to work at Barksdale. Married and divorced, he was drafted into the U.S. Army. He took basic training at Fort Bliss in El Paso, and eventually become a supply sergeant at Fort Polk. Discharged in 1956, Joseph attended Wiley College for two years. In Shreveport, he opened his business, Walker House Cleaning Service. He faced discrimination while bidding for contracts. Meanwhile, he began playing tennis competitively, often winning over white opponents. Of the civil rights struggle he credits the women for doing "wonderful things." Of whites' attitudes towards blacks in that time, he recalls, "You were always a boy." Joseph married Rosa Harrison on June 3, 1964. They have eight children (two are hers), fifteen grandchildren, and three great-grandchildren.