Fred Wallace Smith was born on 11th August 1944 in Marks, Mississippi. He was named after his father, James Frederick Smith and was the younger of two sons. His father was an entrepreneur and businessman, who founded the Dixie Greyhound Bus Lines. Fred Sr. further founded the Toddle House restaurant chain together with his first son to supplement the family fortune.