At fourteen he worked as an usher at the NYC Paramount Theatre. His father was an electrician who played guitar and his mother taught pianon. Damone attended PS 163 and sang in St. Finbar's choir and later attended the Alexander Hamilton Vocational High School and then Lafayette High School in Brooklyn. He left school at sixteen to support his family, but returned to graduate from Lafayette in 1997. Damone won first prize in an Arthur Godfrey talent scouts contest in 1945. His first night club appearance at the LA Martinique Club was set up by comedian Milton Berle. He was drafted and served in the army from 1951 to 1953. After he was discharged he married Pier Angeli, the actress whom he later divorced. Angeli remarried and died due to a drug overdose of barbituates in Hollywood, California in 1971. Damone was later married to Becky Ann Jones from 1974 to 1982 and Diahann Carroll from 1987 to 1996. He married Rena Rowan, fashion designer and co-founder of Jones New York, in 1998. In 1999, he received a certificate of advanced study from Philadelphia University.
Damone was his mother's maiden name. He has a son with Pier Angeli, Perry, who works as a DJ for a radio station in Phoenix, Arizona. Cousin of singer/actress Doretta Morrow. His sister Sandy was shot to death by her ex-husband Abby who then killed himself at the bingo hall in North Miami where they both worked in the mid 80's.