Rankin died Sunday at Cedars-Sinai Medical Center, Mack Avenue Records spokesman Don Lucoff said.
The musician, who first gained acclaim as one of the guitarists on Bob Dylan's landmark 1965 album, "Bringing it all Back Home," had been preparing to record an album of new material when he became ill a few weeks ago. Recording sessions scheduled with producer Phil Ramone were canceled as his health began to deteriorate.
"That he was still at the top of his game is one of the saddest parts of his passing for me," Denny Stilwell, president of Mack Avenue Records, said in a statement. "He performed the new material in our office over the last few months and his voice was still in its finest form _ he sounded absolutely amazing. Our hearts and prayers are with his family."
Rankin wrote and recorded the pop standard "Peaceful" and also wrote "In The Name of Love," which was recorded by Peggy Lee, and "Haven't We Met," performed by Carmen McRae and Mel Torme.
His own "The Kenny Rankin Album" was recorded live in 1976 with a 60-piece orchestra.
Rankin, who signed with Decca Records while still in his teens, once said his music career really began in the fourth grade when he sang "O Holy Night" in a school Christmas play and his teacher walked up to him afterward, patted him on the head and said, "Kenneth, that was lovely."
"She set me on the path in music that I find myself on today," he said.
After signing with Decca in the late 1950s he released a handful of singles before moving on to Columbia Records, also the home of Dylan. There, he took part in the recording of "Bringing it All Back Home," the album in which Dylan moved firmly from an all-acoustic folk music sound to an electric mixture of pop and rock 'n' roll.
Soon after, he made his first appearance on "The Tonight Show," where he impressed host Johnny Carson so much that Carson contributed liner notes to Rankin's first album, 1967's "Mind Dusters." Other albums included "Family," "Like a Seed," "Inside and "Silver Morning:"
He would go on to appear as Carson's "Tonight Show" guest more than 20 times.
His supple tenor voice on such recordings as "Spanish Harlem," "'Round Midnight" and the Beatles' "Blackbird" and "I've Just a Face" also won him the respect of fellow musicians as a singer's singer.
Mack Records said he so impressed the Beatles' Paul McCartney that McCartney asked him to perform "Blackbird" when he and songwriting partner John Lennon were inducted into the Songwriters Hall Of Fame.
Rankin spent much of the past 20 years touring, although he returned to the studio in the mid-1990s to release a pair of albums, "Professional Dreamer" and "Here In My Heart." He also released a Christmas album in 1999.
Born in New York City on Feb. 10, 1940, Rankin was raised in the city's Washington Heights neighborhood, where he said he grew up listening to a broad spectrum of music, including Afro-Cuban, jazz, Top 40 and Brazilian.
He is survived by his son, Chris Rankin, daughters Chanda Rankin and Jena Rankin-Ray and a granddaughter.
His record label said funeral arrangements are pending.