The 43-year-old composer, hailed in India as the Mozart of the Madras, won Oscars Sunday for best original score and song, both from the movie "Slumdog Millionaire."
Other Indians to collect Oscar gold before him were costume designer Bhanu Athaiya, honored for "Gandhi" in 1982, and arthouse director Satyajit Ray, who received a lifetime achievement award in 1992.
"I was excited and terrified," the soft-spoken Rahman said as he collected his first Oscar, for best original score. "The last time I felt like that was during my marriage."
He recalled the title of a Hindi film ballad, "I Have Nothing But I Have a Mother" as he thanked his mother Kareema Begum, who made the trip from Chennai, India, to join the audience at the Kodak Theatre.
He then added a phrase in the Tamil language, meaning "God is great," which he normally says after receiving an award.
Instead of leaving the stage like other Oscar recipients, Rahman then sang his two Oscar-nominated songs, "O ... Saya" and "Jai Ho," backed by dancers and an international drum line with Indian, African, Japanese and Chinese percussionists.
Then he picked up the best song Oscar for "Jai Ho," the title of which translates in English to "Be Victorious." The global celebratory anthem is sung during the Bollywood musical finale of "Slumdog Millionaire."
Rahman is no stranger to success in India where he has written scores for more than 130 films since 1992. His fans there and elsewhere have made him one of the world's best-selling recording artists, on par globally with the Rolling Stones and Madonna.
He believes music and film can bring people together despite boundaries of race, nationality and religion.
"All my life, I had a choice of hate and love. I chose love, and I'm here," he said as he collected his second Oscar.
Long an advocate of the universality of music, Rahman has called on people to be open to all styles, from heavy metal to jazz to hip hop to R&B.
Danny Boyle, who won the best director Oscar for "Slumdog Millionaire," said that he attempted to give Rahman the leeway he needed to pursue various musical forms in scoring the movie.
"I'm really happy and grateful it worked out," Rahman said.
Rahman first gained wider recognition in the West when he wrote the score for Andrew Lloyd Webber's 2002 London production of the musical "Bombay Dreams."