"I have everything in the world I want, and then I've got all this surplus around - and I give away the surplus," Buffett said.
So the two men, with a combined net worth of somewhere near $100 billion threw a very exclusive dinner 13 months ago. They invited 14 billionaires, known for their charitable giving, including Ted Turner, Oprah Winfrey and Michael Bloomberg.
"We just started going around the table," Buffett told PBS' Charlie Rose. "And I asked each either couple or individual to sort of give a story of the evolution of their own philanthropic thinking and how they got where they are now and what they hope to accomplish."
The conversation last night led to this website Wednesday - givingpledge.org, where billionaires can post public pledges to give 50 percent of their fortunes away during the course of their lives or when they die.
Do they suffer from a guilty conscience?
"Warren says in his pledge that guilt is not part of his thinking at all, instead it's gratitude," said Fortune magazine's Carol Loomis.
Loomis knows Buffett well. He shared his ultimate goal with her.
"They sort of set their sights at $600 billion," Loomis said. "It's a lot of money and if it went back to society it would do enormous good."
Enormous - as in enough money to wipe out pediatric AIDS around the world - and ensure the legacy of giving that Buffett and gates think is the true measure of a rich life.